JingsongLi commented on PR #633:
URL: https://github.com/apache/paimon-rust/pull/633#issuecomment-5129159642
Row-group producers fail to stop in a timely manner after downstream
cancellation
Location: parquet.rs:517, parquet.rs:667
Issue: The producer first calls `reserve()` to reserve a channel, then waits
for `stream.next()`; if the downstream drops the receiver due to `LIMIT`, an
error, or query cancellation, the detached task waiting for I/O will not
receive the cancellation notification and will continue to hold the
`ParquetReadPermit`.
Trigger Conditions: Start `forward_row_group_batches` using a pending
stream, then drop the receiver; if the task has not exited within 100
milliseconds, the test consistently fails.
Impact: Unnecessary object storage requests and decoding continue to run;
more importantly, they continue to consume scan-shared row-group/byte permits,
which may slow down DataFusion partitions that are still active.
The fix is very simple:
```
let next = tokio::select! {
_ = sender.closed() => return,
next = stream.next() => next,
};
```
It is recommended that the coordinator listen for `row_group_tx.closed()`
while waiting for `read_budget.acquire(...)`; after canceling the `acquire`
future, the acquired semaphore permits will be automatically released. After
applying the above fix, both my cancellation reproduction test and the original
backpressure test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
