XiaoHongbo-Hope commented on PR #633:
URL: https://github.com/apache/paimon-rust/pull/633#issuecomment-5158056173
> Row-group producers fail to stop in a timely manner after downstream
cancellation Location: parquet.rs:517, parquet.rs:667 Issue: The producer first
calls `reserve()` to reserve a channel, then waits for `stream.next()`; if the
downstream drops the receiver due to `LIMIT`, an error, or query cancellation,
the detached task waiting for I/O will not receive the cancellation
notification and will continue to hold the `ParquetReadPermit`. Trigger
Conditions: Start `forward_row_group_batches` using a pending stream, then drop
the receiver; if the task has not exited within 100 milliseconds, the test
consistently fails. Impact: Unnecessary object storage requests and decoding
continue to run; more importantly, they continue to consume scan-shared
row-group/byte permits, which may slow down DataFusion partitions that are
still active. The fix is very simple:
>
> ```
> let next = tokio::select! {
> _ = sender.closed() => return,
> next = stream.next() => next,
> };
> ```
>
> It is recommended that the coordinator listen for `row_group_tx.closed()`
while waiting for `read_budget.acquire(...)`; after canceling the `acquire`
future, the acquired semaphore permits will be automatically released. After
applying the above fix, both my cancellation reproduction test and the original
backpressure test passed.
Thanks, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
