sunchao opened a new issue, #5504: URL: https://github.com/apache/datafusion-comet/issues/5504
### Describe the bug The async I/O execution path discards its producer `JoinHandle`. When Spark closes a plan early, `releasePlan` snapshots metrics before the receiver and context are dropped. The producer can still own the stream, so stream-drop metric guards can update counters only after the final snapshot. Closing the consumer also does not explicitly request producer cancellation. ### Steps to reproduce Start an async batch producer whose stream owns a drop-time metric guard and remains pending, then close its consumer before it finishes. A final snapshot taken before the producer releases the stream misses the guard's update. A producer inside synchronous decoding is a distinct case: it cannot observe cancellation until it yields. ### Expected behavior Retain producer ownership, close the receiver, request cancellation, and wait only within a bounded budget before the final metric snapshot. Cooperative completion should expose drop-time updates; timeout must explicitly remain a potentially incomplete snapshot. There must be no claim that a timeout makes external JVM task state safe to tear down. ### Additional context Extracted from #5453 to review cancellation behavior and close latency independently of new I/O counters. Existing producer ownership is `async move` over the stream and sender, not a captured raw `ExecutionContext` pointer. Cancellation remains cooperative; this issue does not promise preemption of synchronous decoding or complete accounting after a timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
