andygrove commented on issue #5504: URL: https://github.com/apache/datafusion-comet/issues/5504#issuecomment-5850466367
This affects memory as well as metrics, which makes it more urgent than `priority:low`. The producer that `releasePlan` leaves running still owns the stream's reservations. When a task stops consuming early or fails, Spark's end-of-task cleanup finds them (`Managed memory leak detected`) and frees them for other tasks while the native buffers are still allocated. The producer releases them a few milliseconds later, and that is the `Internal error: release called on ... but task only has 0 bytes` warning in #2453. I reproduced it on main, and the details are in https://github.com/apache/datafusion-comet/issues/2453#issuecomment-5850458856. The fix this issue proposes, keeping the handle and having `releasePlan` cancel and wait within a bound, would fix both. I'm raising the priority and adding `area:memory`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
