peterxcli opened a new issue, #5446: URL: https://github.com/apache/datafusion-comet/issues/5446
### What is the problem the feature request solves? PR #5038 reuses the Arrow IPC write context across shuffle blocks, but Comet currently pins Arrow 58.4.0. In that version, `CompressionContext` only owns an optional zstd compressor. Comet does not enable Arrow IPC buffer compression, and the `FlatBufferBuilder` is still allocated separately for every batch, so the merged lifetime change provides no allocation savings on current `main`. Starting with Arrow 59.1, `IpcWriteContext` owns the reusable `FlatBufferBuilder`, making the context reuse introduced by #5038 effective. The broader DataFusion/Arrow dependency upgrade is already tracked by #4865 and draft PR #5262. This issue tracks validating the deferred shuffle performance benefit after that upgrade lands. ### Describe the potential solution Close this issue when: - Comet uses Arrow 59.1 or newer through a compatible DataFusion upgrade. - The shuffle writer paths retain the per-writer and row-conversion-loop context lifetimes introduced by #5038 and use the Arrow 59.1+ write-context API. - Focused, repeated same-machine measurements demonstrate fewer IPC metadata allocations on the production shuffle paths, with the results recorded. - Existing shuffle IPC round-trip and writer tests pass. Comet's outer per-block compression encoders are independent and outside this issue's scope. ### Additional context - Follow-up comment: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5387092772 - Context-reuse implementation: #5038 - Benchmark evidence: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5082906359 - Original implementation issue: #5008 - DataFusion 55 tracker: #4865 - Current upgrade PR: #5262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
