peterxcli commented on PR #5038: URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5082906359
arrow 58.4: Production path | Baseline / patch blocks | FlatBuffer allocations per block | FlatBuffer bytes per block | Context allocations -- | -- | -- | -- | -- JVM sort → row.rs | 2,688 / 2,441 | 11 / 11 | 4,136 / 4,136 | 0 / 0 Native → BufBatchWriter | 5,857 / 4,298 | 11 / 11 | 4,136 / 4,136 | 0 / 0 arrow 59.1: 1,000 blocks | Fresh context | Reused context -- | -- | -- Allocation calls | 212,000 | 201,011 Requested bytes | 8,508,496,000 | 8,504,364,136 --- - In Arrow 58.4, `CompressionContext` contains only an optional zstd compressor—no `FlatBufferBuilder`: [compression.rs (lines 25–34)](https://github.com/apache/arrow-rs/blob/0ff81c1215cc026a1de93ce3d2078df1ecba6f09/arrow-ipc/src/compression.rs#L25-L34) - Comet does not enable Arrow IPC compression, so that compressor remains uninitialized: https://github.com/apache/datafusion-comet/blob/8c504872a1d215a6f454eba841969e5a3a85cd94/native/shuffle/src/writers/shuffle_block_writer.rs#L107-L110 - Arrow 58.4 independently constructs a new FlatBuffer builder for every batch: [writer.rs (lines 520–526)](https://github.com/apache/arrow-rs/blob/0ff81c1215cc026a1de93ce3d2078df1ecba6f09/arrow-ipc/src/writer.rs#L520-L526). - The quoted issue description in https://github.com/apache/datafusion-comet/pull/5006#discussion_r3640605171 matches Arrow 59.1’s newer `IpcWriteContext`, which owns the builder: [compression.rs (lines 27–38)](https://github.com/apache/arrow-rs/blob/b1de629ae82ee417e24a413f8f6815c884ec971b/arrow-ipc/src/compression.rs#L27-L38). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
