rustyconover commented on PR #11374: URL: https://github.com/apache/arrow-rs/pull/11374#issuecomment-6444436419
@Rich-T-kid added a benchmark for the complete in-memory stream use case in [0af11f357](https://github.com/apache/arrow-rs/pull/11374/commits/0af11f357f43d7c1112409779401e0f59666b634), under the existing `ipc_reader` target. It compares `StreamReader<Cursor<&[u8]>>`, `StreamDecoder`, `BufferStreamReader`, and `BufferStreamReader::try_new_from_bytes` over the same complete, uncompressed stream. It covers small and large Float64 batches, multiple batches, and the existing mixed-type workload (integers, strings with nulls, dates, decimals, and fixed-size binary). Measured locally on an Apple M3 with 24 GiB RAM, macOS 15.6.1, Rust 1.99.1. Criterion used 100 samples per case, a 3-second warm-up, and a 5-second measurement target. Times below are **microseconds per complete stream**, including reader construction: | Workload (rows per batch × batches) | StreamReader | StreamDecoder | BufferStreamReader | BufferStreamReader from Bytes | |---|---:|---:|---:|---:| | Float64: 1,024 × 1 | 1.078 | 0.742 | 0.670 | 0.715 | | Float64: 1,000,000 × 1 | 450.19 | 0.731 | 0.684 | 0.714 | | Float64: 10,000,000 × 1 | 9,921.26 | 0.726 | 0.669 | 0.716 | | Float64: 1,000,000 × 10 | 5,055.34 | 3.499 | 3.281 | 3.319 | | Mixed: 8,192 × 10 | 317.87 | 180.38 | 179.95 | 178.70 | Serialization and input allocation happen before timing. The input is aligned, default validation remains enabled, and every iteration decodes through EOS and drops the decoded batches. The owned-input readers include their reference-counted input clone in the timed loop. Outside timing, the benchmark checks that both new constructors return the expected batches and that their column buffers alias the input allocation. For the large numeric case, the difference comes from avoiding the body copy: zero-copy decoding builds array wrappers without scanning the numeric values. These are decode-only timings, not end-to-end application speedups. The mixed-type case still pays for string validation and improves by about **1.77×** over `StreamReader`; `BufferStreamReader` performs similarly to the existing `StreamDecoder` path. Reproduce with: ```sh cargo bench -p arrow-ipc --bench ipc_reader --features lz4,zstd -- in_memory ``` All 20 benchmark cases completed. The 14 targeted stream-reader tests, formatting, and Clippy also passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
