cetra3 opened a new pull request, #11149: URL: https://github.com/apache/arrow-rs/pull/11149
# Which issue does this PR close? - Closes #11148. # Rationale for this change `DeltaBitPackEncoder::new` preallocates a 1MiB buffer for its bit writer. Byte array columns eagerly construct their fallback encoder, and the `DELTA_BYTE_ARRAY` fallback (the default for `PARQUET_2_0`) holds two `DeltaBitPackEncoder`s, so every byte array column pays 2MiB of heap up front, even when it dictionary encodes and the fallback is never used. With 100 columns that is ~200MiB. The preallocation only ever helped the first page: the buffer is a `Vec` that retains its capacity across `clear()`. # What changes are included in this PR? - `DeltaBitPackEncoder` starts with an empty bit writer buffer that grows on demand (removes `DEFAULT_BIT_WRITER_SIZE`). This also benefits `DELTA_BINARY_PACKED` and `DELTA_LENGTH_BYTE_ARRAY`. - Regression test `unused_delta_fallback_does_not_preallocate` in `parquet/tests/arrow_writer`, using the existing peak heap tracking allocator. It fails on `main` (peak ≈ 210MB) and passes with this change. # Are these changes tested? Yes, with the new regression test above. The existing lib and `arrow_writer` tests pass. Benchmarks (criterion, `--save-baseline` on `main` vs this branch): `arrow_writer` delta byte array: | benchmark | change | |---|---| | small_string_shared_prefix/delta_byte_array | −3.8% | | small_string_partial_prefix/delta_byte_array | −6.6% | | small_string_distinct/delta_byte_array | −15.6% | | large_string_shared_prefix/delta_byte_array | −3.7% | | large_string_distinct/delta_byte_array | −10.4% | | large_string_shared_prefix_nullable/delta_byte_array | −6.2% | | large_string_shared_prefix_nullable_dense/delta_byte_array | −15.4% | | large_string_shared_prefix_nullable_trailing/delta_byte_array | −1.4% | | large_string_distinct_nullable/delta_byte_array | −13.0% | | medium_string_shared_prefix_nullable/delta_byte_array | −3.2% | | large_string_shared_prefix_list/delta_byte_array | −8.1% | | string/parquet_2 (dictionary overflow → delta fallback, 3 runs) | −3.6% to −4.4% | `writer_overhead`: | benchmark | change | |---|---| | 1000_cols | no change | | 5000_cols | −2.6% | | 10000_cols | −2.7% | | 1000_cols/repeated_batches | −0.6% (noise) | | 5000_cols/repeated_batches | −14.4% | The other `parquet_2` / `zstd_parquet_2` writer benchmarks are within noise after re-running. The one exception is `int32_ree_95pct_null/parquet_2`, which measured +2.6% to +5.1% across runs. However, `main` measured against its own baseline drifted +0.9% to +2.7% on this benchmark, and it does not exercise the delta encoder, so the difference looks like noise. # Are there any user-facing changes? No API changes. Lower peak memory when writing many byte array columns with `PARQUET_2_0` / delta encodings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
