adriangb opened a new pull request, #10745: URL: https://github.com/apache/arrow-rs/pull/10745
## What `compute_min_max` in `arrow_writer/byte_array.rs` copies both the minimum and the maximum of every mini-batch into fresh `ByteArray`s (`to_vec()`) before `encode` has checked whether either one beats the running page min/max. `write_gather` runs once per mini-batch, and a byte-budgeted mini-batch of large values can hold a single value — so every large value was being allocated and copied twice more, on top of the copies the encoder itself needs. This returns the borrowed extremes and copies only when the running value actually changes. Statistics are unchanged; the comparison is the same byte-lexicographic order `ByteArray` uses. ## Why now Found while profiling #10554 (value-exact mini-batch windows), which showed **+12%** on `large_string_distinct_nullable/delta_byte_array` versus main across three runner runs. `samply` put the whole shift into `memmove` under `ByteArrayEncoder::write_gather`; the encoding work per value is identical, so the cost had to be per-mini-batch — and #10554 roughly doubles the mini-batch count on nullable columns. That was this copy. ## Measurements Local (Apple M4 Pro), two rounds each, three binaries: **A** = #10505 head, **B** = #10554 head, **B′** = #10554 + this change: | bench (ms) | A | B | B′ | |---|---|---|---| | `large_string_distinct_nullable/delta_byte_array` | 23.1 / 24.1 | 24.7 / 24.4 | **22.9 / 23.1** | | `large_string_shared_prefix_nullable/delta_byte_array` | 24.6 / 22.8 | 25.0 / 24.5 | **20.3 / 20.9** | | `large_string_shared_prefix_nullable_dense/delta_byte_array` | 13.4 / 13.6 | 13.2 / 13.1 | **11.0 / 11.1** | | `large_string_shared_prefix_nullable_trailing/delta_byte_array` | 33.2 / 33.7 | 24.9 / 24.5 | **20.5 / 20.9** | | `large_string_shared_prefix/delta_byte_array` (non-null) | 26.0 / 26.0 | 26.0 / 26.6 | **21.6 / 22.0** | | `medium_string_shared_prefix_nullable/delta_byte_array` | 18.9 / 19.0 | 18.9 / 19.2 | **18.1 / 18.3** | It removes #10554's cost entirely and is a ~15–17% win on large-value `DELTA_BYTE_ARRAY` columns on its own, so it stands alone against `main`. Runner numbers (×3 plus a main-vs-main control) to follow in a comment. ## Tests No behaviour change; full `parquet` suite green (1308 lib + integration), `fmt` and `clippy -D warnings` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
