adriangb opened a new pull request, #11997: URL: https://github.com/apache/arrow-rs/pull/11997
# Which issue does this PR close? - Related to #11994 / #11996: this is a follow-up perf tweak, not the fix itself. # Rationale for this change `ByteBudgetChunker::byte_budget_sub_batch` calls `chunk_def.value_count(...)` before it asks the encoder whether the chunk's values fit the page budget. For nullable columns (`Materialized` def levels) that call is an O(chunk_size) scan, and it runs on every chunk of every nullable byte-array column, even chunks that are only a few KB and could never fill a page. # What changes are included in this PR? - New `ColumnValueEncoder::values_fit_within_byte_budget_gather`, an O(1)-only "definitely fits" check. It defaults to `false`, meaning unknown. - `ByteArrayEncoder` implements it with the stage-1 bounds that `count_values_within_byte_budget_gather` already uses: the offset-span subtraction for `Utf8`/`Binary` (and their `Large` variants), and the max-view-length bound for view arrays. Both bounds are moved into the helpers `offsets_span_fits` and `views_bound_fits`, and the existing counting code calls them, so behaviour there is unchanged. - The chunker checks the next `chunk_size` gathered values before running the def-level scan. A chunk holds at most `chunk_size` values, so this window is a superset of the chunk's values. If the superset fits, the chunk fits, and the chunker returns `None` without scanning def levels. # Are these changes tested? Covered by the existing writer and page-size tests: the check only adds an early `None` in cases where the existing path would also have returned `None`. **Benchmarks are still pending.** My local `arrow_writer` runs were taken while the machine had a load average above 30, and they swung from −11% to +160% between runs of the same benchmark, so they show nothing either way. Could someone run the benchmark bot on `arrow_writer`? I'm keeping this as a draft until we have clean numbers. # Are there any user-facing changes? No. The new trait method has a default, so existing `ColumnValueEncoder` implementations are unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
