adriangb opened a new pull request, #11996: URL: https://github.com/apache/arrow-rs/pull/11996
# Which issue does this PR close? - Closes #11994. # Rationale for this change `SubBatchStrategy::Levels` turns a value budget into a level count using the chunk's *average* level:value ratio. That ratio says nothing about how many values a single window holds. If 28 large values come before 996 nulls, the first 37-level window contains all 28 values, so the dictionary page reaches about 2.2 GB before the spill check runs. That breaks the "at most two values per page" bound described in #10554. # What changes are included in this PR? - `SubBatchStrategy::Levels` now carries a `max_values` cap (twice the value budget) alongside the level count. - `write_granular_chunk` ends each level window at whichever comes first: the level count, or the point where `max_values` values have been seen. It reuses `window_end_for_values` to find that point. - When nulls are evenly spread the cap is never reached, so the number of mini-batches is unchanged. The throughput tradeoff from #10554 still holds. The added cost is one walk over the window's def levels, and only for `Materialized` def levels on the granular (large-value) path. # Are these changes tested? Yes. `test_column_writer_caps_dictionary_page_size_with_clustered_nulls` is a scaled-down version of the issue's reproducer: 28 distinct 64 KiB values, then 996 nulls, with a 64 KiB dictionary page limit. It fails without the cap (the dictionary page holds all 28 values) and passes with it. # Are there any user-facing changes? No API changes. Writers with clustered nulls and large values now produce bounded pages instead of failing with `Page uncompressed size overflow`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
