LykoMante opened a new pull request, #11995: URL: https://github.com/apache/arrow-rs/pull/11995
# Which issue does this PR close? Closes #11994. # Rationale for this change When dictionary encoding is enabled on a nullable column where non-null values are clustered at the front of a batch followed by trailing nulls, ratio-based level scaling (`SubBatchStrategy::Levels`) rounds up to a level window that can encompass all non-null values in a single mini-batch before the dictionary fallback check is evaluated. For large values, this causes dictionary pages to overshoot configured byte limits and can trigger a 2 GiB page overflow. # What changes are included in this PR? - Selects `SubBatchStrategy::Values` for dictionary pages when values exceed the remaining dictionary budget, cutting sub-batches on value boundaries. - Adds regression test `test_column_writer_caps_dictionary_page_size_with_clustered_nulls` verifying dictionary page sizes remain bounded under clustered null distributions. # Are these changes tested? Yes, verified with the new regression test and the existing column writer suite. # Are there any significant user-facing changes? No API changes. Prevents memory bloat and page size overflow when writing batches with clustered nulls. Signed-off-by: Garrett Miller <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
