Tpt opened a new pull request, #12038: URL: https://github.com/apache/arrow-rs/pull/12038
Estimate row group boundaries from incoming values instead of extrapolating the average size of buffered rows. Calibrate logical input sizes against the current row group's estimated encoded size. Reuse page-sizing helpers for string and binary offsets and views, and account for nulls, slices, nested arrays, dictionaries, and run-end encoding. Bound input between checks while allowing a single compressible row past that cap when the calibrated budget permits it. # Which issue does this PR close? Follow-up to #11798 triggered by https://github.com/apache/arrow-rs/pull/11798#issuecomment-6353852081 # Rationale for this change Better estimation of row group size to avoid too large row groups. # Are there any significant user-facing changes? None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
