adriangb commented on PR #11798: URL: https://github.com/apache/arrow-rs/pull/11798#issuecomment-6353852081
Thanks for the ping @alamb! But I did not write this part 😅. `max_row_group_bytes` limit came from #9357 by @yonipeleg33. My changes were to the data page byte budget in the column writer (#9972, #10554). That code splits each column's mini-batch on exact value sizes: math for fixed-width types and a running sum for variable-width types. The fix here looks good to me. Writing one row first gives the average something to start from, and it is a small, safe change. A possible follow-up: the row group check could use the in-memory size of the incoming batch, so that it does not depend on the average of rows already buffered. The complexities from from skewed row sizes, sliced views and memory vs encoded sizes. But maybe if we re-use some of the same tricks we used for page sizing we can get better estimates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
