Tpt commented on PR #11798: URL: https://github.com/apache/arrow-rs/pull/11798#issuecomment-6373201659
Thank you for the feedbacks! > Well, strictly speaking this approach doesn't fix the issue, only narrows it down; Yes! Agreed. > A possible follow-up: the row group check could use the in-memory size of the incoming batch, so that it does not depend on the average of rows already buffered. The complexities from from skewed row sizes, sliced views and memory vs encoded sizes. But maybe if we re-use some of the same tricks we used for page sizing we can get better estimates? Thank you! This sounds like a great idea. I plan to open a new PR going into this direction but likely not before a couple of weeks. Glad to either close this PR or see it merged in the meantime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
