yonipeleg33 commented on PR #11798: URL: https://github.com/apache/arrow-rs/pull/11798#issuecomment-6357959212
> The fix here looks good to me. Writing one row first gives the average something to start from, and it is a small, safe change. Well, strictly speaking this approach doesn't fix the issue, only narrows it down; rows can have arbitrary length, so the arrow writer still writes an arbitrary amount of data before there are buffered rows. Not to say this pr isn't valid, as this isn't a regression either. Just pointing out this limitation. > A possible follow-up: the row group check could use the in-memory size of the incoming batch, so that it does not depend on the average of rows already buffered. The different writer implementations for the different language estimate sizes differently. Can't remember the details and I'm typing this from my phone, but I do recall some debate going on on the disagreements between the rust, cpp, and java impls. I don't think there's one strictly correct estimation, and AFAIK it's not specified in the protocol. All this to say - I don't have an opinion on what a "better" estimation might be. Whatever suites most use cases and is the most performant, I guess -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
