tustvold commented on issue #5450: URL: https://github.com/apache/arrow-rs/issues/5450#issuecomment-1973984388
> Btw, can't we just explicitly enforce ArrowWriter to "flush" and start new row group right from AsyncArrowWriter in try_flush? 🤔 Yes, that is an option that is available to users, and with https://github.com/apache/arrow-rs/pull/5251 the necessary meta information is exposed to the clients to make this judgement for themselves. However, as this has come up a few times, providing a conservative default limit of say 1GB is probably a sane modification, users can then lower this if they're happy to accept the trade-off of smaller row groups -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
