tustvold commented on PR #1774: URL: https://github.com/apache/arrow-rs/pull/1774#issuecomment-1147310698
> Maybe this interface would help me simplify things It would theoretically allow for more complex heuristics, but ultimately it is just a different way to access the data returned by row_group_writer.close() > Or is there a way to be more "precise" in the resulting file size Not easily, it is very hard to predict the encoded size accurately, especially with block compression. The way this is handled internally for pages is writing in small batches, and tracking the size as they go. This is much the same as your approach for row groups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
