adriangb commented on PR #7997: URL: https://github.com/apache/arrow-rs/pull/7997#issuecomment-3423443894
> Another thing that I would like to do is to make it possible to buffer only some of the pages needed for a row group rather than all of them (aka what is stored in InMemoryRowGroup). This would reduce memory requirements for files with large row groups. However, it would also increase the number of IO requests (aka object store requests) so it would have to be configurable to let people trade off the IOs and the memory requirements IMO the parquet decoder should produce *as granular as possible* ranges of data to read and the object store implementation can handle coalescing them as needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
