thinkharderdev commented on issue #2430: URL: https://github.com/apache/arrow-rs/issues/2430#issuecomment-1219895080
Was tinkering around with this and we can go one of two ways: 1. Fetch all the index metadata when constructing the stream. 2. Fetch the index on demand as needed. The latter approach, we could add something to `AsyncFileReader` like ``` fn get_page_locations(&mut self, metadata: &ParquetMetadata, mask: &ProjectionMask) -> BoxFuture<'_, Result<Vec<Vec<PageLocation>>> ``` to only fetch what we need for a given projection. But that may be overkill given the amount of data.... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
