alamb commented on issue #2205: URL: https://github.com/apache/arrow-datafusion/issues/2205#issuecomment-1097902378
Thus, if I were doing this I would probably make the following three things 1. Add a "prefetch_hint" certain offsets to the `ObjectStore` API and make the parquet reader call it 2. Implement a "buffered" ObjectStore interface that wrapped another ObjectStore that prefetches and buffers data memory buffer 3. Implement a "cached" ObjectStore interface that also wraps another ObjectStore that simply downloads any request to a local disk cache With those components I think most usecases could be addressed and if someone needed custom caching logic they would likely get a good head start using the "buffered" or "cached" interfaces -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
