rizaon opened a new issue, #4508: URL: https://github.com/apache/iceberg/issues/4508
Iceberg planFiles API has benefit of filtering out unnecessary files given a set of filter predicates. During query planning, a query engine might need to call planFiles against the same table multiple times but with slightly different filter predicates for each scan. This lead to repeated manifest files reading over the same table. It is often expensive, especially if it involve remote reads through network. Certain stream operation such as backward seek and small reads characteristics by Avro library can have high latency depending on the target file system / cloud connector. Having ability to cache manifest content in memory for several planFiles calls can improve performance. One idea is to add method in TableScanContext.java to supply a caffeine cache, say `Cache<String, byte[]>`, and pass it further to AvroIO.java. In turn, AvroIO can consult with this cache and use the cached manifest file content if it is available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
