toutane opened a new issue, #2963: URL: https://github.com/apache/iceberg-rust/issues/2963
### Is your feature request related to a problem or challenge? Eager scan planning introduced by #2671 calls `plan_files()` during `TableProvider::scan()`. This includes reading manifests, binding predicates, resolving the snapshot schema, and associating applicable delete files with each FileScanTask. If DataFusion requests the same scan multiple times during optimization, this work is repeated. The resulting eager plan is retained only by the returned `IcebergTableScan` and cannot be reused by a future equivalent `scan()` call. This limitation was originally discussed in #2298: https://github.com/apache/iceberg-rust/pull/2298#discussion_r3404722001 We should keep in mind that there are two provider types: - `IcebergStaticTableProvider` uses a fixed table and snapshot, making equivalent scans suitable for reuse. - `IcebergTableProvider` must continue loading fresh table metadata so that newly committed snapshots remain visible. ### Describe the solution you'd like Reuse eager scan planning results across equivalent `TableProvider::scan()` calls. A cached result must only be reused when every input affecting file planning is equivalent. This includes at least: - the concrete snapshot ID; - the projection; - the pushed-down predicate. For the catalog-backed `IcebergTableProvider`, the table must still be reloaded before a cached result can be reused. The concrete snapshot ID resolved from the refreshed table must participate in cache lookup so that a new snapshot naturally causes a cache miss. Two caching strategies should be considered: #### Option 1: Cache the complete `EagerScanPlan` Cache the tasks after they have been grouped into DataFusion output partitions. #### Option 2: Cache the planned tasks before grouping Cache the result of Iceberg file planning before assigning the `FileScanTask`s to DataFusion output partitions. Perform the relatively inexpensive grouping after cache lookup. In this design, `target_partitions` does not need to be part of the cache key. ### Willingness to contribute I would be willing to contribute to this feature with guidance from the Iceberg Rust community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
