Perseus opened a new issue, #3675: URL: https://github.com/apache/iceberg-rust/issues/3675
### Is your feature request related to a problem or challenge? `iceberg-rust` already performs many levels of data pruning - skipping files, row groups etc. The predicates used to do this are collected when a scan is being built, and they remain static through the entire execution pipeline. Query engines, like DataFusion, can gather information during a query's execution that can be used to further filter out data being scanned from Iceberg (aka dynamic filtering). However, `iceberg-rust` doesn't have a way for the engine to pass in this information at runtime. Additionally, depending on how the engine plans a scan, it also needs the ability to reuse logic implemented within `iceberg-rust` to skip already-planned file scans based on partition column values that became available while executing the query. There is some work ongoing here - https://github.com/apache/iceberg-rust/issues/3343 , which plans to implement a general runtime/adaptive pruning system. ### Describe the solution you'd like I would like to propose a smaller set of changes that allow consumers of the library to use the partition pruning logic that is currently private. This would allow dropping a `FIleScanTask` based on predicates that became available during query execution. ## Approach 1 Make `ExpressionEvaluator` and `InclusiveProjection` public. These can be used to implement much of the partition-column based `FileScanTask` skipping logic. There is a draft PR [here](https://github.com/datafusion-contrib/datafusion-distributed/pull/869/changes#diff-20f4b0659238d6ef6df8a7f9accc51823727488a229a673d4a6e640f680da836R143) that copies these functioons verbatim to implement this pruning in DataFusion. However, this approach can be brittle - creating external dependencies on internal functions that might be tied to current structure and make it difficult to refactor at a later point. ## Approach 2 Create a public `PartitionFilter` utility which performs the same work - evaluates whether a `FileScanTask` can be skipped based on the given partition-bound predicate - but has its internals abstracted away from the user. I have a draft version of this approach [here](https://github.com/apache/iceberg-rust/pull/3674/changes#diff-aff19163a9618fb8e3acc6f9bbc62b8a5b8d39b58f31c52a0d52e46de93cc388) - the set of changes are fairly small. There might be other approaches that I am missing due to lack of context, and I'm happy to pursue any path that the maintainers feel comfortable with - the goal is to make it possible for query engines to make scan decisions at runtime ### Willingness to contribute I would be willing to contribute to this feature with guidance from the Iceberg Rust community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
