unikdahal opened a new issue, #3343: URL: https://github.com/apache/iceberg-rust/issues/3343
### What's the feature are you trying to implement? ## Summary Add generic execution-time predicate support to `iceberg-rust` so readers can progressively avoid file, row-group, page, and payload reads as predicates become available during query execution. The reader remains producer-agnostic: callers publish Iceberg predicates and generations, while `iceberg-rust` handles safe application through the existing reader pipeline. This epic tracks the upstream reader infrastructure required by execution engines such as DataFusion Comet. ## Related work - Runtime predicate discussion: https://github.com/apache/iceberg-rust/issues/2376 - Live predicate discussion: https://github.com/apache/iceberg-rust/issues/3336 - Arrow 60 upgrade: https://github.com/apache/iceberg-rust/pull/3257 ## Work items ### Dependencies - [ ] #3257 ### Reader foundations - [ ] Resolve unknown delete-file sizes lazily - [ ] Reuse the inclusive statistics evaluator with borrowed metrics - [ ] Add the generic runtime predicate publication API - [ ] Apply runtime predicates at file-task startup - [ ] Add opt-in file statistics retention to scan tasks ### Adaptive pruning - [ ] Reject file tasks before footer and delete-file I/O - [ ] Refresh predicates and prune unread row groups - [ ] Represent static page selections per row group - [ ] Preserve position and equality deletes during live pruning - [ ] Refresh page pruning and row filters after predicate updates - [ ] Preserve late materialization with runtime predicates ### Correctness and robustness - [ ] Bound refresh overhead and cover reader edge cases - [ ] Add adaptive pruning observability and reader-level metrics - [ ] Complete delete, split-task, schema-evolution, and fallback coverage ## Correctness principles - Runtime predicates are conservative restrictions. - Previously skipped data is never required by a later predicate publication. - Missing or unusable statistics retain candidates. - Runtime predicate failures must not disable planned filtering or delete handling. - Position deletes and metadata row positions preserve original file ordinals. - Equality deletes remain active across predicate refreshes. - Existing reader and schema adaptation paths should be reused rather than duplicated. ## Definition of done - [ ] No private Arrow/Parquet patch is required. - [ ] Generic task-start and live runtime predicate APIs are upstream. - [ ] Whole-file, row-group, page, and row-filter pruning use the existing reader pipeline. - [ ] Position deletes, equality deletes, and combined deletes are supported. - [ ] Stable predicates add minimal reader overhead. - [ ] Each work item above is linked to its corresponding issue or PR. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
