unikdahal opened a new issue, #3350: URL: https://github.com/apache/iceberg-rust/issues/3350
Part of #3343. ### What's the feature are you trying to implement? Execution engines learn restrictions while a query runs: the key range of a completed hash-join build, an improving TopK threshold, a running MIN/MAX bound. Today `ArrowReader` only applies the predicate fixed in each `FileScanTask`, so an engine cannot use these restrictions to skip row groups, pages or rows in files it has not read yet. Proposal: let the engine hand such restrictions to the reader as ordinary Iceberg predicates. The reader stays producer-agnostic and applies them through its existing pruning paths. **API (reader layer only)** - `RuntimePredicateProvider`, with a cheap `generation()` for change checks and `snapshot()` for the current predicate. - `RuntimePredicateSnapshot`, a predicate (or `None`) paired with its generation. - `ArrowReaderBuilder::with_runtime_predicate_provider`. Without a provider, behaviour is unchanged. `TableScan::to_arrow` does not accept a provider; wiring it through `TableScan` and engines is a follow-up. **Behaviour** - When a data-file task starts, the reader checks the generation. Only on a change does it take a snapshot, push `NOT` to the leaves and bind the predicate. Refreshes are serialized and the bound predicate is shared by later tasks with the same generation, schema and case policy. The cache never moves to an older generation. - The runtime predicate is planned separately from the task and equality-delete predicates, then combined with them into one Arrow row filter, so shared columns are decoded once. Row-group, page-index and bloom-filter pruning intersect both predicates, and a bloom filter is read at most once per row group and column. - Positional deletes keep their original file positions and equality deletes stay applied. - Failures are advisory. A provider, snapshot, binding, planning or page-pruning failure drops only the runtime part for that file and leaves planned filtering and deletes unchanged. Errors while decoding the predicate's columns or evaluating it on rows are not skipped. - The runtime predicate is skipped for a file that lacks a referenced column (its initial default or null is invisible to physical filters) or stores it with a type that needs promotion (INT to BIGINT, FLOAT to DOUBLE, wider DECIMAL), because a literal cast before promotion can round or overflow and reject valid rows. **Provider contract** - Data skipped under a published predicate must never be required by a later publication, so later publications may tighten but not loosen it. A later `None` only stops runtime pruning for tasks that start afterwards; it does not revoke what earlier tasks captured or restore skipped data. - Generations increase monotonically, are never reused, and become visible only after their predicate. - `snapshot()` returns a self-consistent predicate and generation, not older than one `generation()` already returned. **Measured cost** (2M rows, row-group filtering off): a first version evaluated the runtime predicate as a second Arrow predicate. That was up to about 2.4x slower than planned-only when it removed few rows, because shared columns were decoded twice. The merged filter measures 0.47x-0.99x of planned-only for selective runtime predicates and at worst 1.37x otherwise. Both predicates are evaluated on every row of a batch. Needs nothing beyond the dependencies on `main`; it does not depend on the Arrow upgrade tracked in #3257. This covers the "runtime predicate publication API" and "apply runtime predicates at file-task startup" items of #3343 in one change, since the API has no use without its application. ### Willingness to contribute I can contribute to this feature independently; an implementation with tests is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
