mbutrovich opened a new issue, #3720: URL: https://github.com/apache/iceberg-rust/issues/3720
### Is your feature request related to a problem or challenge? [`ManifestEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/manifest_evaluator.rs#L83) takes a parsed `ManifestFile`, so it can't filter the batches that `read_manifest_list` returns without the slower `ManifestList::parse_with_version` decode. This is part of Task 4 of #3262, and #3649 needs it. ### Describe the solution you'd like Add a public evaluator type to `iceberg::arrow` that returns a `BooleanArray` mask for a manifest list batch. The mask is true where the manifest may contain matching rows, with the same semantics as `ManifestEvaluator`. <partition> and <metrics> add masks for manifest entry batches to the same type. It lives in its own module next to `arrow/manifest.rs` and is re-exported from [`arrow/mod.rs`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/mod.rs#L31-L34) like the readers whose batches it reads. Predicate evaluation over Arrow batches already lives in `iceberg::arrow` ([`PredicateConverter`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L199)). It's built once per scan with a `TypedBuilder` from the table schema, the partition specs, the row filter as an unbound `Predicate`, case sensitivity (default true), and `include_empty_files` (default false). These match what [`TableScanBuilder`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L206-L215) takes. `build` is fallible. It binds the filter the way the scan does ([`bind_scan_predicate`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L143-L152)) and projects it for every spec up front, so evaluation needs no locks or caches. A later option is a new builder field with a default, which doesn't break callers. The methods return masks, not filtered batches. That way callers combine them with their own conditions, such as entry status or sequence numbers, using arrow-rs kernels. The mask for manifest lists, the partition mask, and the metrics mask are separate methods for the same reason: PyIceberg applies the metrics mask only to some manifests. Internally the evaluator implements [`BoundPredicateVisitor`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs#L25) with `type T = BooleanArray`, as `page_index_evaluator` does with [`RowSelection`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/page_index_evaluator.rs#L606). It reuses [`CollectFieldIdVisitor`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L44) to decode each referenced field once per batch, and [`get_arrow_datum`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/schema.rs#L1002) for literals. The per-spec projection in [`scan/cache.rs`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/cache .rs#L83-L107), including the dropped-source-column case, moves into a shared `pub(crate)` function so the scan and the evaluator can't diverge. Rows in one batch can have different `partition_spec_id` values, so the evaluator evaluates each spec's rows with that spec's projection. Summary bounds stay in [binary single-value serialization](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/manifest.rs#L221-L222). The evaluator decodes them straight into typed Arrow arrays, not through a `Datum` per value. String, binary, fixed, and uuid bounds [compare bytewise](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L161-L168), so they need no decoding. Tests compare the mask with `ManifestEvaluator` on the same manifest lists. They should cover v1 and v2 lists, several specs in one batch, a spec whose source column was dropped, null `partitions`, missing bounds, `contains_null`, `contains_nan`, and an empty list. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
