mbutrovich opened a new issue, #3720:
URL: https://github.com/apache/iceberg-rust/issues/3720

   ### Is your feature request related to a problem or challenge?
   
   
[`ManifestEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/manifest_evaluator.rs#L83)
 takes a parsed `ManifestFile`, so it can't filter the batches that 
`read_manifest_list` returns without the slower 
`ManifestList::parse_with_version` decode. This is part of Task 4 of #3262, and 
#3649 needs it.
   
   ### Describe the solution you'd like
   
   Add a public evaluator type to `iceberg::arrow` that returns a 
`BooleanArray` mask for a manifest list batch. The mask is true where the 
manifest may contain matching rows, with the same semantics as 
`ManifestEvaluator`. <partition> and <metrics> add masks for manifest entry 
batches to the same type.
   
   It lives in its own module next to `arrow/manifest.rs` and is re-exported 
from 
[`arrow/mod.rs`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/mod.rs#L31-L34)
 like the readers whose batches it reads. Predicate evaluation over Arrow 
batches already lives in `iceberg::arrow` 
([`PredicateConverter`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L199)).
   
   It's built once per scan with a `TypedBuilder` from the table schema, the 
partition specs, the row filter as an unbound `Predicate`, case sensitivity 
(default true), and `include_empty_files` (default false). These match what 
[`TableScanBuilder`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L206-L215)
 takes. `build` is fallible. It binds the filter the way the scan does 
([`bind_scan_predicate`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L143-L152))
 and projects it for every spec up front, so evaluation needs no locks or 
caches. A later option is a new builder field with a default, which doesn't 
break callers.
   
   The methods return masks, not filtered batches. That way callers combine 
them with their own conditions, such as entry status or sequence numbers, using 
arrow-rs kernels. The mask for manifest lists, the partition mask, and the 
metrics mask are separate methods for the same reason: PyIceberg applies the 
metrics mask only to some manifests.
   
   Internally the evaluator implements 
[`BoundPredicateVisitor`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs#L25)
 with `type T = BooleanArray`, as `page_index_evaluator` does with 
[`RowSelection`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/page_index_evaluator.rs#L606).
 It reuses 
[`CollectFieldIdVisitor`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L44)
 to decode each referenced field once per batch, and 
[`get_arrow_datum`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/schema.rs#L1002)
 for literals. The per-spec projection in 
[`scan/cache.rs`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/cache
 .rs#L83-L107), including the dropped-source-column case, moves into a shared 
`pub(crate)` function so the scan and the evaluator can't diverge.
   
   Rows in one batch can have different `partition_spec_id` values, so the 
evaluator evaluates each spec's rows with that spec's projection. Summary 
bounds stay in [binary single-value 
serialization](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/manifest.rs#L221-L222).
 The evaluator decodes them straight into typed Arrow arrays, not through a 
`Datum` per value. String, binary, fixed, and uuid bounds [compare 
bytewise](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L161-L168),
 so they need no decoding.
   
   Tests compare the mask with `ManifestEvaluator` on the same manifest lists. 
They should cover v1 and v2 lists, several specs in one batch, a spec whose 
source column was dropped, null `partitions`, missing bounds, `contains_null`, 
`contains_nan`, and an empty list.
   
   ### Willingness to contribute
   
   I can contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to