mbutrovich opened a new issue, #3721:
URL: https://github.com/apache/iceberg-rust/issues/3721

   ### Is your feature request related to a problem or challenge?
   
   
[`ExpressionEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/expression_evaluator.rs#L46)
 checks a parsed `DataFile`'s partition values against the projected filter 
([scan 
usage](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L681)).
 It can't filter the batches that `read_manifest_entries` returns without the 
slower `Manifest::parse_avro` decode. This is part of Task 4 of #3262.
   
   ### Describe the solution you'd like
   
   Add a partition mask method to the Arrow manifest evaluator in 
`iceberg::arrow`. It takes a manifest entry batch and the manifest's partition 
spec ID, and returns true where the entry's partition matches the filter 
projected to that spec. It evaluates the projection over the children of the 
`data_file.partition` struct column with arrow-rs kernels. A manifest has one 
spec, so one projection covers the whole batch.
   
   It reuses the literal and null helpers from the Parquet reader's 
`PredicateConverter` 
([`constant_bool_array`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L255),
 
[`compute_is_nan`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L281),
 
[`try_cast_literal`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L647)),
 widened to `pub(crate)`. It doesn't generalize `PredicateConverter` itself. 
That type is tied to the Parquet schema, rejects struct columns, and runs on 
the data read path.
   
   Tests compare the mask with `ExpressionEvaluator` on the same manifests. 
They should cover null partition values, an unpartitioned spec, each 
transform's result type, and v1 and v2 manifests.
   
   ### Willingness to contribute
   
   I can contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to