mbutrovich opened a new issue, #3721: URL: https://github.com/apache/iceberg-rust/issues/3721
### Is your feature request related to a problem or challenge? [`ExpressionEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/expression_evaluator.rs#L46) checks a parsed `DataFile`'s partition values against the projected filter ([scan usage](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L681)). It can't filter the batches that `read_manifest_entries` returns without the slower `Manifest::parse_avro` decode. This is part of Task 4 of #3262. ### Describe the solution you'd like Add a partition mask method to the Arrow manifest evaluator in `iceberg::arrow`. It takes a manifest entry batch and the manifest's partition spec ID, and returns true where the entry's partition matches the filter projected to that spec. It evaluates the projection over the children of the `data_file.partition` struct column with arrow-rs kernels. A manifest has one spec, so one projection covers the whole batch. It reuses the literal and null helpers from the Parquet reader's `PredicateConverter` ([`constant_bool_array`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L255), [`compute_is_nan`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L281), [`try_cast_literal`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L647)), widened to `pub(crate)`. It doesn't generalize `PredicateConverter` itself. That type is tied to the Parquet schema, rejects struct columns, and runs on the data read path. Tests compare the mask with `ExpressionEvaluator` on the same manifests. They should cover null partition values, an unpartitioned spec, each transform's result type, and v1 and v2 manifests. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
