mbutrovich opened a new issue, #3722: URL: https://github.com/apache/iceberg-rust/issues/3722
### Is your feature request related to a problem or challenge? [`InclusiveMetricsEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs#L45-L57) reads column metrics from a parsed `DataFile` ([scan usage](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L686-L692)). It can't filter the batches that `read_manifest_entries` returns without the slower `Manifest::parse_avro` decode. This is part of Task 4 of #3262. ### Describe the solution you'd like Add a metrics mask method to the Arrow manifest evaluator in `iceberg::arrow`. It takes a manifest entry batch and returns true where the entry's metrics allow a match, with the same semantics as `InclusiveMetricsEvaluator`, including `include_empty_files` from the builder. It's separate from the partition mask because PyIceberg skips the metrics filter for delete manifests when it ignores residuals ([`table/__init__.py#L2947-L2949`](https://github.com/apache/iceberg-python/blob/9955502c73758611d5ad69220b67aedb3d4a443e/pyiceberg/table/__init__.py#L2947-L2949)). For each field ID the filter references, it extracts the matching values from the `value_counts`, `null_value_counts`, `nan_value_counts`, `lower_bounds`, and `upper_bounds` map columns once per batch. It decodes bounds from their [binary single-value serialization](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/manifest.rs#L66-L67) the same way as the manifest list mask. Tests compare the mask with `InclusiveMetricsEvaluator` on the same manifests. They should cover entries without metrics (files added from non-Iceberg writers often have none), a field missing from some entries' maps, truncated string and binary bounds, float and double bounds with NaN counts, decimal bounds, `record_count` of 0, and nested fields. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
