mbutrovich opened a new issue, #3722:
URL: https://github.com/apache/iceberg-rust/issues/3722

   ### Is your feature request related to a problem or challenge?
   
   
[`InclusiveMetricsEvaluator::eval`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs#L45-L57)
 reads column metrics from a parsed `DataFile` ([scan 
usage](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L686-L692)).
 It can't filter the batches that `read_manifest_entries` returns without the 
slower `Manifest::parse_avro` decode. This is part of Task 4 of #3262.
   
   ### Describe the solution you'd like
   
   Add a metrics mask method to the Arrow manifest evaluator in 
`iceberg::arrow`. It takes a manifest entry batch and returns true where the 
entry's metrics allow a match, with the same semantics as 
`InclusiveMetricsEvaluator`, including `include_empty_files` from the builder. 
It's separate from the partition mask because PyIceberg skips the metrics 
filter for delete manifests when it ignores residuals 
([`table/__init__.py#L2947-L2949`](https://github.com/apache/iceberg-python/blob/9955502c73758611d5ad69220b67aedb3d4a443e/pyiceberg/table/__init__.py#L2947-L2949)).
   
   For each field ID the filter references, it extracts the matching values 
from the `value_counts`, `null_value_counts`, `nan_value_counts`, 
`lower_bounds`, and `upper_bounds` map columns once per batch. It decodes 
bounds from their [binary single-value 
serialization](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/arrow/manifest.rs#L66-L67)
 the same way as the manifest list mask.
   
   Tests compare the mask with `InclusiveMetricsEvaluator` on the same 
manifests. They should cover entries without metrics (files added from 
non-Iceberg writers often have none), a field missing from some entries' maps, 
truncated string and binary bounds, float and double bounds with NaN counts, 
decimal bounds, `record_count` of 0, and nested fields.
   
   ### Willingness to contribute
   
   I can contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to