rich7420 opened a new issue, #3494:
URL: https://github.com/apache/iceberg-rust/issues/3494
### Apache Iceberg Rust version
main (a9bd6c6)
### Describe the bug
`RowGroupMetricsEvaluator::visit_inequality` checks only the literal for
NaN. If a row group's min statistic is NaN, `f < 5.0` and `f <= 5.0` prune it
even when it holds smaller values. parquet-mr 1.9 and earlier record a NaN that
starts a row group as both min and max, and `add_files` or `migrate` can bring
such files into a table.
`eq` and `in` in the same evaluator already treat a NaN bound as
might-match. On the Java side, parquet-java's reader drops float and double
min/max statistics that contain NaN, so `ParquetMetricsRowGroupFilter` never
sees them.
### To Reproduce
In the `row_group_metrics_evaluator.rs` tests:
```rust
let row_group_metadata = create_row_group_metadata(
1, 1,
Some(Statistics::float(Some(f32::NAN), Some(f32::NAN), None, Some(0),
false)),
1, None,
)?;
let (schema, field_id_map) = build_iceberg_schema_and_field_map()?;
let filter = Reference::new("col_float")
.less_than(Datum::float(5.0_f32))
.bind(schema.clone(), false)?;
let might_match =
RowGroupMetricsEvaluator::eval(&filter, &row_group_metadata,
&field_id_map, schema.as_ref())?;
```
`might_match` is false, and the same for `less_than_or_equal_to`.
`equal_to(1.0)` on the same statistics returns true.
### Expected behavior
`visit_inequality` returns might-match for a NaN bound, as `eq` and `in` do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
