anoopj opened a new issue, #3415:
URL: https://github.com/apache/iceberg-rust/issues/3415
### Apache Iceberg Rust version
main@b154603 (latest code)
### Describe the bug
If a FileScanTask carries a predicate with a `NOT` node, the Arrow reader
mishandles it in two of its pruning stages.
The reader hands the task predicate to the row-group metrics evaluator and
the page-index evaluator. Both are conservative (might-match) filters, and
neither can soundly handle a NOT:
- Row-group pruning negates the inner result `(!inner)`. That flips
`MIGHT_MATCH` to `CANT_MATCH` and prunes a row group that actually matches.
This will cause a silent daa loss.
- Page-index pruning returns an error on a NOT node, which aborts the whole
read.
Example: `NOT(x > 5)` means `x <= 5`. On a row group spanning `x` in `[0,
10]`, the `inner x > 5` is `MIGHT_MATCH`, so row-group pruning drops the group
even though rows with `x <= 5` are in it.
A NOT reaches the reader when a FileScanTask is built directly or
deserialized without going through TableScan (for example a scan plan sent to a
distributed worker). The normal TableScan path rewrites NOT away in
with_filter, so this doesn't show up there.
### To Reproduce
```rust
let predicate = (!Reference::new("x").greater_than(Datum::int(5)))
.bind(schema, false).unwrap(); // BoundPredicate::Not(...)
let task = FileScanTask::builder().with_predicate(Some(predicate)) /* ... */
.build().unwrap();
// row-group filtering (on by default): rows with x <= 5 go missing from the
result
// with_row_selection_enabled(true) on a file with page indexes: read fails
with
// "NOT unsupported at this point. NOT-rewrite should be performed first"
```
### Expected behavior
Expected: a NOT predicate returns the same rows as its rewritten form (NOT(x
> 5) returns rows with x <= 5), with no error.
Fix: normalize the predicate with rewrite_not once before the pruners run,
the same thing TableScan::with_filter already does.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
