eneskeles opened a new issue, #2131:
URL: https://github.com/apache/iceberg-go/issues/2131
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Came across this one while working on #2111
A filter like `NOT(x < 5)` on a float column sometimes matches a NaN row and
sometimes does not depending on the other values in the same file.
Scan with `NOT(x < 5)`, one data file per table:
| `x` values in the file | rows returned |
|---|---|
| `[10, NaN, 20]` | all three, with the NaN row |
| `[1, NaN, 2]` | none |
`Delete` with the same filter does the same thing: it removes the NaN row
from the first file and keeps it in the second. I see this with both
copy-on-write and merge-on-read, on main (0b0828b).
### Why
The filter is checked in two different forms:
- The row filter runs it as written. `NaN < 5` is false, so `NOT(NaN < 5)`
is true and the NaN row matches.
- The stats checks first call `RewriteNotExpr`, which changes it to `x >=
5`. That is false for NaN. Min/max stats do not include NaN, so the file `[1,
NaN, 2]` is skipped before the row filter runs.
### Repro
```go
filter := iceberg.NewNot(iceberg.LessThan(iceberg.Reference("x"), 5.0))
// tblHigh has one file with x = [10, NaN, 20]
// tblLow has one file with x = [1, NaN, 2]
high, _ := tblHigh.Scan(table.WithRowFilter(filter)).ToArrowTable(ctx)
low, _ := tblLow.Scan(table.WithRowFilter(filter)).ToArrowTable(ctx)
fmt.Println(high.NumRows()) // 3
fmt.Println(low.NumRows()) // 0
```
### Expected
The NaN row should match or not match the same way in both files.
I think Java rewrites NOT before it evaluates the filter, so `NOT(x < 5)`
never matches NaN there. Doing the same here would mean calling
`RewriteNotExpr` before the row filter is built too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
