eneskeles opened a new issue, #2110:
URL: https://github.com/apache/iceberg-go/issues/2110
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
A copy-on-write `Delete` (or a filtered `Overwrite`) also deletes rows where
the filter column is NULL, even though those rows don't match the filter.
Copy-on-write is the default `write.delete.mode`, so this happens with
default settings. The commit succeeds and nothing reports an error. The rows
are lost permanently once the old snapshot expires.
### Repro
```go
// table: id long, age long (nullable), default properties
// rows: {1, 25}, {2, 30}, {3, NULL}, {4, 40}
txn.Delete(ctx, iceberg.EqualTo(iceberg.Reference("age"), int64(30)), nil)
// expected rows: 1, 3, 4
// actual rows: 1, 4 <- id=3 (age=NULL) was deleted too
```
`age > 35` behaves the same way. It happens on v2 and v3. Merge-on-read is
not affected.
### Cause
The rows to keep are selected with `NOT(filter)`:
https://github.com/apache/iceberg-go/blob/fa5309b/table/transaction.go#L2826
For `v = NULL`, `v = 5` evaluates to NULL, and `NOT(NULL)` is also NULL, not
true. So the row is not kept.
### Possible fix
Keep the rows where the filter is **not true**: evaluate the filter, treat
NULL as false, then negate. PyIceberg does this in
`_expression_to_complementary_pyarrow`.
### Other implementations
This is iceberg-go specific:
- **PyIceberg** handles it explicitly:
`_expression_to_complementary_pyarrow` exists because `~` in pyarrow "does not
handle null".
- **Spark (Java)** keeps the remaining rows with `NOT (cond <=> true)` in
`RewriteDeleteFromTable`, so NULL rows survive.
- **Java core** doesn't rewrite rows; `DeleteFiles` only removes whole files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
