mbutrovich opened a new issue, #2780: URL: https://github.com/apache/iceberg-rust/issues/2780
## Apache Iceberg Rust version `main` (commit `148afc5`; crate `iceberg` 0.9.0-dev). ## Describe the bug An equality delete keyed on a nullable column incorrectly deletes data rows whose value in that equality column is `null`. Per the Iceberg spec, [Equality Delete Files](https://iceberg.apache.org/spec/#equality-delete-files): *"A `null` value in a delete column matches a row if the row's value is `null`."* A **non-null** delete value therefore does not match a `null` data value, so such rows must be **kept**. `CachingDeleteFileLoader::parse_equality_deletes_record_batch_stream` (`crates/iceberg/src/arrow/caching_delete_file_loader.rs`) builds each delete row's "keep" predicate by negating the equality match (`equal_to(v).not()` → `col != v`). Applied as a row filter, `null != v` evaluates to `null` and the row is dropped, so rows with a `null` equality-column value are silently deleted. For a delete value `status = "INACTIVE"`, the built predicate is: ``` status != "INACTIVE" ``` which drops rows where `status IS NULL`. It should be: ``` (status IS NULL) OR (status != "INACTIVE") ``` This surfaces under schema evolution: rows written before a column was added read back as `null` in that column and are wrongly deleted by an equality delete keyed on it. ## To Reproduce Found while adding Apache Iceberg 1.11 support to Apache DataFusion Comet. It originates from Iceberg-Java's `org.apache.iceberg.spark.source.TestSparkReaderDeletes.testEqualityDeleteWithSchemaEvolution`, which passes in Iceberg-Java but returns wrong results when the scan runs through iceberg-rust. Reproduced directly in iceberg-rust with a unit test on `parse_equality_deletes_record_batch_stream`: feed a single-column equality delete (`status = "INACTIVE"`) and inspect the resulting keep predicate. - Actual: `status != "INACTIVE"` (drops rows where `status` is null) - Expected: `(status IS NULL) OR (status != "INACTIVE")` I have the unit test and a fix ready to submit as a PR. ## Expected behavior Rows whose equality-column value is `null` are not deleted by an equality delete that has a non-null value in that column (per the spec). The keep predicate should be `col IS NULL OR col != v` for a non-null delete value, and `col IS NOT NULL` for a null delete value. ## Willingness to contribute I can contribute a fix for this bug independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
