Hi everyone, I’d like to bring up a topic currently being discussed in PR https://github.com/apache/iceberg/pull/18338 to the dev list to get broader community input.
*Background* As a quick refresher, the PR addresses a *silent correctness bug* regarding equality delete files. Currently, when equality delete files are written with the full row schema, columns outside the equality key might carry null or sentinel values. The ManifestReader has been treating these non-key column stats as reliable for metrics pruning. Consequently, certain predicates (like col IS NOT NULL) can incorrectly skip valid delete manifest entries, leaving rows in the query result that should have been deleted (first observed in Trino https://github.com/trinodb/trino/issues/31399). *Discussion* The open question (relevant for v2, v3, v4 format tables) I'm bringing up is whether there are good reasons to keep on reading statistics for non-equality ID columns from the equality delete files. Additional note: Filtering out stats when reading equality deletes is changing what DeleteFile.lowerBounds()/valueCounts()/nullValueCounts() return, not just how they're used for predicate matching. Because this touches on core read-path behavior and how engines interpret delete manifest stats, I think it would be valuable to get more eyes on this and establish a consensus on the best path forward. *Additional context* https://iceberg.apache.org/spec/#equality-delete-files Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row Let me know your thoughts! Thank you, Marius Grama.
