The column stats are indeed collected by Flink in non-upsert mode. (In that mode the equality delete row schema is the full table schema, and the delete writer uses the table's metrics config. So non-key columns get value/null counts and lower/upper bounds.)
If equality deletes were going to stay with us long term, I'd suggest changing the spec to say: "Stats for columns not in equality_ids should not be stored for equality delete files." Since, equality deletes are being phased out, and v4 already forbids them. Existing tables still carry these stats, so the practical fix is to ignore them. I see two options: 1. Drop the non-key stats when the manifests are read. This is a localized change in core. The non-key stats would no longer appear in metadata tables and would be permanently removed when manifests are rewritten. Since nothing legitimately uses them, that's harmless, and arguably a cleanup. 2. Ignore them where they are used for filtering. This is what this PR does, in InclusiveMetricsEvaluator and StrictMetricsEvaluator. Stored metadata stays untouched, but the public evaluators now treat equality delete files differently. My preference would be 1, but I would like to hear what others think, or I miss some other uses of these stats. Thanks, Peter Marius Grama <[email protected]> ezt írta (időpont: 2026. okt. 22., Cs, 15:11): > Hi everyone, > > I’d like to bring up a topic currently being discussed in PR > https://github.com/apache/iceberg/pull/18338 to the dev list to get > broader community input. > > *Background* > As a quick refresher, the PR addresses a *silent correctness bug* > regarding equality delete files. Currently, when equality delete files are > written with the full row schema, columns outside the equality key might > carry null or sentinel values. The ManifestReader has been treating these > non-key column stats as reliable for metrics pruning. Consequently, certain > predicates (like col IS NOT NULL) can incorrectly skip valid delete > manifest entries, leaving rows in the query result that should have been > deleted (first observed in Trino > https://github.com/trinodb/trino/issues/31399). > > *Discussion* > > The open question (relevant for v2, v3, v4 format tables) I'm bringing up > is whether there are good reasons to keep on reading statistics for > non-equality ID columns from the equality delete files. > > Additional note: > Filtering out stats when reading equality deletes is changing what > DeleteFile.lowerBounds()/valueCounts()/nullValueCounts() return, not just > how they're used for predicate matching. > > Because this touches on core read-path behavior and how engines interpret > delete manifest stats, I think it would be valuable to get more eyes on > this and establish a consensus on the best path forward. > > *Additional context* > > https://iceberg.apache.org/spec/#equality-delete-files > > Equality delete files identify deleted rows in a collection of data files > by one or more column values, and may optionally contain additional columns > of the deleted row > > > Let me know your thoughts! > > Thank you, > > Marius Grama. >
