Hi everyone, below I intend to contextualize an observation I’ve noticed in hopes of discussing with the community.
Context: Some query engines construct scan plans dynamically based on the metrics provided in Iceberg table's metadata.json. For example, when an engine encounters a table with equality deletes, it may rely on the 'total-equality-deletes' metric (as defined in the Iceberg specification here: https://iceberg.apache.org/spec/#metrics) to determine whether equality delete handling logic needs to be engaged during scan planning. A similar approach is commonly taken for position deletes. Engines may use the 'total-position-deletes' metric to decide whether position deletes need to be accounted for. However, with the introduction of Deletion Vectors (DV) in Iceberg V3, this interpretation of the 'total-position-deletes' field becomes more ambiguous. Problem: The core issue is this: when total-position-deletes > 0 in a V3 table, it may indicate: Legacy position delete files (V2) exist Deletion vectors (V3) exist Or both This ambiguity introduces complexity in scan planning. In cases where the physical plan for reading legacy position deletes differs meaningfully from reading deletion vectors, engines must conservatively assume both mechanisms might be in play—even if only one is present. This can lead to unnecessarily complex or suboptimal planning. I’ve noticed there is an 'added-dvs' metric, but no 'total-dvs' equivalent listed in the Iceberg spec’s Metrics <https://iceberg.apache.org/spec/#metrics> section. As a result, total-position-deletes appears to serve as a catch-all for both V2 and V3 position deletes. For engines that rely solely on snapshot-level metrics, this becomes a blind spot. The issue extends beyond the transition period between V2 and V3 too - Even after migrating fully to V3, a table might still retain legacy delete files. Currently, there appears to be no consistent, guaranteed way to prove at the metadata level that only V3 deletion vectors are in use. Some inference is possible by walking the snapshot history and aggregating metrics, but this is fragile and case-specific. It is not viable in to perform manifest scans at runtime to infer delete formats I’m curious if others in the community have encountered this challenge — and if so, how you’re addressing it. Is there an established pattern to help distinguish V2 vs V3 deletes at the metadata level, without relying on manifest/file-level inspection? Looking forward to hearing your thoughts. Best, Jordan
