xiaoxuandev opened a new pull request, #17764:
URL: https://github.com/apache/iceberg/pull/17764
## Problem
On format-v3 merge-on-read tables, compaction and other commit-time
validations can fail with
ValidationException: Can't index multiple DVs for <data file>
on a table that is not corrupt, scans succeed, and the current snapshot has
exactly one live DV per data file.
`MergingSnapshotProducer.addedDeleteFiles` builds a `DeleteFileIndex` over
the delete manifests added across the whole validation history between the
starting snapshot and the current parent. A data file can legitimately have
more than one DV in that window:
* S0: append data file A (compactor starts here)
* S1: writer adds DV1 for A
* S2: writer removes DV1 and adds DV2 for A (valid MoR upsert)
Manifests are immutable: DV1 is still live in the manifest S1 wrote, DV2 in
the one S2 wrote. The index reads live entries from every collected manifest,
filtered only by sequence number, so it sees both. `Builder.build()` then
throws from `putIfAbsent` on `dvByPath`, before validation reaches its conflict
check.
The index covers the whole window and is not scoped to the data files being
validated, so this rejects even a rewrite of an unrelated, delete-free file,
stalling compaction on sustained-upsert workloads. The single-live-DV invariant
is correct for scans, where a data file really does have at most one live DV.
It does not hold across snapshots.
## Fix
Add a package-private, default-off `indexDVsAsPositionDeletes()` to
`DeleteFileIndex.Builder`. When set, DVs are indexed in `posDeletesByPath`,
which already allows many entries per data file, instead of the strict
`dvByPath`.
The same delete files are indexed either way; only the container changes.
Per data file the tolerant path returns a superset: where a DV exists, the
strict path returns the equality deletes plus that one DV and short-circuits the
position-delete buckets, while the tolerant path returns those buckets and
every DV for the file. `findDV` asserts `dataSequenceNumber >= seq` where
`PositionDeletes.filter` silently drops below it, but that difference is
unreachable here — `afterSequenceNumber` has already dropped every entry at or
below `startingSequenceNumber`, the same value later passed to `forDataFile`.
So validation sees more candidate deletes, never fewer, and cannot miss a
conflict
it previously caught.
The index answers whether a data file has new deletes, not which deletes
apply to it. That is sufficient because all three consumers only test emptiness
or content type; anything needing the applicable set must use the strict
default.
Only commit-time validation enables the flag: `buildDeleteFileIndex`,
reached from `validateNoNewDeletesForDataFiles` and both
`validateNoNewDeleteFiles` overloads. Scans keep the default and stay strict,
so genuinely corrupt live metadata still surfaces there.
## Tests
New regression tests across all three validation entry points (all format
v3+):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
