yadavay-amzn opened a new pull request, #17207:
URL: https://github.com/apache/iceberg/pull/17207
Closes #17206.
## Problem
When a snapshot contains more than one deletion vector (DV) referencing the
same data file, scan planning fails permanently in
`DeleteFileIndex.Builder.add(...)` with `Can't index multiple DVs for
<data-file>`. The table becomes unreadable from every engine, and there is no
built-in way to recover it.
The spec forbids more than one DV per data file per snapshot, so this state
is metadata corruption and the `ValidationException` is a correct detection of
an invalid state. This is the read-side manifestation of the write-side
conflict-detection gap addressed by #15966: that change prevents future
corruption, but it cannot heal tables that are already corrupted, which remain
bricked.
## Change
Recover from the corruption on the read path without losing any deletes:
- DVs continue to be indexed one-per-data-file in `dvByPath` (the healthy
fast path, with no added overhead).
- When a second DV is seen for the same data file, the path moves to an
overflow map (`duplicateDvsByPath`), which is null on healthy tables:
- Byte-identical duplicates (same location, offset, and length) are
deduplicated silently (logged at DEBUG) - this covers the same DV referenced
from two manifest entries.
- Genuinely different DVs are retained and their position bitmaps are
unioned at read time via the existing merge path
(`PositionDeleteIndexUtil.merge`), and a WARNING is logged about the spec
violation. No new Puffin files are written on the read path.
- The union preserves all deletes from all DVs; a DV is never silently
dropped (which would resurrect deleted rows).
## Testing
- `DeleteFileIndex` tests covering two and three different DVs for the same
data file (all retained and applied), and byte-identical deduplication.
- An end-to-end test that serializes two DV bitmaps with positions `{0, 2}`
and `{1, 3}`, deserializes and merges them, and asserts all four positions are
deleted.
- Existing `DeleteFileIndex` behavior is unchanged; spotless and checkstyle
pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
