vgkowski opened a new issue, #17498:
URL: https://github.com/apache/iceberg/issues/17498
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
`rewrite_table_path` permanently fails with `NotFoundException` /
`NoSuchKey` on tables where a position delete file (or deletion vector) was
dropped by compaction and later physically removed by `expire_snapshots`. The
table itself is perfectly healthy — no reader ever touches the removed file —
but it can no longer be replicated, either with a full rewrite or with an
incremental rewrite whose delta spans the expiry.
#### Root cause
`RewriteTablePathUtil.writeDeleteFileEntry` applies a filter when building
the copy plan: only entries that are live and added within the requested delta
are copied. Entries that are `DELETED` (history markers) or outside the delta
are still written into the rewritten manifest, but correctly excluded from the
copy plan.
However, the physical-rewrite set does not apply the same filter — every
position delete entry is queued for rewriting unconditionally:
```java
if (entry.isLive() && snapshotIds.contains(entry.snapshotId())) {
result.copyPlan().add(...);
}
result.toRewrite().add(file.copy()); // unconditional
```
Position delete files must be physically rewritten (they embed absolute data
file paths), so `RewriteTablePathSparkAction` opens every file in `toRewrite`.
When the entry is a `DELETED` marker whose underlying file was removed by
`expire_snapshots`, the open fails and the whole procedure aborts — even though
the file was never going to be copied.
Because the `DELETED` entry is baked into the current snapshot's delete
manifest, the failure is not transient: every subsequent `rewrite_table_path`
run fails the same way.
#### Reproduction (format version 3)
1. Create a table, append data files.
2. Commit a row delta with a deletion vector
(`newRowDelta().addDeletes(dv)`).
3. Run `rewrite_data_files` — compaction merges the deletes and drops the
DV, leaving a `DELETED` entry in the current snapshot's delete manifest.
4. Run `expire_snapshots` expiring everything older than the compaction
snapshot — the DV file is now physically removed from storage, while the
`DELETED` entry referencing it survives.
5. Run `rewrite_table_path` (full rewrite) — fails with `NotFoundException`
from `PuffinReader` while opening the removed DV.
Note the `DELETED` entry actually passes the
`snapshotIds.contains(entry.snapshotId())` check (it is stamped with the
compaction snapshot's id), so it is only the `isLive()` half of the copy-plan
guard that excludes it from copying — while the rewrite set ignores both.
#### Incremental variant
1. Full `rewrite_table_path` to a target (run 1).
2. On the source: compaction drops a position delete file, then
`expire_snapshots` removes it from storage.
3. Incremental `rewrite_table_path` with a `startVersion`/`endVersion`
spanning the expiry — the delta manifests still carry the entry (as `DELETED`,
or as `EXISTING` outside the delta), it is queued for physical rewriting, and
the run fails.
This breaks the common setup of periodic incremental replication combined
with routine table maintenance on the source.
#### Non-fatal variant (format version 2)
In v2, compaction leaves the dropped position delete file on storage, so
nothing fails — but every dead entry's file is opened, fully rewritten, and
written to staging, then never copied (it is not in the copy plan). Wasted I/O
and staging storage proportional to the table's history of dropped delete files.
#### Expected behavior
The physical-rewrite set should be aligned with the copy plan: an entry that
is excluded from the copy plan is kept in the rewritten manifest exactly as the
source has it and must not be opened. This is already how the data-file and
equality-delete branches behave — only the position-delete branch is
inconsistent.
The `main` branch is not affected: `positionDeletesToRewrite()` collects
files to rewrite from live manifest entries. This affects the released line,
where the set is built inside `writeDeleteFileEntry`.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
