lucasgameiroborges opened a new pull request, #9343:
URL: https://github.com/apache/paimon/pull/9343
### Purpose
Closes #9342.
`IcebergCommitCallback.expireAllBefore` fails permanently on a manifest list
that a previous from-scratch rebuild already deleted.
It reads every expired version's manifest list so it can delete the
manifests that list references:
```java
for (IcebergManifestFileMeta meta : manifestList.read(listName)) { // hard
read
table.fileIO().deleteQuietly(new Path(meta.manifestPath())); //
tolerant
}
table.fileIO().deleteQuietly(listPath); //
tolerant
```
The deletes are `deleteQuietly` and tolerate absence. The read does not.
A metadata JSON outliving its own manifest list is reachable in normal
operation, because the two cleanup steps use different windows:
| Step | Deletes |
|---|---|
| `expireAllBefore(N)` | manifest lists for **every** version `< N`,
unconditionally |
| `deleteApplicableMetadataFiles(N)` | metadata JSONs below `N -
previous-versions-max` |
So after a from-scratch rebuild at `N`, `v(N-1).metadata.json` is retained
while the manifest list it points at has just been deleted. The next rebuild on
that table walks the retained JSON and throws:
```
java.lang.RuntimeException: Failed to read
snap-1-1f878b70-445c-4332-98a4-73ea0c549437.avro
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:131)
at
org.apache.paimon.iceberg.IcebergCommitCallback.expireAllBefore(IcebergCommitCallback.java:985)
at
org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithoutBase(IcebergCommitCallback.java:386)
at
org.apache.paimon.iceberg.IcebergCommitCallback.retry(IcebergCommitCallback.java:249)
at
org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:277)
at
org.apache.paimon.flink.sink.RestoreCommittableStateManager.recover(RestoreCommittableStateManager.java:81)
at
org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:147)
Caused by: java.io.FileNotFoundException: File
'.../metadata/snap-1-1f878b70-….avro' not found
```
On Flink this is worse than noisy. It surfaces through
`CommitterOperator.initializeState`, so the task dies during state
initialisation, no checkpoint completes, the restored committable is never
dropped, and recovery replays it — until later churn happens to delete the
retained JSON. We have observed tables looping for ~12h on this.
**Fix:** skip a manifest list that no longer exists. The only reason to open
it here is to delete what it references, and those manifests were deleted by
the same earlier pass that deleted the list. That is the one assumption in this
patch — happy to be told if there is a case where the read must be strict.
No API or format change; the behaviour change is confined to cleanup of
already-expired metadata.
Encountered on Paimon 1.4.1 with `metadata.iceberg.storage = rest-catalog`,
`previous-versions-max = 1`, `delete-after-commit.enabled = true` (default),
Flink 2.0. The code is identical on current `master`.
Related context on the same code path: #8875 asks whether an Iceberg
metadata-sync failure should fail the Flink job at all. With that issue's
proposal in place this bug would degrade to a logged warning rather than a
restart loop, but the dangling read would still need fixing. The
`recreateTable` / from-scratch path that creates the dangling reference is also
the subject of a comment on #8875 reporting that it fails outright on Glue.
### Tests
`IcebergCompatibilityTest#testExpireAllBeforeSkipsAlreadyDeletedManifestList`.
It reproduces the state directly: commit twice, delete the manifest list
that the retained `v(N-1).metadata.json` points at (what an earlier rebuild
does), then delete the base metadata JSON so the next commit takes the
from-scratch path and walks that retained JSON.
Verified to be a genuine regression test — with the production change
reverted it fails with exactly the error above (`Failed to read
snap-1-<uuid>.avro` / `FileNotFoundException`), and passes with it. All 42
tests in `IcebergCompatibilityTest` pass; `spotless:check` and
`checkstyle:check` are clean on `paimon-core`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
