lucasgameiroborges opened a new issue, #9342: URL: https://github.com/apache/paimon/issues/9342
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 1.4.1 (but code seems identical on current `master` `4a346f450`) ### Compute Engine Flink 2.0 ### Minimal reproduce step Unit test for reproduction will be included in the PR. Not sure we can reliably reproduce in a live system but the process was: 1. Writing continuously to a Paimon table with Iceberg REST metadata sync enabled. 2. Get a committer recovery that replays a committable more than one snapshot behind the catalog, ie `createMetadataWithoutBase` runs. 3. `expireAllBefore(N)` deletes the manifest lists of every version < N. Then at some point we need to have `v(N-1).metadata.json` retained while the manifest lists it references have just been deleted. 4. Again get to a committer recovery where createMetadataWithoutBase runs. `expireAllBefore` now walks `v(N-1).metadata.json` and calls `manifestList.read(listName)` on a list that was already deleted in step 3. This will lead to: ``` java.lang.RuntimeException: Failed to read snap-1-1f878b70-445c-4332-98a4-73ea0c549437.avro at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:131) at org.apache.paimon.iceberg.IcebergCommitCallback.expireAllBefore(IcebergCommitCallback.java:985) at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithoutBase(IcebergCommitCallback.java:386) at org.apache.paimon.iceberg.IcebergCommitCallback.retry(IcebergCommitCallback.java:249) at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:277) at org.apache.paimon.flink.sink.RestoreCommittableStateManager.recover(RestoreCommittableStateManager.java:81) at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:147) Caused by: java.io.FileNotFoundException: File '…/metadata/snap-1-1f878b70-….avro' not found ``` ### What doesn't meet your expectations? `expireAllBefore` exists to delete the manifests and manifest lists of expired versions. It deletes them with `deleteQuietly` (tolerating absence) but reads the manifest list with a hard `manifestList.read(...)` that throws when the file is already gone. If the two cleanup steps lead to a retained metadata JSON referencing an already-deleted list, the task dies during Flink state initialisation, the committable is never dropped, and recovery replays it until the retained JSON happens to be deleted by later churn. Skipping a manifest list that no longer exists seems safe since the loop's only purpose in reading it is to delete what it references, and those manifests were deleted by the same earlier pass. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
