sundapeng opened a new pull request, #9851:
URL: https://github.com/apache/paimon/pull/9851
### Purpose
`SnapshotManager#tryFromPath` reports a missing snapshot only when
`FileIO#readFileUtf8` throws `FileNotFoundException`. Some FileIOs report a
snapshot that is deleted after its input stream was opened as a generic
`IOException` on the first read. A Flink writer reading snapshots from OSS
through the Jindo SDK failed its checkpoint with:
```
java.lang.RuntimeException: Fails to read snapshot from path
oss://<bucket>/<database>.db/<table>/snapshot/snapshot-<id>
at org.apache.paimon.utils.SnapshotManager.tryFromPath
at org.apache.paimon.utils.SnapshotManager.tryGetSnapshot
at org.apache.paimon.utils.SnapshotManager.latestSnapshotOfUser
at
org.apache.paimon.utils.SnapshotManager.latestSnapshotOfUserFromFilesystem
at
org.apache.paimon.operation.FileSystemWriteRestore.latestCommittedIdentifier
at
org.apache.paimon.operation.AbstractFileStoreWrite.createConflictAwareWriterCleanChecker
at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit
...
Caused by: java.io.IOException: Read from
oss://<bucket>/<database>.db/<table>/snapshot/snapshot-<id> with error message:
404 Not Found NoSuchKey
at com.aliyun.jindodata.common.JindoLakeInputStream.innerRead
...
at org.apache.paimon.fs.FileIO.readFileUtf8
at org.apache.paimon.utils.SnapshotManager.tryFromPath
```
The writer walks snapshots from the latest one to find its last committed
identifier. Since #7572, `latestSnapshotOfUser` stops the walk only on
`FileNotFoundException`. In this case a concurrent snapshot expiration deleted
the snapshot right after the writer opened it, the read failed with a plain
`IOException`, and `tryFromPath` wrapped it into a `RuntimeException`. The same
not-found contract is also used by `latestSnapshotFromFileSystem`,
`safelyGetAllSnapshots` and the earliest snapshot retries.
With this change, when reading fails with any other `IOException`,
`tryFromPath` checks whether the snapshot file still exists:
- If it is gone, it throws `FileNotFoundException` with the original error
as the cause.
- If it still exists, or the existence check fails as well, it keeps
throwing the previous `RuntimeException`.
The existence check only runs on the failure path.
### Tests
Added to `SnapshotManagerTest`:
- `testTryFromPathReportsSnapshotDeletedDuringReadAsNotFound`
- `testTryFromPathKeepsReadFailureOfExistingSnapshot`
- `testLatestSnapshotOfUserStopsAtSnapshotDeletedDuringRead`, which fails
without the change in `SnapshotManager`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
