[
https://issues.apache.org/jira/browse/FLINK-39964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Savonin updated FLINK-39964:
--------------------------------------
Description:
When a job restores from a NATIVE-format savepoint in CLAIM mode with RocksDB
incremental checkpoints enabled, the next checkpoint reuses the savepoint's SST
files instead of re-uploading them. The checkpoint completes successfully, but
its metadata records the reused files with the relative (file-name-only)
encoding, which a restore later resolves against the checkpoint's own
directory. The bytes physically live in the savepoint directory, so the
resolved paths point at files that do not exist.
The problem stays silent while the job keeps running, because failover restores
from the JobManager's completed-checkpoint store and never re-reads the
metadata file. It surfaces only when a fresh job is started from the
checkpoint's path: the restore fails with a FileNotFoundException for SST files
under the checkpoint directory. The first checkpoint after the restore is
always affected and later ones remain affected until the reused files have been
compacted away.
Only filesystem checkpoint storage is affected, setups with entropy injection
always write absolute paths and are immune.
was:The first incremental checkpoint a job takes after restoring from a
*native savepoint* in RestoreMode.CLAIM can be completed successfully but be
*unrestorable*: it references reused RocksDB SST files at paths inside the new
checkpoint directory, while the bytes physically live in the savepoint
directory. Restoring a fresh job from such a checkpoint fails with a
file-not-found error. The problem is silent until a cold restore actually reads
the checkpoint back.
> Checkpoints taken after a CLAIM-mode restore from a native savepoint can be
> unrestorable
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-39964
> URL: https://issues.apache.org/jira/browse/FLINK-39964
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
> Reporter: Aleksandr Savonin
> Assignee: Aleksandr Savonin
> Priority: Critical
>
> When a job restores from a NATIVE-format savepoint in CLAIM mode with RocksDB
> incremental checkpoints enabled, the next checkpoint reuses the savepoint's
> SST files instead of re-uploading them. The checkpoint completes
> successfully, but its metadata records the reused files with the relative
> (file-name-only) encoding, which a restore later resolves against the
> checkpoint's own directory. The bytes physically live in the savepoint
> directory, so the resolved paths point at files that do not exist.
> The problem stays silent while the job keeps running, because failover
> restores from the JobManager's completed-checkpoint store and never re-reads
> the metadata file. It surfaces only when a fresh job is started from the
> checkpoint's path: the restore fails with a FileNotFoundException for SST
> files under the checkpoint directory. The first checkpoint after the restore
> is always affected and later ones remain affected until the reused files have
> been compacted away.
> Only filesystem checkpoint storage is affected, setups with entropy injection
> always write absolute paths and are immune.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
