[
https://issues.apache.org/jira/browse/FLINK-41486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145589#comment-18145589
]
Elliot Bancroft commented on FLINK-41486:
-----------------------------------------
Would be more than happy to contribute to a fix here if helpful - especially
given the narrow scope and precedent set in the fixes for FLINK-36421 and
FLINK-35217
> Flink HA checkpoint state handles are published without fsync before node
> power loss
> ------------------------------------------------------------------------------------
>
> Key: FLINK-41486
> URL: https://issues.apache.org/jira/browse/FLINK-41486
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 2.1.0, 2.3.0, 2.2.1, 2.1.3
> Environment: The failure occurs on a Lenovo 10FLS1GM00 running Flink
> 2.1.0 with Kubernetes HA.
> Reporter: Elliot Bancroft
> Priority: Major
>
> This is recurrent for power loss on edge compute. HA recovery fails after
> almost every unexpected non-graceful node shutdown. Power-loss incidents show
> unreadable newest checkpoint state handles. Removing the newest HA ConfigMap
> checkpoint entries allowed recovery from older retained checkpoints.
> After an abrupt node power loss on edge compute, JobManager HA recovery fails
> while reading the newest completed-checkpoint state handle. The Kubernetes HA
> ConfigMap still contained its checkpointID-... entry, but deserializing the
> referenced handle fails with:
> ```
> Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
> This indicates that the retrieved state handle is broken.
> Caused by: java.io.EOFException
> at java.base/java.io.ObjectInputStream.readStreamHeader
> ```
> The effect is consistent across application and a session clusters.
> FileSystemStateStorageHelper.store() serializes the HA handle to an
> FSDataOutputStream and closes it without calling sync().
> KubernetesStateHandleStore.addAndLock() then publishes the handle reference
> in the ConfigMap. For Flink’s local filesystem stream, close() calls
> FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power
> loss can therefore leave a persisted ConfigMap reference to handle bytes that
> were not made durable. {*}This concerns the HA completedCheckpoint handle,
> not the checkpoint _metadata file{*}.
> Expected behaviour: make the HA state handle durable before publishing its
> reference. The same storage helper also writes other HA state, including
> submittedExecutionPlan* handles.
> Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421
> addressed separate checkpoint-file fsync gaps.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)