[ 
https://issues.apache.org/jira/browse/FLINK-41486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145589#comment-18145589
 ] 

Elliot Bancroft commented on FLINK-41486:
-----------------------------------------

Would be more than happy to contribute to a fix here if helpful - especially 
given the narrow scope and precedent set in the fixes for FLINK-36421 and 
FLINK-35217

> Flink HA checkpoint state handles are published without fsync before node 
> power loss
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-41486
>                 URL: https://issues.apache.org/jira/browse/FLINK-41486
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing
>    Affects Versions: 2.1.0, 2.3.0, 2.2.1, 2.1.3
>         Environment: The failure occurs on a Lenovo 10FLS1GM00 running Flink 
> 2.1.0 with Kubernetes HA. 
>            Reporter: Elliot Bancroft
>            Priority: Major
>
> This is recurrent for power loss on edge compute. HA recovery fails after 
> almost every unexpected non-graceful node shutdown. Power-loss incidents show 
> unreadable newest checkpoint state handles. Removing the newest HA ConfigMap 
> checkpoint entries allowed recovery from older retained checkpoints.
> After an abrupt node power loss on edge compute, JobManager HA recovery fails 
> while reading the newest completed-checkpoint state handle. The Kubernetes HA 
> ConfigMap still contained its checkpointID-... entry, but deserializing the 
> referenced handle fails with:
> ```
> Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
> This indicates that the retrieved state handle is broken.
> Caused by: java.io.EOFException
>   at java.base/java.io.ObjectInputStream.readStreamHeader
> ```
> The effect is consistent across application and a session clusters.
> FileSystemStateStorageHelper.store() serializes the HA handle to an 
> FSDataOutputStream and closes it without calling sync(). 
> KubernetesStateHandleStore.addAndLock() then publishes the handle reference 
> in the ConfigMap. For Flink’s local filesystem stream, close() calls 
> FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power 
> loss can therefore leave a persisted ConfigMap reference to handle bytes that 
> were not made durable. {*}This concerns the HA completedCheckpoint handle, 
> not the checkpoint _metadata file{*}.
> Expected behaviour: make the HA state handle durable before publishing its 
> reference. The same storage helper also writes other HA state, including 
> submittedExecutionPlan* handles.
> Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421 
> addressed separate checkpoint-file fsync gaps.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to