[ 
https://issues.apache.org/jira/browse/FLINK-41486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliot Bancroft updated FLINK-41486:
------------------------------------
    Description: 
This is recurrent for power loss on edge compute. HA recovery fails after 
almost every unexpected non-graceful node shutdown. Power-loss incidents show 
unreadable newest checkpoint state handles. Removing the newest HA ConfigMap 
checkpoint entries allowed recovery from older retained checkpoints.

After an abrupt node power loss on edge compute, JobManager HA recovery fails 
while reading the newest completed-checkpoint state handle. The Kubernetes HA 
ConfigMap still contained its checkpointID-... entry, but deserializing the 
referenced handle fails with:

```
Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
This indicates that the retrieved state handle is broken.
Caused by: java.io.EOFException
  at java.base/java.io.ObjectInputStream.readStreamHeader
```
The effect is consistent across application and a session clusters.

FileSystemStateStorageHelper.store() serializes the HA handle to an 
FSDataOutputStream and closes it without calling sync(). 
KubernetesStateHandleStore.addAndLock() then publishes the handle reference in 
the ConfigMap. For Flink’s local filesystem stream, close() calls 
FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power 
loss can therefore leave a persisted ConfigMap reference to handle bytes that 
were not made durable. {*}This concerns the HA completedCheckpoint handle, not 
the checkpoint _metadata file{*}.

Expected behaviour: make the HA state handle durable before publishing its 
reference. The same storage helper also writes other HA state, including 
submittedExecutionPlan* handles.

Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421 
addressed separate checkpoint-file fsync gaps.

  was:
This is recurrent on small edge compute. HA recovery fails after almost every 
unexpected non-graceful node shutdown. Power-loss incidents show unreadable 
newest checkpoint state handles. Removing the newest HA ConfigMap checkpoint 
entries allowed recovery from older retained checkpoints.

After an abrupt node power loss on edge compute, JobManager HA recovery fails 
while reading the newest completed-checkpoint state handle. The Kubernetes HA 
ConfigMap still contained its checkpointID-... entry, but deserializing the 
referenced handle fails with:

```
Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
This indicates that the retrieved state handle is broken.
Caused by: java.io.EOFException
  at java.base/java.io.ObjectInputStream.readStreamHeader
```
The effect is consistent across application and a session clusters.

FileSystemStateStorageHelper.store() serializes the HA handle to an 
FSDataOutputStream and closes it without calling sync(). 
KubernetesStateHandleStore.addAndLock() then publishes the handle reference in 
the ConfigMap. For Flink’s local filesystem stream, close() calls 
FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power 
loss can therefore leave a persisted ConfigMap reference to handle bytes that 
were not made durable. {*}This concerns the HA completedCheckpoint handle, not 
the checkpoint _metadata file{*}.

Expected behaviour: make the HA state handle durable before publishing its 
reference. The same storage helper also writes other HA state, including 
submittedExecutionPlan* handles.

Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421 
addressed separate checkpoint-file fsync gaps.


> Flink HA checkpoint state handles are published without fsync before node 
> power loss
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-41486
>                 URL: https://issues.apache.org/jira/browse/FLINK-41486
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing
>    Affects Versions: 2.1.0, 2.3.0, 2.2.1, 2.1.3
>         Environment: The failure occurs on a Lenovo 10FLS1GM00 running Flink 
> 2.1.0 with Kubernetes HA. 
>            Reporter: Elliot Bancroft
>            Priority: Major
>
> This is recurrent for power loss on edge compute. HA recovery fails after 
> almost every unexpected non-graceful node shutdown. Power-loss incidents show 
> unreadable newest checkpoint state handles. Removing the newest HA ConfigMap 
> checkpoint entries allowed recovery from older retained checkpoints.
> After an abrupt node power loss on edge compute, JobManager HA recovery fails 
> while reading the newest completed-checkpoint state handle. The Kubernetes HA 
> ConfigMap still contained its checkpointID-... entry, but deserializing the 
> referenced handle fails with:
> ```
> Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
> This indicates that the retrieved state handle is broken.
> Caused by: java.io.EOFException
>   at java.base/java.io.ObjectInputStream.readStreamHeader
> ```
> The effect is consistent across application and a session clusters.
> FileSystemStateStorageHelper.store() serializes the HA handle to an 
> FSDataOutputStream and closes it without calling sync(). 
> KubernetesStateHandleStore.addAndLock() then publishes the handle reference 
> in the ConfigMap. For Flink’s local filesystem stream, close() calls 
> FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power 
> loss can therefore leave a persisted ConfigMap reference to handle bytes that 
> were not made durable. {*}This concerns the HA completedCheckpoint handle, 
> not the checkpoint _metadata file{*}.
> Expected behaviour: make the HA state handle durable before publishing its 
> reference. The same storage helper also writes other HA state, including 
> submittedExecutionPlan* handles.
> Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421 
> addressed separate checkpoint-file fsync gaps.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to