[
https://issues.apache.org/jira/browse/FLINK-41486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot Bancroft updated FLINK-41486:
------------------------------------
Description:
This is recurrent for power loss on edge compute. HA recovery fails after
almost every unexpected non-graceful node shutdown. Power-loss incidents show
unreadable newest checkpoint state handles. Removing the newest HA ConfigMap
checkpoint entries allowed recovery from older retained checkpoints.
After an abrupt node power loss on edge compute, JobManager HA recovery fails
while reading the newest completed-checkpoint state handle. The Kubernetes HA
ConfigMap still contained its checkpointID-... entry, but deserializing the
referenced handle fails with:
```
Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
This indicates that the retrieved state handle is broken.
Caused by: java.io.EOFException
at java.base/java.io.ObjectInputStream.readStreamHeader
```
The effect is consistent across application and a session clusters.
FileSystemStateStorageHelper.store() serializes the HA handle to an
FSDataOutputStream and closes it without calling sync().
KubernetesStateHandleStore.addAndLock() then publishes the handle reference in
the ConfigMap. For Flink’s local filesystem stream, close() calls
FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power
loss can therefore leave a persisted ConfigMap reference to handle bytes that
were not made durable. {*}This concerns the HA completedCheckpoint handle, not
the checkpoint _metadata file{*}.
Expected behaviour: make the HA state handle durable before publishing its
reference. The same storage helper also writes other HA state, including
submittedExecutionPlan* handles.
Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421
addressed separate checkpoint-file fsync gaps.
was:
This is recurrent on small edge compute. HA recovery fails after almost every
unexpected non-graceful node shutdown. Power-loss incidents show unreadable
newest checkpoint state handles. Removing the newest HA ConfigMap checkpoint
entries allowed recovery from older retained checkpoints.
After an abrupt node power loss on edge compute, JobManager HA recovery fails
while reading the newest completed-checkpoint state handle. The Kubernetes HA
ConfigMap still contained its checkpointID-... entry, but deserializing the
referenced handle fails with:
```
Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
This indicates that the retrieved state handle is broken.
Caused by: java.io.EOFException
at java.base/java.io.ObjectInputStream.readStreamHeader
```
The effect is consistent across application and a session clusters.
FileSystemStateStorageHelper.store() serializes the HA handle to an
FSDataOutputStream and closes it without calling sync().
KubernetesStateHandleStore.addAndLock() then publishes the handle reference in
the ConfigMap. For Flink’s local filesystem stream, close() calls
FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power
loss can therefore leave a persisted ConfigMap reference to handle bytes that
were not made durable. {*}This concerns the HA completedCheckpoint handle, not
the checkpoint _metadata file{*}.
Expected behaviour: make the HA state handle durable before publishing its
reference. The same storage helper also writes other HA state, including
submittedExecutionPlan* handles.
Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421
addressed separate checkpoint-file fsync gaps.
> Flink HA checkpoint state handles are published without fsync before node
> power loss
> ------------------------------------------------------------------------------------
>
> Key: FLINK-41486
> URL: https://issues.apache.org/jira/browse/FLINK-41486
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 2.1.0, 2.3.0, 2.2.1, 2.1.3
> Environment: The failure occurs on a Lenovo 10FLS1GM00 running Flink
> 2.1.0 with Kubernetes HA.
> Reporter: Elliot Bancroft
> Priority: Major
>
> This is recurrent for power loss on edge compute. HA recovery fails after
> almost every unexpected non-graceful node shutdown. Power-loss incidents show
> unreadable newest checkpoint state handles. Removing the newest HA ConfigMap
> checkpoint entries allowed recovery from older retained checkpoints.
> After an abrupt node power loss on edge compute, JobManager HA recovery fails
> while reading the newest completed-checkpoint state handle. The Kubernetes HA
> ConfigMap still contained its checkpointID-... entry, but deserializing the
> referenced handle fails with:
> ```
> Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
> This indicates that the retrieved state handle is broken.
> Caused by: java.io.EOFException
> at java.base/java.io.ObjectInputStream.readStreamHeader
> ```
> The effect is consistent across application and a session clusters.
> FileSystemStateStorageHelper.store() serializes the HA handle to an
> FSDataOutputStream and closes it without calling sync().
> KubernetesStateHandleStore.addAndLock() then publishes the handle reference
> in the ConfigMap. For Flink’s local filesystem stream, close() calls
> FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power
> loss can therefore leave a persisted ConfigMap reference to handle bytes that
> were not made durable. {*}This concerns the HA completedCheckpoint handle,
> not the checkpoint _metadata file{*}.
> Expected behaviour: make the HA state handle durable before publishing its
> reference. The same storage helper also writes other HA state, including
> submittedExecutionPlan* handles.
> Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421
> addressed separate checkpoint-file fsync gaps.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)