Elliot Bancroft created FLINK-41486:
---------------------------------------
Summary: Flink HA checkpoint state handles are published without
fsync before node power loss
Key: FLINK-41486
URL: https://issues.apache.org/jira/browse/FLINK-41486
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 2.1.3, 2.2.1, 2.3.0, 2.1.0
Environment: The failure occurs on a Lenovo 10FLS1GM00 running Flink
2.1.0 with Kubernetes HA.
Reporter: Elliot Bancroft
This is recurrent on small edge compute. HA recovery fails after almost every
unexpected non-graceful node shutdown. Power-loss incidents show unreadable
newest checkpoint state handles. Removing the newest HA ConfigMap checkpoint
entries allowed recovery from older retained checkpoints.
After an abrupt node power loss on edge compute, JobManager HA recovery fails
while reading the newest completed-checkpoint state handle. The Kubernetes HA
ConfigMap still contained its checkpointID-... entry, but deserializing the
referenced handle fails with:
```
Could not retrieve checkpoint <N> from state handle under checkpointID-<N>.
This indicates that the retrieved state handle is broken.
Caused by: java.io.EOFException
at java.base/java.io.ObjectInputStream.readStreamHeader
```
The effect is consistent across application and a session clusters.
FileSystemStateStorageHelper.store() serializes the HA handle to an
FSDataOutputStream and closes it without calling sync().
KubernetesStateHandleStore.addAndLock() then publishes the handle reference in
the ConfigMap. For Flink’s local filesystem stream, close() calls
FileOutputStream.close(), whereas sync() calls FileDescriptor.sync(). A power
loss can therefore leave a persisted ConfigMap reference to handle bytes that
were not made durable. {*}This concerns the HA completedCheckpoint handle, not
the checkpoint _metadata file{*}.
Expected behaviour: make the HA state handle durable before publishing its
reference. The same storage helper also writes other HA state, including
submittedExecutionPlan* handles.
Related: FLINK-25098 discusses syncing HA files; FLINK-35217 and FLINK-36421
addressed separate checkpoint-file fsync gaps.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
