[
https://issues.apache.org/jira/browse/FLINK-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219520#comment-17219520
]
Jiayi Liao commented on FLINK-19778:
------------------------------------
In the JM log, the point is why zk store cannot read checkpoints in Zookeeper.
{code:java}
2020-10-23 06:17:59,635 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Recovering checkpoints from ZooKeeper.
2020-10-23 06:17:59,706 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
0 checkpoints in ZooKeeper.
2020-10-23 06:17:59,706 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying
to fetch 0 checkpoints from storage.
2020-10-23 06:17:59,706 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Starting job
e70c0d75910ce83ea50cc34ce62a241a from savepoint
hdfs://cosmos/flink/user_10342/savepoints/5d6e4d1574aabf4fab5281f4/savepoint-bbbc5a-f381255749ac
(allowing non restored state)
{code}
Do you find any clues in Zookeeper's logs? BTW, can you query the zk data on
command line?
The zk path should be, ${high-availability.zookeeper.path.root} +
${high-availability.cluster-id} +
${high-availability.zookeeper.path.checkpoints} + ${jobId}.
> Failed job reinitiated with wrong checkpoint after a ZK reconnection
> --------------------------------------------------------------------
>
> Key: FLINK-19778
> URL: https://issues.apache.org/jira/browse/FLINK-19778
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.11.0
> Reporter: Paul Lin
> Priority: Critical
> Attachments: jm_log
>
>
> We have a job of Flink 1.11.0 running on YARN that reached FAILED state
> because its jobmanager lost leadership during a ZK full GC. But after the ZK
> connection was recovered, somehow the job was reinitiated again with no
> checkpoints found in ZK, and hence an earlier savepoint was used to restore
> the job, which rewound the job unexpectedly.
>
> For details please see the jobmanager logs in the attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
