[
https://issues.apache.org/jira/browse/FLINK-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621772#comment-16621772
]
Thomas Wozniakowski commented on FLINK-10184:
---------------------------------------------
Hey [~till.rohrmann],
It's kind of non-trivial for me to test the fixes, as our cluster is currently
running the non-hadoop 1.4.3 build. As far as I can see the only snapshot
builds available contain hadoop, so I didn't know if the tests would be
representative. I was waiting on the official release binaries before spending
time testing.
I can have a go at testing from a local maven build, but I've had significant
trouble wrestling with maven on the Flink codebase in the past (trying to build
locally). If you could point me at a branch (say for the 1.5 release) and let
me know what maven command I should use to build it with no hadoop, and scala
2.11, then I would be very grateful. I could then use those binaries for
testing.
Tom
> HA Failover broken due to JobGraphs not being removed from Zookeeper on cancel
> ------------------------------------------------------------------------------
>
> Key: FLINK-10184
> URL: https://issues.apache.org/jira/browse/FLINK-10184
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
> Affects Versions: 1.5.2, 1.6.0
> Reporter: Thomas Wozniakowski
> Priority: Blocker
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We have encountered a blocking issue when upgrading our cluster to 1.5.2.
> It appears that, when jobs are cancelled manually (in our case with a
> savepoint), the JobGraphs are NOT removed from the Zookeeper {{jobgraphs}}
> node.
> This means that, if you start a job, cancel it, restart it, cancel it, etc.
> You will end up with many job graphs stored in zookeeper, but none of the
> corresponding blobs in the Flink HA directory.
> When a HA failover occurs, the newly elected leader retrieves all of those
> old JobGraph objects from Zookeeper, then goes looking for the corresponding
> blobs in the HA directory. The blobs are not there so the JobManager explodes
> and the process dies.
> At this point the cluster has to be fully stopped, the zookeeper jobgraphs
> cleared out by hand, and all the jobmanagers restarted.
> I can see the following line in the JobManager logs:
> {quote}
> 2018-08-20 16:17:20,776 INFO
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore -
> Removed job graph 4e9a5a9d70ca99dbd394c35f8dfeda65 from ZooKeeper.
> {quote}
> But looking in Zookeeper the {{4e9a5a9d70ca99dbd394c35f8dfeda65}} job is
> still very much there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
