[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001319#comment-16001319
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/3599#discussion_r115323196
  
--- Diff: flink-shaded-curator/flink-shaded-curator-recipes/pom.xml ---
@@ -70,6 +70,15 @@ under the License.

org.apache.curator:*


+   
--- End diff --

Do we need this here? I think relocation happens in `flink-runtime`, when 
it puts curator into its shaded jar.


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001318#comment-16001318
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3599
  
Thanks for adding this!
I have a few questions:

  - From the formatting, the code looks like might be adapted from some 
other project. If yes, can you share from where and add a comment above the 
code that it is adopted from another project. Also, formatting in the common 
Flink style would be good.

  - The HA leader election is very critical, so we usually don't just 
replace code that has worked well (for many users with new code that we have 
not extensively tested. Therefore, I suggest to add a config option to activate 
this smarter latch, thus making it optional for users that want to use it 
initially.

  - Once this proves stable, we can make it the default in the next version.


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946765#comment-15946765
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3599
  
@wenlong88 Feel free to review, thanks :)


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946757#comment-15946757
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3599
  
@StephanEwen 
I've done the changes, which introduce a new smarter leader latch(the 
reason why i write a new class is that `handleStateChange` method is private in 
`LeaderLatch` and cannot be overrided) which will wait a connection timeout 
duration when connection to zookeeper is broken, instead of revoke leadership 
immediately.


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941521#comment-15941521
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3599
  
I don't think it's a good idea, as it can not solve the "split brain" issue 
too.

The key problem is that `LeaderLatch` in curator is too sensitive to 
connection state to Zookeeper(it will revoke leadership when connection to 
zookeeper is temporarily broke), and probably the best way is offerring a 
"duller" LeaderLatch, which can be also used in standalone cluster.

I did same work in our own private Spark release, let me see if it can be 
reused.


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940522#comment-15940522
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3599
  
I would suggest to fix this the following way:

  - There is an upcoming patch that makes the Flink codebase use the 
`HighAvailabilityServices` properly in all places.
  - We introduce a new HA mode called `yarnsimple` or so (next to `none` 
and `zookeeper`) and instantiate a new implementation of 
`HighAvailabilityServices` which is ZooKeeper independent.
  - The new implementation of the High Availability Services does not use 
ZooKeeper. It uses a leader service that always grants the JobManager 
leadership, but also implements a way for TaskManagers to find the JobManager 
(to be seen how, possibly a file in HDFS or so). It also implements a ZooKeeper 
independent CompletedCheckpointStore that finds checkpoints by maintaining a 
file with completed checkpoints.

That is all not a "proper" HA setup - it only works as long as there is 
strictly only one master
But it comes close and is ZooKeeper independent.

Is that what you are looking for?


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938416#comment-15938416
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3599
  
-1 sorry.

This needs to go to the drawing board (FLIP or detailed JIRA discussion) 
before we consider a change that is impacting the guarantees and failure mode 
so heavily.

Some initial comments:

  - In proper HA, you need some service that "locks" the leader, otherwise 
you are vulnerable to the "split brain" problem where a network partition makes 
multiple JobManagers work as leaders, each with some TaskManagers.

  - In FLIP-6, we are introducing the `HighAvailabilityServices` to allow 
for multiple levels of guarantees with different implementations. I can see 
that introducing a highly-available but not split-brain-protected is 
interesting, but it should not replace any existing mode, but be a new mode.



> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938000#comment-15938000
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user wenlong88 commented on the issue:

https://github.com/apache/flink/pull/3599
  
Hi, I may have described my concern wrongly in the last comment, my concern 
is that in yarn it is possible that two application master running at the same 
time: 
eg: RM launches a AM and then the machine lost connection with RM by some 
reason, RM will launch another AM. It is possible that the first AM will be 
still running when launching the second AM in some scenario like NM heartbeat 
timeout but running.
When it is possible that there are two AM running at the same time, we may 
go into a dead lock using the AlwaysLeaderService as follows: 
1. the first AM grant leadership
2. the second AM grant leadership
3. the second AM write leader info
4. the first AM write leader info
5. the first AM killed by NM or some cluster monitor tool since RM marked 
NM as unavailable.



> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937848#comment-15937848
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3599
  
Thanks for your comments @wenlong88 .

I also gave a thought about adding retry logic when zk failover, but this 
part should modify `LeaderLatch` in curator, which is a 3rd party library, or 
we can only add a our private LeaderLatch through coping most parts of the 
implementation in curator.

Even with adding this AlwaysLeaderService, the JM failover can also go well 
as RM will start a new instance.

about FLIP-6, I'll check the solution and find if anything can help with 
this :)


> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937694#comment-15937694
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

Github user wenlong88 commented on the issue:

https://github.com/apache/flink/pull/3599
  
Hi, @WangTaoTheTonic I think we can improve the reaction of 
ZookeeperLeaderElectionService on zookeeper connection expired or other errors 
instead of introducing the AlwaysLeaderService such as adding a retry before 
revoking leadership, because when the problem is caused by errors on the 
machine which JM is running on, we need to trigger a failover to make the JM 
change a machine. 
On the other hand, in the coming FLIP-6 implementation, JM failover will 
not trigger cancelling all running tasks.



> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6174) Introduce a leader election service in yarn mode to make JobManager always available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937640#comment-15937640
 ] 

ASF GitHub Bot commented on FLINK-6174:
---

GitHub user WangTaoTheTonic opened a pull request:

https://github.com/apache/flink/pull/3599

[FLINK-6174][HA]introduce a new election service to make JobManager always 
available

Now in yarn mode, if we use zookeeper as high availability choice, it will 
create a election service to get a leader depending on zookeeper election.

When zookeeper leader crashes or the connection between JobManager and 
zookeeper instance was broken, JobManager's leadership will be revoked and send 
a Disconnect message to TaskManager, which will cancle all running tasks and 
make them waiting connection rebuild between JM and ZK.

In yarn mode, we have one and only JobManager(AM) in same time, and it 
should be alwasy leader instead of elected through zookeeper. We can introduce 
a new leader election service in yarn mode to achive that.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/WangTaoTheTonic/flink FLINK-6174

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3599


commit a758ec6320b026fb5d767cdc190c29c8043838da
Author: WangTaoTheTonic 
Date:   2017-03-23T03:15:40Z

introduce a new election service to make JobManager always available




> Introduce a leader election service in yarn mode to make JobManager always 
> available
> 
>
> Key: FLINK-6174
> URL: https://issues.apache.org/jira/browse/FLINK-6174
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Tao Wang
>Assignee: Tao Wang
>
> Now in yarn mode, if we use zookeeper as high availability choice, it will 
> create a election service to get a leader depending on zookeeper election.
> When zookeeper leader crashes or the connection between JobManager and 
> zookeeper instance was broken, JobManager's leadership will be revoked and 
> send a Disconnect message to TaskManager, which will cancle all running tasks 
> and make them waiting connection rebuild between JM and ZK.
> In yarn mode, we have one and only JobManager(AM) in same time, and it should 
> be alwasy leader instead of elected through zookeeper. We can introduce a new 
> leader election service in yarn mode to achive that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)