[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875800#comment-15875800
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2903


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874889#comment-15874889
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2903
  
I think this looks good, thanks!
Merging this...


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863383#comment-15863383
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user shuai-xu commented on the issue:

https://github.com/apache/flink/pull/2903
  
@StephanEwen I have modified according to your comments. Can this pr be 
merged, I have a new jira need to based on this one. Thank you very much


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863374#comment-15863374
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

GitHub user shuai-xu reopened a pull request:

https://github.com/apache/flink/pull/2903

[FLINK-5074] [runtime] add a zookeeper based running job registry

This pr is for jira 
#[5074](https://issues.apache.org/jira/browse/FLINK-5074).

Implement a zookeeper based running job registry for flip-6. It will write 
the running job info to zookeeper and support querying job status.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shuai-xu/flink jira-5074

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2903.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2903


commit 18d9e17f10cf98d9fe351da181cae57b52957f6f
Author: shuai.xus 
Date:   2016-11-30T08:05:55Z

[FLINK-5704] [runtime] add a zookeeper based running job registry

commit 5a623063e2f9a39e6e0eb32319faa7e9d564cf3d
Author: shuai.xus 
Date:   2016-12-01T07:27:11Z

[FLINK-5074] [runtime] HighAvailabilityServicesUtils support new 
ZookeeperHaServices

commit 2f42ec657eb69197ef5280c114d486ccd1271c44
Author: shuai.xus 
Date:   2016-12-05T06:12:39Z

[FLINK-5074] [runtime] fix get zk path bug and add unittest

commit 0fa9a8d7e4d31451f72177f93e3fba3ca5ad3593
Author: shuai.xus 
Date:   2016-12-21T09:44:44Z

[FLINK-5074] [runtime] add throws Exception to before and after




> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834213#comment-15834213
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user shuai-xu closed the pull request at:

https://github.com/apache/flink/pull/2903


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766621#comment-15766621
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user shuai-xu commented on the issue:

https://github.com/apache/flink/pull/2903
  
@StephanEwen done


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751383#comment-15751383
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2903#discussion_r92608715
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/highavailability/ZooKeeperRegistryTest.java
 ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability;
+
+import org.apache.curator.test.TestingServer;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.util.TestLogger;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.junit.Assert.*;
+
+public class ZooKeeperRegistryTest extends TestLogger {
+   private TestingServer testingServer;
+
+   private static Logger LOG = 
LoggerFactory.getLogger(ZooKeeperRegistryTest.class);
+
+   @Before
+   public void before() {
+   try {
+   testingServer = new TestingServer();
+   } catch (Exception e) {
--- End diff --

I would just add `throws Exception` to the method signature here, rather 
than catching and wrapping the exception. Makes error stack traces easier to 
read.

As a general rule: I think one should never need to explicitly throw a 
`RuntimeException`.
(I know Flink has a few places where it does it, but these are in my 
opinion errors from the past, during the time when we were still learning)


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5074) Implement a RunningJobRegistry based on Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715843#comment-15715843
 ] 

ASF GitHub Bot commented on FLINK-5074:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2903
  
Looks good to me!

I think it would be good to add a test to this, based on a Curator 
TestingServer.
This test here gives an example how to use the TestingServer:

https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/ZooKeeperLeaderElectionTest.java


> Implement a RunningJobRegistry based on Zookeeper 
> --
>
> Key: FLINK-5074
> URL: https://issues.apache.org/jira/browse/FLINK-5074
> Project: Flink
>  Issue Type: Task
>  Components: Cluster Management
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> For flip-6, it has implemented the ZookeeperHaServices, but 
> ZookeeperHaServices does not support getRunningJobsRegistry. So need to 
> implement a ZK based running job registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)