[jira] [Commented] (TWILL-264) Discovery.hashCode implementation is wrong

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752765#comment-16752765
 ] 

ASF GitHub Bot commented on TWILL-264:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/80


> Discovery.hashCode implementation is wrong
> --
>
> Key: TWILL-264
> URL: https://issues.apache.org/jira/browse/TWILL-264
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
>Priority: Major
>
> The equals() method compare the payload array content, but the hashCode is 
> computed based on the payload reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-264) Discovery.hashCode implementation is wrong

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752554#comment-16752554
 ] 

ASF GitHub Bot commented on TWILL-264:
--

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/80#discussion_r251089608
  
--- Diff: 
twill-discovery-api/src/test/java/org/apache/twill/discovery/DiscoverableTest.java
 ---
@@ -0,0 +1,36 @@
+package org.apache.twill.discovery;
--- End diff --

please add copyright


> Discovery.hashCode implementation is wrong
> --
>
> Key: TWILL-264
> URL: https://issues.apache.org/jira/browse/TWILL-264
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
>Priority: Major
>
> The equals() method compare the payload array content, but the hashCode is 
> computed based on the payload reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-264) Discovery.hashCode implementation is wrong

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752521#comment-16752521
 ] 

ASF GitHub Bot commented on TWILL-264:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/80

(TWILL-264) Fix Discoverable.hashCode implementation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill TWILL-264-fix-discoverable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/80.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #80


commit b4feee26f827fe4c8f9f667f9af05edd83305a47
Author: Terence Yim 
Date:   2019-01-25T18:00:00Z

(TWILL-264) Fix Discoverable.hashCode implementation




> Discovery.hashCode implementation is wrong
> --
>
> Key: TWILL-264
> URL: https://issues.apache.org/jira/browse/TWILL-264
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
>Priority: Major
>
> The equals() method compare the payload array content, but the hashCode is 
> computed based on the payload reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719298#comment-16719298
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/71


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653178#comment-16653178
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r225824325
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
--- End diff --


https://github.com/apache/hadoop/commit/f02ca4ab158aa2257e839a1f74bc8254e1a3d61b#diff-f42f0d7039ba6a4ce427c67ae74dd7ec

HDFS-8185 Separate client related routines in HAUtil into a new clas also 
said this.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653160#comment-16653160
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r225817747
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
--- End diff --


https://github.com/apache/hadoop/commit/d12a0a25b4da1957ba0f658694729d2f2d400c5f#diff-f42f0d7039ba6a4ce427c67ae74dd7ec

HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client said 
this.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640720#comment-16640720
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r223182267
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem 
getFileSystem(LocationFactory locationFactory) throws
 return null;
   }
 
+  private static void handleLogAction(Exception e) {
--- End diff --

add new code, modify the code as you suggested and add the HAutils method 
judgement. 


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640721#comment-16640721
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on the issue:

https://github.com/apache/twill/pull/71
  
add new code, modify the code as you suggested and add the HAutils method 
judgement. 


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640715#comment-16640715
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r223181503
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
--- End diff --

ok


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640714#comment-16640714
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r223181498
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+  }
+
+  private static Set>> 
invoke(Configuration config) {
--- End diff --

ok


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640713#comment-16640713
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r223181489
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+  }
+
+  private static Set>> 
invoke(Configuration config) {
+try {
+  return ((Map) getHaNnRpcAddressesMethod.invoke(null, 
config)).entrySet();
--- End diff --

getHaNnRpcAddressesMethod invoke return Map instance.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640708#comment-16640708
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r223181337
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -72,7 +77,23 @@
 HADOOP_26
   }
 
-  private static final Logger LOG = 
LoggerFactory.getLogger(YarnUtils.class);
+  private static boolean iDFSUtilClientExists = false; // use this to 
judge if the hadoop version is above 2.8
--- End diff --

ok， suddenly find HAUtil class does not exists as the DFSUtils class. i 
will add the two judge into the code and make the code more nice as you 
commented.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634233#comment-16634233
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221661448
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
--- End diff --

This is not a very clear javadoc that describe what this method does (it 
should have the same javadoc as the `DFSUtil.getHaNnRpcAddresses` method).


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634235#comment-16634235
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221660130
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+  }
+
+  private static Set>> 
invoke(Configuration config) {
--- End diff --

Please name this method with a more appropriate name, rather than a generic 
name `invoke`.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634234#comment-16634234
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221660462
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
--- End diff --

Use two asterisks instead of three `/**`


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634238#comment-16634238
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221661663
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem 
getFileSystem(LocationFactory locationFactory) throws
 return null;
   }
 
+  private static void handleLogAction(Exception e) {
--- End diff --

This method is unnecessary, better log it in place. E.g. in the static 
initializer above:

```java
static {
  try {
Class dfsUtilsClientClazz = 
Class.forName("org.apache.hadoop.hdfs.DFSUtilClient");
getHaNnRpcAddressesMethod = 
dfsUtilsClientClazz.getMethod("getHaNnRpcAddresses",
Configuration.class);
hasDFSUtilClient = true;
  } catch (ClassNotFoundException e) {
// Expected for Hadoop version < 2.8, hence log it as debug only to no 
polluting the logs
LOG.debug("No DFSUtilClient found", e);
  } catch (NoSuchMethodException e) {
// This is unexpected for not founding the getHaNnRpcAddresses method 
if the DFSUtilClient class exists
LOG.warn("No DFSUtilClient.getHaNnRpcAddresses method found. Getting HA 
NameNode address might fail.", e);
  }
}
```


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634237#comment-16634237
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221663573
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -72,7 +77,23 @@
 HADOOP_26
   }
 
-  private static final Logger LOG = 
LoggerFactory.getLogger(YarnUtils.class);
+  private static boolean iDFSUtilClientExists = false; // use this to 
judge if the hadoop version is above 2.8
--- End diff --

What does the `i` prefix mean? Better just call this `hasDFSUtilClient`.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634239#comment-16634239
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221660324
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
--- End diff --

Please name this method with a more specific name, such as 
`getHaNnRpcAddress`.


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634236#comment-16634236
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/71#discussion_r221661290
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration 
config) throws IOException
 }
   }
 
+  /***
+   * When hadoop_version > 2.8.0, class DFSUtils has no method 
getHaNnRpcAddresses(Configuration config)
+   * @param config
+   * @return
+   */
+  private static Set>> 
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+  }
+
+  private static Set>> 
invoke(Configuration config) {
+try {
+  return ((Map) getHaNnRpcAddressesMethod.invoke(null, 
config)).entrySet();
--- End diff --

Why casting to `Map` while the return type of this method is `Set`??


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632842#comment-16632842
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama commented on the issue:

https://github.com/apache/twill/pull/70
  
seems code style check failed,will try to open a new pull request


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632843#comment-16632843
 ] 

ASF GitHub Bot commented on TWILL-262:
--

Github user piaosama closed the pull request at:

https://github.com/apache/twill/pull/70


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632840#comment-16632840
 ] 

ASF GitHub Bot commented on TWILL-262:
--

GitHub user piaosama opened a pull request:

https://github.com/apache/twill/pull/70

TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, 
which is removed from DFSUtils from hadoop-2.8

TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, 
which is removed from DFSUtils from hadoop-2.8

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/piaosama/twill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/70.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #70


commit 892fdc172522aa67d1581b324503fddc912e839c
Author: lihongyuan 
Date:   2018-09-29T06:33:05Z

TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, 
which is removed from DFSUtils from hadoop-2.8




> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> 
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is 
> removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws 
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple 
> name node associated with it.
> for (Map.Entry> entry : 
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>   String nsId = entry.getKey();
>   Map addressesInNN = entry.getValue();
>   if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || 
> addressesInNN.isEmpty()) {
> continue;
>   }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-258) Use getLoopbackAddress() instead of getLocalHost() for getting local host for ZK server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406929#comment-16406929
 ] 

ASF GitHub Bot commented on TWILL-258:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/68


> Use getLoopbackAddress() instead of getLocalHost() for getting local host for 
> ZK server
> ---
>
> Key: TWILL-258
> URL: https://issues.apache.org/jira/browse/TWILL-258
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.13.0
>
>
> This makes tests agnostic to local network configurations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-258) Use getLoopbackAddress() instead of getLocalHost() for getting local host for ZK server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406923#comment-16406923
 ] 

ASF GitHub Bot commented on TWILL-258:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/68
  
Updated commit message to include more details about the fixes.


> Use getLoopbackAddress() instead of getLocalHost() for getting local host for 
> ZK server
> ---
>
> Key: TWILL-258
> URL: https://issues.apache.org/jira/browse/TWILL-258
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
>
> This makes tests agnostic to local network configurations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-258) Use getLoopbackAddress() instead of getLocalHost() for getting local host for ZK server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406859#comment-16406859
 ] 

ASF GitHub Bot commented on TWILL-258:
--

Github user anew commented on the issue:

https://github.com/apache/twill/pull/68
  
Can you change the commit message to mention the race condition fixes?


> Use getLoopbackAddress() instead of getLocalHost() for getting local host for 
> ZK server
> ---
>
> Key: TWILL-258
> URL: https://issues.apache.org/jira/browse/TWILL-258
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
>
> This makes tests agnostic to local network configurations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-244) Provide access to ResourceReport to TwillRunnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404505#comment-16404505
 ] 

ASF GitHub Bot commented on TWILL-244:
--

Github user chtyim closed the pull request at:

https://github.com/apache/twill/pull/61


> Provide access to ResourceReport to TwillRunnable
> -
>
> Key: TWILL-244
> URL: https://issues.apache.org/jira/browse/TWILL-244
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Terence Yim
>Priority: Major
>
> One of the use case for this is to let TwillRunnable know about what 
> containers are running at the moment, which could be useful when building 
> fencing mechanism for single-master + HA type of applications.
> This information can be available via the TwillContext object.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-258) Use getLoopbackAddress() instead of getLocalHost() for getting local host for ZK server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404503#comment-16404503
 ] 

ASF GitHub Bot commented on TWILL-258:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/68

(TWILL-258) Use loopback address for ZK server

- Fix a race condition in the LocationCacheTest
- Fix race condition in LogLevelChangeTestRun

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-258

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit 0172a5d1f549ea2485fdce35034d1dec61a26ba3
Author: Terence Yim 
Date:   2018-03-19T07:28:20Z

(TWILL-258) Use loopback address for ZK server

- Fix a race condition in the LocationCacheTest
- Fix race condition in LogLevelChangeTestRun




> Use getLoopbackAddress() instead of getLocalHost() for getting local host for 
> ZK server
> ---
>
> Key: TWILL-258
> URL: https://issues.apache.org/jira/browse/TWILL-258
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
>
> This makes tests agnostic to local network configurations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399632#comment-16399632
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174644467
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterMain.java
 ---
@@ -170,45 +167,7 @@ private ApplicationKafkaService(ZKClient zkClient, 
String kafkaZKConnect) {
 protected void startUp() throws Exception {
   // Create the ZK node for Kafka to use. If the node already exists, 
delete it to make sure there is
   // no left over content from previous AM attempt.
-  final SettableOperationFuture completion = 
SettableOperationFuture.create(kafkaZKPath,
-   
 Threads.SAME_THREAD_EXECUTOR);
-  LOG.info("Preparing Kafka ZK path {}{}", 
zkClient.getConnectString(), kafkaZKPath);
--- End diff --

oh yeah, accidentally removed.


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399626#comment-16399626
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174643542
  
--- Diff: 
twill-zookeeper/src/main/java/org/apache/twill/zookeeper/ZKOperations.java ---
@@ -281,6 +286,73 @@ public void onFailure(Throwable t) {
 return resultFuture;
   }
 
+  /**
+   * Creates a ZK node of the given path. If the node already exists, 
deletion of the node (recursively) will happen
+   * and the creation will be retried.
+   */
+  public static OperationFuture createDeleteIfExists(final 
ZKClient zkClient, final String path,
+ @Nullable 
final byte[] data, final CreateMode createMode,
+ final boolean 
createParent, final ACL...acls) {
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(path,
+   
 Threads.SAME_THREAD_EXECUTOR);
+final List createACLs = acls.length == 0 ? 
ZooDefs.Ids.OPEN_ACL_UNSAFE : Arrays.asList(acls);
+createNode(zkClient, path, data, createMode, createParent, createACLs, 
new FutureCallback() {
+
+  final FutureCallback createCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+// Create succeeded, just set the result to the resultFuture
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+// If create failed not because of the NodeExistsException, just 
set the exception to the result future
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+  return;
+}
+
+// Try to delete the path
+LOG.info("Node {}{} already exists. Deleting it and retry 
creation", zkClient.getConnectString(), path);
+Futures.addCallback(recursiveDelete(zkClient, path), new 
FutureCallback() {
+  @Override
+  public void onSuccess(String result) {
+// If delete succeeded, perform the creation again.
+createNode(zkClient, path, data, createMode, createParent, 
createACLs, createCallback);
+  }
+
+  @Override
+  public void onFailure(Throwable t) {
+// If deletion failed because of NoNodeException, fail the 
result operation future
+if (!(t instanceof KeeperException.NoNodeException)) {
+  createFailure.addSuppressed(t);
+  resultFuture.setException(createFailure);
+  return;
+}
+
+// If can't delete because the node no longer exists, just go 
ahead and recreate the node
+createNode(zkClient, path, data, createMode, createParent, 
createACLs, createCallback);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+  }
+});
+
+return resultFuture;
+  }
+
+  /**
+   * Private helper method to create a ZK node based on the parameter. The 
result of the creation is always
+   * communicate via the provided {@link FutureCallback}.
+   */
+  private static void createNode(ZKClient zkClient, String path, @Nullable 
byte[] data,
+ CreateMode createMode, boolean 
createParent,
+ Iterable acls, 
FutureCallback callback) {
+Futures.addCallback(zkClient.create(path, data, createMode, 
createParent, acls),
+callback, Threads.SAME_THREAD_EXECUTOR);
+  }
--- End diff --

yes, easier to understand and extracted into separate methods. Like it much 
better now 


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399625#comment-16399625
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174643070
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterMain.java
 ---
@@ -170,45 +167,7 @@ private ApplicationKafkaService(ZKClient zkClient, 
String kafkaZKConnect) {
 protected void startUp() throws Exception {
   // Create the ZK node for Kafka to use. If the node already exists, 
delete it to make sure there is
   // no left over content from previous AM attempt.
-  final SettableOperationFuture completion = 
SettableOperationFuture.create(kafkaZKPath,
-   
 Threads.SAME_THREAD_EXECUTOR);
-  LOG.info("Preparing Kafka ZK path {}{}", 
zkClient.getConnectString(), kafkaZKPath);
--- End diff --

maybe keep the log message?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399593#comment-16399593
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174639182
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
+// Try to delete the node and recreate it
+LOG.info("Live node already exist. Deleting node {}{}", 
zkClient.getConnectString(), liveNodePath);
+Futures.addCallback(zkClient.delete(liveNodePath), new 
FutureCallback() {
+  @Override
+  public void onSuccess(String result) {
+Futures.addCallback(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
+thisCallback, 
Threads.SAME_THREAD_EXECUTOR);
+  }
+
+  @Override
+  public void onFailure(Throwable t) {
+createFailure.addSuppressed(t);
+resultFuture.setException(createFailure);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+
+return resultFuture;
--- End diff --

I was thinking it does not have to be async. Create the ZK node, get the 
future. If success, done. If not delete the ZK node, get the future. If 
failure, throw. Else try again. But maybe that would be equally complex?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399591#comment-16399591
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174638923
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
--- End diff --

got it


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399531#comment-16399531
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/67
  
I've addressed the comments and fixed one more issue (one deletion failure, 
if the node not exist, we can just go ahead and recreate the node instead of 
failing).

 Also refactored the callback code a bit to try to make it cleaner.


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399527#comment-16399527
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174626043
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/kafka/client/SimpleKafkaPublisher.java
 ---
@@ -159,30 +159,37 @@ public void changed(BrokerService brokerService) {
   }
 
   String newBrokerList = brokerService.getBrokerList();
-  if (newBrokerList.isEmpty()) {
-LOG.warn("Broker list is empty. No Kafka producer is created.");
-return;
-  }
 
+  // If there is no change, whether it is empty or not, just return
   if (Objects.equal(brokerList, newBrokerList)) {
 return;
   }
 
-  Properties props = new Properties();
-  props.put("metadata.broker.list", newBrokerList);
-  props.put("serializer.class", ByteBufferEncoder.class.getName());
-  props.put("key.serializer.class", IntegerEncoder.class.getName());
-  props.put("partitioner.class", IntegerPartitioner.class.getName());
-  props.put("request.required.acks", Integer.toString(ack.getAck()));
-  props.put("compression.codec", compression.getCodec());
+  Producer newProducer = null;
+  if (!newBrokerList.isEmpty()) {
+Properties props = new Properties();
+props.put("metadata.broker.list", newBrokerList);
+props.put("serializer.class", ByteBufferEncoder.class.getName());
+props.put("key.serializer.class", IntegerEncoder.class.getName());
+props.put("partitioner.class", IntegerPartitioner.class.getName());
+props.put("request.required.acks", Integer.toString(ack.getAck()));
+props.put("compression.codec", compression.getCodec());
+
+ProducerConfig config = new ProducerConfig(props);
+newProducer = new Producer<>(config);
+  }
 
-  ProducerConfig config = new ProducerConfig(props);
-  Producer oldProducer = producer.getAndSet(new 
Producer(config));
+  // If the broker list is empty, the producer will be set to null
+  Producer oldProducer = 
producer.getAndSet(newProducer);
   if (oldProducer != null) {
 oldProducer.close();
   }
 
-  LOG.info("Update Kafka producer broker list: {}", newBrokerList);
+  if (newBrokerList.isEmpty()) {
+LOG.warn("Empty Kafka producer broker list, publish will fail.");
--- End diff --

Yes


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399470#comment-16399470
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174618874
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
--- End diff --

That's right. Missed it.


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399466#comment-16399466
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174618395
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
+// Try to delete the node and recreate it
+LOG.info("Live node already exist. Deleting node {}{}", 
zkClient.getConnectString(), liveNodePath);
+Futures.addCallback(zkClient.delete(liveNodePath), new 
FutureCallback() {
+  @Override
+  public void onSuccess(String result) {
+Futures.addCallback(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
+thisCallback, 
Threads.SAME_THREAD_EXECUTOR);
+  }
+
+  @Override
+  public void onFailure(Throwable t) {
+createFailure.addSuppressed(t);
+resultFuture.setException(createFailure);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+
+return resultFuture;
--- End diff --

I can pull the common code between this and the ApplicationMasterMain class 
into a util function. But still, inside the util function, there would be three 
callbacks.


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399465#comment-16399465
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174618087
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
+// Try to delete the node and recreate it
+LOG.info("Live node already exist. Deleting node {}{}", 
zkClient.getConnectString(), liveNodePath);
+Futures.addCallback(zkClient.delete(liveNodePath), new 
FutureCallback() {
+  @Override
+  public void onSuccess(String result) {
+Futures.addCallback(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
+thisCallback, 
Threads.SAME_THREAD_EXECUTOR);
+  }
+
+  @Override
+  public void onFailure(Throwable t) {
+createFailure.addSuppressed(t);
+resultFuture.setException(createFailure);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+
+return resultFuture;
--- End diff --

Well, we do need that many levels of callback (create -> delete -> create) 
for the operation. Any suggestions on how to simplify it?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399462#comment-16399462
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174617783
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
--- End diff --

Ephemeral node won't go away immediate if the process die. It will stay 
there till ZK session timeout, which is typically multiple seconds. In the 
meantime, the next AM process may already be started by YARN, hence the new AM 
process will see the ephemeral node.


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399162#comment-16399162
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174573795
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
+// Try to delete the node and recreate it
+LOG.info("Live node already exist. Deleting node {}{}", 
zkClient.getConnectString(), liveNodePath);
+Futures.addCallback(zkClient.delete(liveNodePath), new 
FutureCallback() {
+  @Override
+  public void onSuccess(String result) {
+Futures.addCallback(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
+thisCallback, 
Threads.SAME_THREAD_EXECUTOR);
+  }
+
+  @Override
+  public void onFailure(Throwable t) {
+createFailure.addSuppressed(t);
+resultFuture.setException(createFailure);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+  }
+}, Threads.SAME_THREAD_EXECUTOR);
+
+return resultFuture;
--- End diff --

while this code appears correct (after addressing my comment), the three 
nested levels of callback make it almost impossible to read. Is there some way 
to unwind this?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399160#comment-16399160
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174571435
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
--- End diff --

do you need to return here?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399163#comment-16399163
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174580690
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/kafka/client/SimpleKafkaPublisher.java
 ---
@@ -159,30 +159,37 @@ public void changed(BrokerService brokerService) {
   }
 
   String newBrokerList = brokerService.getBrokerList();
-  if (newBrokerList.isEmpty()) {
-LOG.warn("Broker list is empty. No Kafka producer is created.");
-return;
-  }
 
+  // If there is no change, whether it is empty or not, just return
   if (Objects.equal(brokerList, newBrokerList)) {
 return;
   }
 
-  Properties props = new Properties();
-  props.put("metadata.broker.list", newBrokerList);
-  props.put("serializer.class", ByteBufferEncoder.class.getName());
-  props.put("key.serializer.class", IntegerEncoder.class.getName());
-  props.put("partitioner.class", IntegerPartitioner.class.getName());
-  props.put("request.required.acks", Integer.toString(ack.getAck()));
-  props.put("compression.codec", compression.getCodec());
+  Producer newProducer = null;
+  if (!newBrokerList.isEmpty()) {
+Properties props = new Properties();
+props.put("metadata.broker.list", newBrokerList);
+props.put("serializer.class", ByteBufferEncoder.class.getName());
+props.put("key.serializer.class", IntegerEncoder.class.getName());
+props.put("partitioner.class", IntegerPartitioner.class.getName());
+props.put("request.required.acks", Integer.toString(ack.getAck()));
+props.put("compression.codec", compression.getCodec());
+
+ProducerConfig config = new ProducerConfig(props);
+newProducer = new Producer<>(config);
+  }
 
-  ProducerConfig config = new ProducerConfig(props);
-  Producer oldProducer = producer.getAndSet(new 
Producer(config));
+  // If the broker list is empty, the producer will be set to null
+  Producer oldProducer = 
producer.getAndSet(newProducer);
   if (oldProducer != null) {
 oldProducer.close();
   }
 
-  LOG.info("Update Kafka producer broker list: {}", newBrokerList);
+  if (newBrokerList.isEmpty()) {
+LOG.warn("Empty Kafka producer broker list, publish will fail.");
--- End diff --

So when will this happen? If the AM dies (and its broker with it)? 


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399161#comment-16399161
 ] 

ASF GitHub Bot commented on TWILL-61:
-

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/67#discussion_r174571807
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java ---
@@ -216,11 +217,52 @@ protected final void shutDown() throws Exception {
 return zkClient.setData(liveNodePath, serializeLiveNode());
   }
 
+  /**
+   * Creates the live node for the service. If the node already exists, it 
will be deleted before creation.
+   *
+   * @return A {@link OperationFuture} that will be completed when the 
creation is done.
+   */
   private OperationFuture createLiveNode() {
-String liveNodePath = getLiveNodePath();
+final String liveNodePath = getLiveNodePath();
 LOG.info("Create live node {}{}", zkClient.getConnectString(), 
liveNodePath);
-return ZKOperations.ignoreError(zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL),
-
KeeperException.NodeExistsException.class, liveNodePath);
+final SettableOperationFuture resultFuture = 
SettableOperationFuture.create(liveNodePath,
+   
 Threads.SAME_THREAD_EXECUTOR);
+OperationFuture createFuture = zkClient.create(liveNodePath, 
serializeLiveNode(), CreateMode.EPHEMERAL);
+Futures.addCallback(createFuture, new FutureCallback() {
+  final FutureCallback thisCallback = this;
+
+  @Override
+  public void onSuccess(String result) {
+LOG.info("Live node created {}{}", zkClient.getConnectString(), 
liveNodePath);
+resultFuture.set(result);
+  }
+
+  @Override
+  public void onFailure(final Throwable createFailure) {
+if (!(createFailure instanceof 
KeeperException.NodeExistsException)) {
+  resultFuture.setException(createFailure);
+}
+
+// If the node already exists, it is due to previous run attempt 
that left an ephemeral node.
--- End diff --

how can it leave an ephemeral node? Doesn't that mean there must be a 
zombie process holding on to that node?


> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393689#comment-16393689
 ] 

ASF GitHub Bot commented on TWILL-61:
-

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/67

(TWILL-61) Fix to allow higher attempts to relaunch the app after the first 
attempt failed

- Delete the Kafka root zk node for the application if already exist
- Delete the AM instance zk node if already exist
- For runnables parent zk node, it is not an error if it already exist
- Enhance KafkaClient publisher / consumer to deal with Kafka cluster 
changes
  - When AM killed and restarted, the embedded Kafka will be running in 
different host and port

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-61

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/67.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #67


commit 9decca071a9e067b30be2150a6097463c939b6af
Author: Terence Yim 
Date:   2018-03-09T20:21:26Z

(TWILL-61) Fix to allow higher attempts to relaunch the app after the first 
attempt failed

- Delete the Kafka root zk node for the application if already exist
- Delete the AM instance zk node if already exist
- For runnables parent zk node, it is not an error if it already exist
- Enhance KafkaClient publisher / consumer to deal with Kafka cluster 
changes
  - When AM killed and restarted, the embedded Kafka will be running in 
different host and port




> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-255) incorrect logging after memory/cpu was adjusted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382400#comment-16382400
 ] 

ASF GitHub Bot commented on TWILL-255:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/66


> incorrect logging after memory/cpu was adjusted
> ---
>
> Key: TWILL-255
> URL: https://issues.apache.org/jira/browse/TWILL-255
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>Priority: Minor
> Fix For: 0.13.0
>
>
> While adjusting resources for Containers when logging what was adjusted it 
> shows values after adjustment, so it's not known what it was adjusted from.
> Affected are: adjustCapability()
> Hadoop20YarnAMClient
> Hadoop21YarnAMClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-255) incorrect logging after memory/cpu was adjusted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381313#comment-16381313
 ] 

ASF GitHub Bot commented on TWILL-255:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/66
  
+1
LGTM


> incorrect logging after memory/cpu was adjusted
> ---
>
> Key: TWILL-255
> URL: https://issues.apache.org/jira/browse/TWILL-255
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>Priority: Minor
>
> While adjusting resources for Containers when logging what was adjusted it 
> shows values after adjustment, so it's not known what it was adjusted from.
> Affected are: adjustCapability()
> Hadoop20YarnAMClient
> Hadoop21YarnAMClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-255) incorrect logging after memory/cpu was adjusted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381170#comment-16381170
 ] 

ASF GitHub Bot commented on TWILL-255:
--

GitHub user yufeldman opened a pull request:

https://github.com/apache/twill/pull/66

(TWILL-255) Incorrect logging after memory was adjusted. Does not sho…

…w memory before adjustment

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yufeldman/twill branch-TWILL-255

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/66.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #66


commit 5617478fc2ebf315ada125e1d3242f53c8545987
Author: Yuliya Feldman 
Date:   2018-02-28T22:52:56Z

(TWILL-255) Incorrect logging after memory was adjusted. Does not show 
memory before adjustment




> incorrect logging after memory/cpu was adjusted
> ---
>
> Key: TWILL-255
> URL: https://issues.apache.org/jira/browse/TWILL-255
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>Priority: Minor
>
> While adjusting resources for Containers when logging what was adjusted it 
> shows values after adjustment, so it's not known what it was adjusted from.
> Affected are: adjustCapability()
> Hadoop20YarnAMClient
> Hadoop21YarnAMClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353178#comment-16353178
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/65


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
> Fix For: 0.13.0
>
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353176#comment-16353176
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/65
  
Changes LGTM.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353173#comment-16353173
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r166160088
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

Yes, that's correct.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353027#comment-16353027
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r166133525
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

@chtyim You are asking me to leave the 2.5 profile alone and add the 2.6 
profile?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353022#comment-16353022
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Hi @chtyim, thank you for the review. Could you please expand on what you 
see concerning in the `pom.xml`? For reference, I need to provide a 2.6 target 
as it is the first version to provide the `ContainerId.fromString()` API.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352347#comment-16352347
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165965620
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

Ah, okay then.  sorry, I missed that.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351984#comment-16351984
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Thank you for the super fast code review @serranom! I think all comments 
have been addressed and the code should be complete now. Please let me know if 
you see anything else necessary. It is great working with Twill! Hopefully I 
can provide some other ideas as I work with it.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351925#comment-16351925
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862926
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

If marked `final`, I can not extend it for 
[`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351924#comment-16351924
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862857
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

Thank you


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351922#comment-16351922
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862673
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

I didn't see any code specific to 2.5 that 2.3 did not provide?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351800#comment-16351800
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845936
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

this should be protected not private


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351797#comment-16351797
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845875
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

why are you eliminating the hadoop-2.5 profile?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351798#comment-16351798
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845855
  
--- Diff: pom.xml ---
@@ -171,7 +171,7 @@
 4.11
 3.2
 1.5
-[2.0.2-alpha,2.3.0]
+2.7.2
 target/hadoop20-classes
--- End diff --

is there any reason to keep this property at all?  seems better to just 
have it defined in each profile as necessary.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351799#comment-16351799
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845929
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

why remove final?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351654#comment-16351654
 ] 

ASF GitHub Bot commented on TWILL-254:
--

GitHub user cbaenziger opened a pull request:

https://github.com/apache/twill/pull/65

(TWILL-254) Update to use ContainerId.fromString

This moves away from the deprecated `ConverterUitls.toContainerId` and 
updates the build to use Hadoop 2.7.2 to avoid folks grabbing master and having 
a failure trying to run on clusters with code newer than November 18th, 2014 
(release of Hadoop 2.6.0).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cbaenziger/twill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/65.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #65


commit 391553308ccc0539b72fac6bf13cd819ce3bb5e1
Author: Clay Baenziger 
Date:   2018-02-03T03:35:37Z

(TWILL-254) Update to use ContainerId.fromString




> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277190#comment-16277190
 ] 

ASF GitHub Bot commented on TWILL-251:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/64


> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274843#comment-16274843
 ] 

ASF GitHub Bot commented on TWILL-251:
--

Github user gsps1 commented on the issue:

https://github.com/apache/twill/pull/64
  
LGTM


> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274815#comment-16274815
 ] 

ASF GitHub Bot commented on TWILL-251:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/64

(TWILL-251) Reduce log level of YarnNMClient

- Also reduce the polling frequency

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-251-reduce-log

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/64.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #64


commit 86a769db37650ff199ea5183f27c32dc0aa4cee3
Author: Terence Yim 
Date:   2017-12-01T19:21:07Z

(TWILL-251) Reduce log level of YarnNMClient

- Also reduce the polling frequency




> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227203#comment-16227203
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/63


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225851#comment-16225851
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/62


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225843#comment-16225843
 ] 

ASF GitHub Bot commented on TWILL-248:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/63

(TWILL-248) Speedup shutdown of tracker service



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-248-speedup-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/63.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #63


commit ef87c6210f01b5d4c24193b4e808220973e89cdc
Author: Terence Yim 
Date:   2017-10-30T22:10:34Z

(TWILL-248) Speedup shutdown of tracker service




> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199368#comment-16199368
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/62#discussion_r143854276
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/ResourceReportClient.java ---
@@ -52,12 +54,16 @@
   public ResourceReport get() {
 for (URL url : resourceUrls) {
   try {
-Reader reader = new BufferedReader(new 
InputStreamReader(url.openStream(), Charsets.UTF_8));
-try {
+HttpURLConnection urlConn = (HttpURLConnection) 
url.openConnection();
--- End diff --

Not entirely. This is to enable compression when fetching the report.


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199328#comment-16199328
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/62
  
One question about change form using BufferedReader,  but looks good.

If passes tests +1


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199326#comment-16199326
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user hsaputra commented on a diff in the pull request:

https://github.com/apache/twill/pull/62#discussion_r143848112
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/ResourceReportClient.java ---
@@ -52,12 +54,16 @@
   public ResourceReport get() {
 for (URL url : resourceUrls) {
   try {
-Reader reader = new BufferedReader(new 
InputStreamReader(url.openStream(), Charsets.UTF_8));
-try {
+HttpURLConnection urlConn = (HttpURLConnection) 
url.openConnection();
--- End diff --

Is this change related to Netty upgrade?


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199301#comment-16199301
 ] 

ASF GitHub Bot commented on TWILL-248:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/62

(TWILL-248) Upgrade to use Netty-4.1

- Also enable ResourceReportClient to use HTTP compression

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-248-upgrade-netty-4.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/62.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #62


commit 321e7ba0116ccb7ba1c9b814ff60ad0dfd4ac3e5
Author: Terence Yim 
Date:   2017-10-10T20:26:11Z

(TWILL-248) Upgrade to use Netty-4.1

- Also enable ResourceReportClient to use HTTP compression




> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-244) Provide access to ResourceReport to TwillRunnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149820#comment-16149820
 ] 

ASF GitHub Bot commented on TWILL-244:
--

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/61#discussion_r136476455
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/ResourceReportClient.java ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.internal;
+
+import com.google.common.base.Charsets;
+import org.apache.twill.api.ResourceReport;
+import org.apache.twill.api.ResourceReporter;
+import org.apache.twill.api.TwillRunResources;
+import org.apache.twill.internal.json.ResourceReportAdapter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.lang.reflect.Type;
+import java.net.URL;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+/**
+ * A {@link ResourceReporter} that fetches reports from the given set of 
URLs.
+ */
+public final class ResourceReportClient implements ResourceReporter {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(ResourceReportClient.class);
+
+  private final List resourceUrls;
+
+  public ResourceReportClient(List resourceUrls) {
+this.resourceUrls = resourceUrls;
+  }
+
+  @Nullable
+  @Override
+  public ResourceReport getResourceReport() {
+for (URL url : resourceUrls) {
+  try {
+return fetchURL(url, "", ResourceReport.class);
+  } catch (IOException e) {
+// Just log a trace as it's ok to not able to fetch resource report
+LOG.trace("Exception raised when getting resource report from 
{}.", url, e);
+  }
+}
--- End diff --

even if all urls fail to fetch - why is that ok?


> Provide access to ResourceReport to TwillRunnable
> -
>
> Key: TWILL-244
> URL: https://issues.apache.org/jira/browse/TWILL-244
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Terence Yim
>
> One of the use case for this is to let TwillRunnable know about what 
> containers are running at the moment, which could be useful when building 
> fencing mechanism for single-master + HA type of applications.
> This information can be available via the TwillContext object.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-244) Provide access to ResourceReport to TwillRunnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144636#comment-16144636
 ] 

ASF GitHub Bot commented on TWILL-244:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/61

(TWILL-244) Make resource report available to TwillRunnable

- Make resource report available to TwillContext
  - Enhance tracker service to provide more resource related endpoints
- Provide endpoints to fetch specific part of the report to reduce 
bandwidth
- Added a ResourceReporter API to provide programmatic access
  - Make resource report available via TwillContext
- Fixed a concurrent modification bug in ZKDiscoveryService.close

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-244-resource-report-runnable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/61.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #61


commit 53312962b33cc20c2f272b217d076730c498644d
Author: Terence Yim 
Date:   2017-08-28T23:47:45Z

(TWILL-244) Make resource report available to TwillContext

- Enhance tracker service to provide more resource related endpoints
  - Provide endpoints to fetch specific part of the report to reduce
bandwidth
  - Added a ResourceReporter API to provide programmatic access
- Make resource report available via TwillContext

commit 31facdf2f7f93f06c8fd131960be6d4c52fd646e
Author: Terence Yim 
Date:   2017-08-29T01:33:53Z

Fix a concurrent modification bug in ZKDiscoveryService.close




> Provide access to ResourceReport to TwillRunnable
> -
>
> Key: TWILL-244
> URL: https://issues.apache.org/jira/browse/TWILL-244
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Terence Yim
>
> One of the use case for this is to let TwillRunnable know about what 
> containers are running at the moment, which could be useful when building 
> fencing mechanism for single-master + HA type of applications.
> This information can be available via the TwillContext object.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120944#comment-16120944
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/58


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120825#comment-16120825
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on the issue:

https://github.com/apache/twill/pull/58
  
squashed


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-242) Extra JVM options is dropped when custom classloader is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120799#comment-16120799
 ] 

ASF GitHub Bot commented on TWILL-242:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/60


> Extra JVM options is dropped when custom classloader is used
> 
>
> Key: TWILL-242
> URL: https://issues.apache.org/jira/browse/TWILL-242
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> It is introduced by the fix in TWILL-241



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120748#comment-16120748
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322898
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

I think so.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120746#comment-16120746
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322804
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

call it at the end of `doStop() `?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120745#comment-16120745
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322316
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

Shouldn't call `destroy()` here anymore. It should be called after call 
other calls to the event handler, as it is acted as a cleanup call.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120743#comment-16120743
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322087
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -181,19 +190,130 @@ public Reader getInput() throws IOException {
   }
 
   @SuppressWarnings("unchecked")
-  @Nullable
   private EventHandler createEventHandler(TwillSpecification twillSpec) 
throws ClassNotFoundException {
 // Should be able to load by this class ClassLoader, as they packaged 
in the same jar.
 EventHandlerSpecification handlerSpec = twillSpec.getEventHandler();
 if (handlerSpec == null) {
-  return null;
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
 }
 
 Class handlerClass = 
getClass().getClassLoader().loadClass(handlerSpec.getClassName());
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+// wrap all calls to the delegate EventHandler methods except 
initialize so that all errors will be caught
+return new EventHandler() {
+  private boolean initialized;
+
+  @Override
+  public void initialize(EventHandlerContext context) {
+delegate.initialize(context);
+initialized = true;
--- End diff --

Since any exception thrown from `initialize` will terminate the app, I 
think we don't need this, right?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-242) Extra JVM options is dropped when custom classloader is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120717#comment-16120717
 ] 

ASF GitHub Bot commented on TWILL-242:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/60

(TWILL-242) Fixed the dropping of extra jvm opts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-242-fix-extra-jvm-opts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/60.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #60


commit 7187ed966139fd05ce4a6a8d79b65feac34545ae
Author: Terence Yim 
Date:   2017-08-09T21:56:33Z

(TWILL-242) Fixed the dropping of extra jvm opts




> Extra JVM options is dropped when custom classloader is used
> 
>
> Key: TWILL-242
> URL: https://issues.apache.org/jira/browse/TWILL-242
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> It is introduced by the fix in TWILL-241



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120614#comment-16120614
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132298585
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
+}
+// wrap the delegate EventHandler so that all errors will be caught
+return new EventHandler() {
--- End diff --

originally, if `EventHandler#initialize` fails, the app will fail to start. 
Do we want to change this behavior?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120612#comment-16120612
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132297954
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -401,6 +409,24 @@ void stopAll() {
 // When we acquire this lock, all stopped runnables should have been 
cleaned up by handleCompleted() method
 containerLock.lock();
 try {
+  for (Map.Entry> entry 
: containers.rowMap().entrySet()) {
+String runnableName = entry.getKey();
+Collection containerInfos = 
containerStats.get(runnableName);
+for (Map.Entry 
containerControllerEntry : entry.getValue().entrySet()) {
+  boolean containerExist = false;
+  for (ContainerInfo containerInfo : containerInfos) {
+if 
(containerInfo.getId().equals(containerControllerEntry.getKey())) {
+  containerExist = true;
+  break;
+}
+  }
+  // Only call eventHandler.containerStopped if container is not 
removed by handleCompleted
+  if (containerExist) {
+eventHandler.containerStopped(runnableName, 
containerControllerEntry.getValue().getInstanceId(),
--- End diff --

you can move this inside the `for` loop and no need to use `containerExist`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120609#comment-16120609
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132297305
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
+}
+// wrap the delegate EventHandler so that all errors will be caught
+return new EventHandler() {
--- End diff --

You'll need to wrap the `initialize` as well. Also, I'd suggest if 
`initialize` throw exception, you'll just ignore all calls to other lifecycle 
methods to avoid excessive logging.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120603#comment-16120603
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132296198
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
--- End diff --

`Instances.newInstance` never return `null`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120602#comment-16120602
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132295847
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -181,7 +189,6 @@ public Reader getInput() throws IOException {
   }
 
   @SuppressWarnings("unchecked")
-  @Nullable
   private EventHandler createEventHandler(TwillSpecification twillSpec) 
throws ClassNotFoundException {
 // Should be able to load by this class ClassLoader, as they packaged 
in the same jar.
 EventHandlerSpecification handlerSpec = twillSpec.getEventHandler();
--- End diff --

if `handlerSpec` is `null`, you still want to return a no-op handler, right?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119110#comment-16119110
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132048851
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
+  if (exitStatus == null) {
--- End diff --

`controller.stopAndWait();` at line 269 blocks until `handleCompleted` 
method runs


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119073#comment-16119073
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132038521
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
--- End diff --

Should remove it from the exit status map, otherwise the map can keep 
growing in size.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119072#comment-16119072
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132037568
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -221,7 +229,8 @@ protected void doStart() throws Exception {
 
 // initialize the event handler, if it fails, it will fail the 
application.
 if (eventHandler != null) {
-  eventHandler.initialize(new 
BasicEventHandlerContext(twillSpec.getEventHandler()));
+  eventHandler.initialize(new 
BasicEventHandlerContext(twillRuntimeSpec, twillSpec.getEventHandler()));
--- End diff --

The `twillRuntimeSpec` already contains `twillSpec`. Seems unnecessary to 
pass in `twillSpec.getEventHandler()` as a separate parameter.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119074#comment-16119074
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132040093
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
+  if (exitStatus == null) {
--- End diff --

So this is for handling the case when `handleCompleted` was not called 
(e.g. the runnable container is not stopping after receiving the "stop" 
command), right? If that's the case, isn't that the exit status is always not 
there?

Also, do we need to do the similar logic to call event handler in the 
`stopAll` method?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119075#comment-16119075
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132038164
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -173,6 +183,9 @@ void start(String runnableName, ContainerInfo 
containerInfo, TwillContainerLaunc
 startSequence.addLast(runnableName);
   }
   containerChange.signalAll();
+  if (eventHandler != null) {
+eventHandler.containerLaunched(runnableName, instanceId, 
containerInfo.getId());
--- End diff --

How do we handle exception raised from event handler methods?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119022#comment-16119022
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132029465
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/EventHandlerTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableMap;
+import org.apache.twill.api.AbstractTwillRunnable;
+import org.apache.twill.api.EventHandler;
+import org.apache.twill.api.EventHandlerContext;
+import org.apache.twill.api.TwillApplication;
+import org.apache.twill.api.TwillController;
+import org.apache.twill.api.TwillSpecification;
+import org.apache.twill.api.logging.PrinterLogHandler;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Tests {@link EventHandler} methods
+ */
+public final class EventHandlerTest extends BaseYarnTest {
+  private static final Logger LOG = 
LoggerFactory.getLogger(EventHandlerTest.class);
+
+  @ClassRule
+  public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+  public static final String STARTED_FILE = "started_file";
+  public static final String RUN_FILE = "run_file";
+  public static final String CONTAINER_LAUNCHED_FOLDER = "launched_folder";
+  public static final String CONTAINER_STOPPED_FOLDER = "stopped_folder";
+  public static final String COMPLETED_FILE = "completed_file";
+  public static final String KILLED_FILE = "killed_file";
+  public static final String ABORTED_FILE = "aborted_file";
+
+  @Test
+  public void testComplete() throws InterruptedException, 
ExecutionException, TimeoutException, IOException {
+// Create a parent folder to be written by EventHandler
+File parentFolder = TMP_FOLDER.newFolder();
+parentFolder.setWritable(true, false);
+TwillController controller = getTwillRunner().prepare(new 
CompleteApplication(parentFolder.getAbsolutePath()))
+  .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out, 
true)))
+  .start();
+
+// Wait for the app to complete within 120 seconds.
+try {
+  controller.awaitTerminated(120, TimeUnit.SECONDS);
+  Set handlerFiles = new 
HashSet<>(Arrays.asList(parentFolder.list()));
+  Assert.assertEquals(5, handlerFiles.size());
+  // EventHandler#started() method should be called to create a file
+  Assert.assertTrue(handlerFiles.contains(STARTED_FILE));
+  // CompleteRunnable#run() method should be called to create a file 
after EventHandler#started() method is called
+  Assert.assertTrue(handlerFiles.contains(RUN_FILE));
+  // EventHandler#containerLaunched(String, int, String) method should 
be called to create a folder
+  Assert.assertTrue(handlerFiles.contains(CONTAINER_LAUNCHED_FOLDER));
+  // EventHandler#containerStopped(String, int, String, int) method 
should be called to create a folder
+  Assert.assertTrue(handlerFiles.contains(CONTAINER_STOPPED_FOLDER));
+  // Assert that containerLaunched and containerStopped are called for 
the 

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119021#comment-16119021
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132029365
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/EventHandlerTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableMap;
+import org.apache.twill.api.AbstractTwillRunnable;
+import org.apache.twill.api.EventHandler;
+import org.apache.twill.api.EventHandlerContext;
+import org.apache.twill.api.TwillApplication;
+import org.apache.twill.api.TwillController;
+import org.apache.twill.api.TwillSpecification;
+import org.apache.twill.api.logging.PrinterLogHandler;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Tests {@link EventHandler} methods
+ */
+public final class EventHandlerTest extends BaseYarnTest {
--- End diff --

Rename the test class to `EventHandlerTestRun` and add it to 
`YarnTestSuite`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117509#comment-16117509
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/59
  
@anew Thanks for the review.


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117383#comment-16117383
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user anew commented on the issue:

https://github.com/apache/twill/pull/59
  
LGTM


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117272#comment-16117272
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761743
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
--- End diff --

Should be called `getApplicationName()`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117276#comment-16117276
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761787
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
+
+  String getTwillAppRunId();
--- End diff --

Just call it `getRunId()`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117277#comment-16117277
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762365
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
+
+  String getTwillAppRunId();
--- End diff --

The run id should be of type `RunId`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117271#comment-16117271
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761843
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
--- End diff --

Also please add javadoc.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117274#comment-16117274
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762894
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ProvisionTimeoutTestRun.java ---
@@ -85,13 +105,29 @@ public TimeoutAction 
launchTimeout(Iterable timeoutEvents) {
 return TimeoutAction.recheck(10, TimeUnit.SECONDS);
   }
 }
+
+@Override
+public void aborted() {
+  try {
+new 
File(context.getSpecification().getConfigs().get("parentFolderPath") + 
File.separator
--- End diff --

Also, we are only testing abort? How about other lifecycle methods?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)