[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495353#comment-16495353
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5593


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.6.0
>
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495172#comment-16495172
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5593
  
Thanks for the reminder @yanghua. I forgot about it and will merge it now. 
Thanks again for your contributions. Flink wouldn't be what it is without you!


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493684#comment-16493684
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5593
  
hi @tillrohrmann can this PR been merged into master branch, so that we can 
close it?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493683#comment-16493683
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5593
  
hi @tillrohrmann can this PR been merged into master branch, so that we can 
close it?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381710#comment-16381710
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5593
  
@tillrohrmann yes I tested this feature in our inner YARN cluster, both 
yarn flink session and single job mode. It works fine.  


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381564#comment-16381564
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5593
  
Hi @tillrohrmann should do more work for this PR?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380239#comment-16380239
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171231898
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -1317,15 +1340,18 @@ public static ApplicationSubmissionContextReflector 
getInstance() {
private static final String APPLICATION_TAGS_METHOD_NAME = 
"setApplicationTags";
private static final String ATTEMPT_FAILURES_METHOD_NAME = 
"setAttemptFailuresValidityInterval";
private static final String KEEP_CONTAINERS_METHOD_NAME = 
"setKeepContainersAcrossApplicationAttempts";
+   private static final String NODE_LABEL_EXPRESSION_NAME = 
"setNodeLabelExpression";
 
private final Method applicationTagsMethod;
private final Method attemptFailuresValidityIntervalMethod;
private final Method keepContainersMethod;
+   private final Method nodeLabelExpressionMethod;
--- End diff --

Please mark the field `@Nullable`.


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380053#comment-16380053
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171187437
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -979,6 +989,16 @@ public ApplicationReport startAppMaster(
appContext.setApplicationType("Apache Flink");
appContext.setAMContainerSpec(amContainer);
appContext.setResource(capability);
+
+   if (nodeLabel != null) {
+   try {
+   Method method = 
appContext.getClass().getMethod("setNodeLabelExpression", String.class);
+   method.invoke(appContext, nodeLabel);
+   } catch (NoSuchMethodException e) {
+   LOG.warn("Ignoring node label setting because 
the version of YARN does not support it");
+   }
+   }
--- End diff --

@tillrohrmann  I know, I just find there is a inner static class 
`ApplicationSubmissionContextReflector`, it uses reflection to set fields for 
`ApplicationSubmissionContext` such as `setApplicationTags`. I think add this 
code segment to this class and wrap it to a method named 
`setApplicationNodeLabel` could be better.
What's your opinion?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380026#comment-16380026
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171182303
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -979,6 +989,16 @@ public ApplicationReport startAppMaster(
appContext.setApplicationType("Apache Flink");
appContext.setAMContainerSpec(amContainer);
appContext.setResource(capability);
+
+   if (nodeLabel != null) {
+   try {
+   Method method = 
appContext.getClass().getMethod("setNodeLabelExpression", String.class);
+   method.invoke(appContext, nodeLabel);
+   } catch (NoSuchMethodException e) {
+   LOG.warn("Ignoring node label setting because 
the version of YARN does not support it");
+   }
+   }
--- End diff --

Do you mean wrap this code segment to a new method named this?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380024#comment-16380024
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171182155
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -321,6 +323,14 @@ public void setZookeeperNamespace(String 
zookeeperNamespace) {
this.zookeeperNamespace = zookeeperNamespace;
}
 
+   public String getNodeLabel() {
+   return nodeLabel;
+   }
+
+   public void setNodeLabel(String nodeLabel) {
+   this.nodeLabel = nodeLabel;
+   }
--- End diff --

Yes, at some point we should change it in order to make it better 
maintainable. This will of course not be in the scope of this PR.


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380021#comment-16380021
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171181582
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -321,6 +323,14 @@ public void setZookeeperNamespace(String 
zookeeperNamespace) {
this.zookeeperNamespace = zookeeperNamespace;
}
 
+   public String getNodeLabel() {
+   return nodeLabel;
+   }
+
+   public void setNodeLabel(String nodeLabel) {
+   this.nodeLabel = nodeLabel;
+   }
--- End diff --

@tillrohrmann the setter for `nodelabel` I followed the example of 
`setZookeeperNamespace` and `setQueue`, it seems that the 
`AbstractYarnClusterDescriptor` constructor can not set this field if we do not 
change it. 


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379986#comment-16379986
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171177337
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -979,6 +989,16 @@ public ApplicationReport startAppMaster(
appContext.setApplicationType("Apache Flink");
appContext.setAMContainerSpec(amContainer);
appContext.setResource(capability);
+
+   if (nodeLabel != null) {
+   try {
+   Method method = 
appContext.getClass().getMethod("setNodeLabelExpression", String.class);
+   method.invoke(appContext, nodeLabel);
+   } catch (NoSuchMethodException e) {
+   LOG.warn("Ignoring node label setting because 
the version of YARN does not support it");
+   }
+   }
--- End diff --

Could we wrap this call in something like 
`RegisterApplicationMasterResponseReflector`?


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379985#comment-16379985
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171176203
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -321,6 +323,14 @@ public void setZookeeperNamespace(String 
zookeeperNamespace) {
this.zookeeperNamespace = zookeeperNamespace;
}
 
+   public String getNodeLabel() {
+   return nodeLabel;
+   }
+
+   public void setNodeLabel(String nodeLabel) {
+   this.nodeLabel = nodeLabel;
+   }
--- End diff --

We should get rid of this setter pattern. All values should be set at 
creation time of the `AbstractYarnClusterDescriptor`.


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379984#comment-16379984
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5593#discussion_r171176869
  
--- Diff: docs/ops/cli.md ---
@@ -262,6 +262,8 @@ Action "run" compiles and runs a program.
  -z,--zookeeperNamespaceNamespace to create the
 Zookeeper sub-paths 
for high
 availability mode
+ -nl,--nodeLabel   Specify YARN node 
label for 
--- End diff --

I think this should only be a Yarn specific option and not a general 
option. Mesos and other environments don't necessarily support this.


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379780#comment-16379780
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5593
  
@tillrohrmann could you review this PR? Thanks!


> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7836) specifying node label for flink job to run on yarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379722#comment-16379722
 ] 

ASF GitHub Bot commented on FLINK-7836:
---

GitHub user yanghua opened a pull request:

https://github.com/apache/flink/pull/5593

[FLINK-7836][Client] specifying node label for flink job to run on yarn

## What is the purpose of the change

*This pull request support flink on yarn to specify yarn node label for 
yarn flink application both flink session and single job mode*


## Brief change log

  - *define a new command option to specify a yarn node label*
  - *set the node label for the application through reflection (to support 
low version hadoop)*
  - *add command line option description*


## Verifying this change

This change without any test coverage.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanghua/flink FLINK-7836

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5593.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5593


commit 5f7342716223adea6ec1745d319a44a2bd3188da
Author: vinoyang 
Date:   2018-02-28T03:02:07Z

[FLINK-7836][Client] specifying node label for flink job to run on yarn




> specifying node label for flink job to run on yarn
> --
>
> Key: FLINK-7836
> URL: https://issues.apache.org/jira/browse/FLINK-7836
> Project: Flink
>  Issue Type: New Feature
>  Components: Client
>Affects Versions: 1.3.2
>Reporter: zhaibaba
>Assignee: vinoyang
>Priority: Major
>
> flink client cannot specify node label for flink job to run on yarn



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)