[jira] [Commented] (YARN-8769) [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job
[ https://issues.apache.org/jira/browse/YARN-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612989#comment-16612989 ] Wangda Tan commented on YARN-8769: -- Attached ver.1 patch, this patch depends on YARN-8757. > [Submarine] Allow user to specify customized quicklink(s) when submit > Submarine job > --- > > Key: YARN-8769 > URL: https://issues.apache.org/jira/browse/YARN-8769 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8769.001.patch > > > This will be helpful when user submit a job and some links need to be shown > on YARN UI2 (service page). For example, user can specify a quick link to > Zeppelin notebook UI when a Zeppelin notebook got launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8769) [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job
[ https://issues.apache.org/jira/browse/YARN-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612989#comment-16612989 ] Wangda Tan edited comment on YARN-8769 at 9/13/18 4:16 AM: --- Attached ver.1 patch, this patch depends on YARN-8757. [~sunil.gov...@gmail.com] could u help to review? was (Author: leftnoteasy): Attached ver.1 patch, this patch depends on YARN-8757. > [Submarine] Allow user to specify customized quicklink(s) when submit > Submarine job > --- > > Key: YARN-8769 > URL: https://issues.apache.org/jira/browse/YARN-8769 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8769.001.patch > > > This will be helpful when user submit a job and some links need to be shown > on YARN UI2 (service page). For example, user can specify a quick link to > Zeppelin notebook UI when a Zeppelin notebook got launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8769) [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job
[ https://issues.apache.org/jira/browse/YARN-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8769: - Attachment: YARN-8769.001.patch > [Submarine] Allow user to specify customized quicklink(s) when submit > Submarine job > --- > > Key: YARN-8769 > URL: https://issues.apache.org/jira/browse/YARN-8769 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8769.001.patch > > > This will be helpful when user submit a job and some links need to be shown > on YARN UI2 (service page). For example, user can specify a quick link to > Zeppelin notebook UI when a Zeppelin notebook got launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/YARN-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612961#comment-16612961 ] Sunil Govindan commented on YARN-8770: -- Thanks [~leftnoteasy]. I will help to support pytorch in submarine. > [Submarine] Support using Submarine to submit Pytorch job > - > > Key: YARN-8770 > URL: https://issues.apache.org/jira/browse/YARN-8770 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil Govindan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/YARN-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan reassigned YARN-8770: Assignee: Sunil Govindan > [Submarine] Support using Submarine to submit Pytorch job > - > > Key: YARN-8770 > URL: https://issues.apache.org/jira/browse/YARN-8770 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil Govindan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612957#comment-16612957 ] Sunil Govindan commented on YARN-8768: -- Thanks [~shaneku...@gmail.com] for reviewing and committing the same. Appreciate the same. (y) And many thanks [~rohithsharma] for helping in reviews. > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
Wangda Tan created YARN-8770: Summary: [Submarine] Support using Submarine to submit Pytorch job Key: YARN-8770 URL: https://issues.apache.org/jira/browse/YARN-8770 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8769) [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job
Wangda Tan created YARN-8769: Summary: [Submarine] Allow user to specify customized quicklink(s) when submit Submarine job Key: YARN-8769 URL: https://issues.apache.org/jira/browse/YARN-8769 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan This will be helpful when user submit a job and some links need to be shown on YARN UI2 (service page). For example, user can specify a quick link to Zeppelin notebook UI when a Zeppelin notebook got launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8648) Container cgroups are leaked when using docker
[
https://issues.apache.org/jira/browse/YARN-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612788#comment-16612788
]
Jason Lowe commented on YARN-8648:
--
Thanks for updating the patch!
It seems a bit awkward that remove_docker_container and exec_docker_container
receives an array of argument pointers and the number of pointers but then also
an offset into that array to start parsing. I think it would be simpler if the
functions interpreted their args parameters as-is, and it was up to the caller
to pass only relevant arguments. For example, main.c could pass (argv +
optind, argc - optind) for argv and argc, respectively. It's cheap, easy, and
simplifies the logic in those docker functions since they don't have to deal
with skipping unrelated arguments.
Otherwise patch looks good to me.
> Container cgroups are leaked when using docker
> --
>
> Key: YARN-8648
> URL: https://issues.apache.org/jira/browse/YARN-8648
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: Docker
> Attachments: YARN-8648.001.patch, YARN-8648.002.patch,
> YARN-8648.003.patch, YARN-8648.004.patch, YARN-8648.005.patch
>
>
> When you run with docker and enable cgroups for cpu, docker creates cgroups
> for all resources on the system, not just for cpu. For instance, if the
> {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn}},
> the nodemanager will create a cgroup for each container under
> {{/sys/fs/cgroup/cpu/hadoop-yarn}}. In the docker case, we pass this path
> via the {{--cgroup-parent}} command line argument. Docker then creates a
> cgroup for the docker container under that, for instance:
> {{/sys/fs/cgroup/cpu/hadoop-yarn/container_id/docker_container_id}}.
> When the container exits, docker cleans up the {{docker_container_id}}
> cgroup, and the nodemanager cleans up the {{container_id}} cgroup, All is
> good under {{/sys/fs/cgroup/hadoop-yarn}}.
> The problem is that docker also creates that same hierarchy under every
> resource under {{/sys/fs/cgroup}}. On the rhel7 system I am using, these
> are: blkio, cpuset, devices, freezer, hugetlb, memory, net_cls, net_prio,
> perf_event, and systemd.So for instance, docker creates
> {{/sys/fs/cgroup/cpuset/hadoop-yarn/container_id/docker_container_id}}, but
> it only cleans up the leaf cgroup {{docker_container_id}}. Nobody cleans up
> the {{container_id}} cgroups for these other resources. On one of our busy
> clusters, we found > 100,000 of these leaked cgroups.
> I found this in our 2.8-based version of hadoop, but I have been able to
> repro with current hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612772#comment-16612772 ] Hudson commented on YARN-8768: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14937 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14937/]) YARN-8768. Javadoc error in node attributes. Contributed by Sunil (skumpf: rev 8e9afbfbedcddcdd946e7dad0aac6e09265b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesRequest.java > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612753#comment-16612753 ] Naganarasimha G R commented on YARN-8768: - Thanks [~shaneku...@gmail.com] for merging it > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612746#comment-16612746 ] Shane Kumpf commented on YARN-8768: --- Thanks to [~sunilg] for the contribution and [~rohithsharma] for the review. I validated that this fixed the build for me as well. Committed to trunk. > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[
https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612732#comment-16612732
]
Hadoop QA commented on YARN-8768:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
53s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 74m
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8768 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939444/YARN-8768.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ac5e8af709ce 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Commented] (YARN-8619) Automate docker network configuration through YARN API
[
https://issues.apache.org/jira/browse/YARN-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612715#comment-16612715
]
Eric Yang commented on YARN-8619:
-
A step by step process to create a docker overlay network cross nodes look like
this:
h2. Step 1 Configure a cluster store
Consul is a key/value pair store for docker to store cluster network states. A
simple cluster store can be deploy by running:
{code}
docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server
-bootstrap
{code}
h2. Step 2 Configure docker daemon on all nodes (/etc/docker/daemon.json)
nsert cluster-advertise and cluster-store JSON to daemon.json
Note, br0 is the network interface name that should be listening for cluster
network change.
Where 172.21.195.5 is the IP address of where consul container is running.
{code}
{
"cluster-advertise" : "br0:2375",
"cluster-store" : "consul://172.21.195.5:8500"
}
{code}
h2. Step 3 restart docker daemons on all nodes
{code}
systemctl restart docker
{code}
h2. Step 4 Validate docker is running with new cluster-store
{code}
docker info
{code}
The output will have:
{code}
Cluster Store: consul://172.21.195.5:8500
Cluster Advertise: 172.21.195.6:2375
{code}
This means the docker daemon is running with external kv store.
h2. Step 5 Create a overlay network
On any of the docker node, run:
{code}
docker network create -d overlay --subnet=192.168.3.0/24 hadoop
{code}
New containers can be started using the network alias name hadoop. The
containers should be able to ping each other across hosts.
If a docker container is launched using hadoop network, there is no outbound
traffic by default, and additional network connection might need to be setup
using
{code}
docker network connect bridge [container-id]
{code}
This will provide outbound traffic to container network. Majority of the work
depends on docker daemon configuration settings being correct and external kv
store is deployed. If YARN is aware of the network configuration, it can
potentially run the last command to enable container for outbound access. I am
not quite sure if YARN should venture into configuring docker, and my
impression is that YARN shouldn't be configuring docker daemon. However, the
config might be more seamless if these procedures are automated by Ambari
installation and hand over a knob for YARN to connect overlay network with
physical network. Thoughts?
> Automate docker network configuration through YARN API
> --
>
> Key: YARN-8619
> URL: https://issues.apache.org/jira/browse/YARN-8619
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Reporter: Eric Yang
>Priority: Major
> Labels: Docker
>
> Docker supports bridge, host, overlay, macvlan networking. It might be
> useful to automate docker network setup through a set of YARN API to improve
> management of docker networks. Each type of network driver requires
> different type of parameters. For Hadoop use case, it seems more useful to
> focus on macvlan networking for ease of use and configuration. It would be
> great addition to support commands like:
> {code}
> yarn network create -d macvlan \
> --subnet=172.16.86.0/24 \
> --gateway=172.16.86.1 \
> -o parent=eth0 \
> my-macvlan-net
> {code}
> This changes docker configuration to hosts managed by YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8648) Container cgroups are leaked when using docker
[
https://issues.apache.org/jira/browse/YARN-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612632#comment-16612632
]
Hadoop QA commented on YARN-8648:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8648 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939448/YARN-8648.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux ed93838db598 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1f6c454 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21826/testReport/ |
| Max. process+thread count | 409 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21826/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (YARN-8658) [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612626#comment-16612626 ] Hudson commented on YARN-8658: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14936 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14936/]) YARN-8658. [AMRMProxy] Metrics for AMRMClientRelayer inside (gifuma: rev 02b9bfdf9e4bd0b3c05ca5fd75399dedcb656e09) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestableFederationInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedApplicationManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/uam/TestUnmanagedApplicationManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedAMPoolManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/metrics/TestAMRMClientRelayerMetrics.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/AMRMClientRelayer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/TestAMRMClientRelayer.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/AMRMClientRelayerMetrics.java > [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor > -- > > Key: YARN-8658 > URL: https://issues.apache.org/jira/browse/YARN-8658 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Young Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8658.01.patch, YARN-8658.02.patch, > YARN-8658.03.patch, YARN-8658.04.patch, YARN-8658.05.patch, > YARN-8658.06.patch, YARN-8658.07.patch, YARN-8658.08.patch, YARN-8658.09.patch > > > AMRMClientRelayer (YARN-7900) is introduced for stateful > FederationInterceptor (YARN-7899), to keep track of all pending requests sent > to every subcluster YarnRM. We need to add metrics for AMRMClientRelayer to > show the state of things in FederationInterceptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8768: - Target Version/s: 3.2.0 > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8658) [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612605#comment-16612605 ] Giovanni Matteo Fumarola commented on YARN-8658: Thanks [~youchen] for the patch and [~botong] for the review. Committed to trunk. The patch does not apply in Branch-2. Please rebase it. Resolving the Jira for now. > [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor > -- > > Key: YARN-8658 > URL: https://issues.apache.org/jira/browse/YARN-8658 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Young Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8658.01.patch, YARN-8658.02.patch, > YARN-8658.03.patch, YARN-8658.04.patch, YARN-8658.05.patch, > YARN-8658.06.patch, YARN-8658.07.patch, YARN-8658.08.patch, YARN-8658.09.patch > > > AMRMClientRelayer (YARN-7900) is introduced for stateful > FederationInterceptor (YARN-7899), to keep track of all pending requests sent > to every subcluster YarnRM. We need to add metrics for AMRMClientRelayer to > show the state of things in FederationInterceptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8658) [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-8658: --- Fix Version/s: 3.2.0 > [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor > -- > > Key: YARN-8658 > URL: https://issues.apache.org/jira/browse/YARN-8658 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Young Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8658.01.patch, YARN-8658.02.patch, > YARN-8658.03.patch, YARN-8658.04.patch, YARN-8658.05.patch, > YARN-8658.06.patch, YARN-8658.07.patch, YARN-8658.08.patch, YARN-8658.09.patch > > > AMRMClientRelayer (YARN-7900) is introduced for stateful > FederationInterceptor (YARN-7899), to keep track of all pending requests sent > to every subcluster YarnRM. We need to add metrics for AMRMClientRelayer to > show the state of things in FederationInterceptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8658) [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612560#comment-16612560 ] Botong Huang commented on YARN-8658: [^YARN-8658.09.patch] lgtm. Thanks [~youchen] for the patch! > [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor > -- > > Key: YARN-8658 > URL: https://issues.apache.org/jira/browse/YARN-8658 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Young Chen >Priority: Major > Attachments: YARN-8658.01.patch, YARN-8658.02.patch, > YARN-8658.03.patch, YARN-8658.04.patch, YARN-8658.05.patch, > YARN-8658.06.patch, YARN-8658.07.patch, YARN-8658.08.patch, YARN-8658.09.patch > > > AMRMClientRelayer (YARN-7900) is introduced for stateful > FederationInterceptor (YARN-7899), to keep track of all pending requests sent > to every subcluster YarnRM. We need to add metrics for AMRMClientRelayer to > show the state of things in FederationInterceptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8648) Container cgroups are leaked when using docker
[
https://issues.apache.org/jira/browse/YARN-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612556#comment-16612556
]
Jim Brennan commented on YARN-8648:
---
Thanks [~jlowe] for the review! I have addressed all of this issues you raised
in patch 005.
I did change it so we no longer ignore EBUSY errors. This might cause remove
operations for containers in a docker-in-docker situation to be reported as
failed even though the container was removed (but some of the cgroups were
not). I think the only impact would be log messages to that effect.
[~billie.rinaldi], are you ok with this?
> Container cgroups are leaked when using docker
> --
>
> Key: YARN-8648
> URL: https://issues.apache.org/jira/browse/YARN-8648
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: Docker
> Attachments: YARN-8648.001.patch, YARN-8648.002.patch,
> YARN-8648.003.patch, YARN-8648.004.patch, YARN-8648.005.patch
>
>
> When you run with docker and enable cgroups for cpu, docker creates cgroups
> for all resources on the system, not just for cpu. For instance, if the
> {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn}},
> the nodemanager will create a cgroup for each container under
> {{/sys/fs/cgroup/cpu/hadoop-yarn}}. In the docker case, we pass this path
> via the {{--cgroup-parent}} command line argument. Docker then creates a
> cgroup for the docker container under that, for instance:
> {{/sys/fs/cgroup/cpu/hadoop-yarn/container_id/docker_container_id}}.
> When the container exits, docker cleans up the {{docker_container_id}}
> cgroup, and the nodemanager cleans up the {{container_id}} cgroup, All is
> good under {{/sys/fs/cgroup/hadoop-yarn}}.
> The problem is that docker also creates that same hierarchy under every
> resource under {{/sys/fs/cgroup}}. On the rhel7 system I am using, these
> are: blkio, cpuset, devices, freezer, hugetlb, memory, net_cls, net_prio,
> perf_event, and systemd.So for instance, docker creates
> {{/sys/fs/cgroup/cpuset/hadoop-yarn/container_id/docker_container_id}}, but
> it only cleans up the leaf cgroup {{docker_container_id}}. Nobody cleans up
> the {{container_id}} cgroups for these other resources. On one of our busy
> clusters, we found > 100,000 of these leaked cgroups.
> I found this in our 2.8-based version of hadoop, but I have been able to
> repro with current hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8648) Container cgroups are leaked when using docker
[
https://issues.apache.org/jira/browse/YARN-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated YARN-8648:
--
Attachment: YARN-8648.005.patch
> Container cgroups are leaked when using docker
> --
>
> Key: YARN-8648
> URL: https://issues.apache.org/jira/browse/YARN-8648
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: Docker
> Attachments: YARN-8648.001.patch, YARN-8648.002.patch,
> YARN-8648.003.patch, YARN-8648.004.patch, YARN-8648.005.patch
>
>
> When you run with docker and enable cgroups for cpu, docker creates cgroups
> for all resources on the system, not just for cpu. For instance, if the
> {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn}},
> the nodemanager will create a cgroup for each container under
> {{/sys/fs/cgroup/cpu/hadoop-yarn}}. In the docker case, we pass this path
> via the {{--cgroup-parent}} command line argument. Docker then creates a
> cgroup for the docker container under that, for instance:
> {{/sys/fs/cgroup/cpu/hadoop-yarn/container_id/docker_container_id}}.
> When the container exits, docker cleans up the {{docker_container_id}}
> cgroup, and the nodemanager cleans up the {{container_id}} cgroup, All is
> good under {{/sys/fs/cgroup/hadoop-yarn}}.
> The problem is that docker also creates that same hierarchy under every
> resource under {{/sys/fs/cgroup}}. On the rhel7 system I am using, these
> are: blkio, cpuset, devices, freezer, hugetlb, memory, net_cls, net_prio,
> perf_event, and systemd.So for instance, docker creates
> {{/sys/fs/cgroup/cpuset/hadoop-yarn/container_id/docker_container_id}}, but
> it only cleans up the leaf cgroup {{docker_container_id}}. Nobody cleans up
> the {{container_id}} cgroups for these other resources. On one of our busy
> clusters, we found > 100,000 of these leaked cgroups.
> I found this in our 2.8-based version of hadoop, but I have been able to
> repro with current hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612536#comment-16612536 ] Rohith Sharma K S commented on YARN-8768: - +1 pending jenkins.. I build locally and java doc errors are passing.. > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8768) Javadoc error in node attributes
[
https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612527#comment-16612527
]
Sunil Govindan commented on YARN-8768:
--
{code:java}
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 09:38 min
[INFO] Finished at: 2018-09-12T23:19:09+05:30
[INFO] Final Memory: 184M/1635M
[INFO]
HW15892:hadoop sunilgovindan{code}
> Javadoc error in node attributes
>
>
> Key: YARN-8768
> URL: https://issues.apache.org/jira/browse/YARN-8768
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-8768.001.patch
>
>
> fix java doc error from node attributes in yarn-api module.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8768) Javadoc error in node attributes
[ https://issues.apache.org/jira/browse/YARN-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8768: - Attachment: YARN-8768.001.patch > Javadoc error in node attributes > > > Key: YARN-8768 > URL: https://issues.apache.org/jira/browse/YARN-8768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-8768.001.patch > > > fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8768) Javadoc error in node attributes
Sunil Govindan created YARN-8768: Summary: Javadoc error in node attributes Key: YARN-8768 URL: https://issues.apache.org/jira/browse/YARN-8768 Project: Hadoop YARN Issue Type: Bug Reporter: Sunil Govindan Assignee: Sunil Govindan fix java doc error from node attributes in yarn-api module. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612486#comment-16612486
]
Hadoop QA commented on YARN-6456:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
27s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem
[jira] [Assigned] (YARN-8767) TestStreamingStatus fails
[
https://issues.apache.org/jira/browse/YARN-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor reassigned YARN-8767:
--
Assignee: Andras Bokor
> TestStreamingStatus fails
> -
>
> Key: YARN-8767
> URL: https://issues.apache.org/jira/browse/YARN-8767
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> The test tries to connect to RM through 0.0.0.0:8032, but it cannot.
> On the console I see the following error message:
> {code}Your endpoint configuration is wrong; For more details see:
> http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking
> ApplicationClientProtocolPBClientImpl.getNewApplication over null after 1
> failover attempts. Trying to failover after sleeping for 44892ms.{code}
> Do I miss some configuration?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612427#comment-16612427
]
Eric Yang commented on YARN-6456:
-
[~ccondit-target] {quote}This configuration is here for parity with
allowed-runtimes. I don't believe this is redundant, as container-executor
configuration is meant to lock down yarn-to-root escalation paths.{quote}
There are two configurations: docker.trusted.registries and
docker.privileged.registries. The first one governs if the image is allowed to
run, and the second one check if the image can be ran as privileged. At this
time, only docker.trusted.registries is implemented, and images in trusted
privileges can run as privileged container as well as normal user's containers.
Non-trusted images are running in sandbox mode (no external mounts). The
purpose of the settings are similar, but not identical. We decided to build
the check in container-executor level instead of node manager level to prevent
unauthorized yarn user from spawning container with unauthorized binaries. The
newly introduced setting will not be enforced at root level. Therefore, it
does not have the same strength of protection as existing mechanism. There is
a high chance that new container runtime requires root privileges to start
container. If this is the case, it would be better to build the allowed image
and default image check in container-executor to be in parity with decisions
made last year.
[~jlowe] This extra check will generate support headache. From Java code to
container-executor is only a few millisecond difference, therefore fail fast in
this case might not make much difference. I also prefer to have more
configuration in yarn-site.xml, but container-executor doesn't parse xml file
is a limitation that can be solved in another story.
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
> Attachments: YARN-6456-ForceDockerRuntimeIfSupported.patch,
> YARN-6456.001.patch, YARN-6456.002.patch, YARN-6456.003.patch
>
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[
https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612409#comment-16612409
]
Hadoop QA commented on YARN-8234:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 260 unchanged - 0 fixed = 266 total (was 260) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
12s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8234 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939408/YARN-8234.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 52d7669d0675 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64
[jira] [Created] (YARN-8767) TestStreamingStatus fails
Andras Bokor created YARN-8767:
--
Summary: TestStreamingStatus fails
Key: YARN-8767
URL: https://issues.apache.org/jira/browse/YARN-8767
Project: Hadoop YARN
Issue Type: Bug
Reporter: Andras Bokor
The test tries to connect to RM through 0.0.0.0:8032, but it cannot.
On the console I see the following error message:
{code}Your endpoint configuration is wrong; For more details see:
http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking
ApplicationClientProtocolPBClientImpl.getNewApplication over null after 1
failover attempts. Trying to failover after sleeping for 44892ms.{code}
Do I miss some configuration?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8759) Copy of "resource-types.xml" is not deleted if test fails, causes other test failures
[
https://issues.apache.org/jira/browse/YARN-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612248#comment-16612248
]
Hadoop QA commented on YARN-8759:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
7s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8759 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939400/YARN-8759.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6836254f6616 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5e64e62 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21821/testReport/ |
| Max. process+thread count | 910 (vs. ulimit of 1) |
[jira] [Commented] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612220#comment-16612220
]
Jason Lowe commented on YARN-6456:
--
bq. My initial thought was that we already validate this at runtime, but I can
update the code to do the regex check here as well.
Yeah, we definitely are checking this at runtime and are covered there. I was
just thinking this was a way of failing fast for a bad config, similar to the
check for the default image being in the list of allowed images during init to
fail fast. I'm OK if people think this extra check is overkill and let a
container launch fail later when the allowed image is requested but disallowed
by the regex.
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
> Attachments: YARN-6456-ForceDockerRuntimeIfSupported.patch,
> YARN-6456.001.patch, YARN-6456.002.patch
>
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8759) Copy of "resource-types.xml" is not deleted if test fails, causes other test failures
[ https://issues.apache.org/jira/browse/YARN-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612215#comment-16612215 ] Szilard Nemeth commented on YARN-8759: -- Hi [~bsteinbach]! Thanks for the patch. LGTM, +1 (non-binding) > Copy of "resource-types.xml" is not deleted if test fails, causes other test > failures > - > > Key: YARN-8759 > URL: https://issues.apache.org/jira/browse/YARN-8759 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Major > Attachments: YARN-8759.001.patch, YARN-8759.002.patch, > YARN-8759.003.patch > > > resource-types.xml is copied in several tests to the test machine, but it is > deleted only at the end of the test. In case the test fails the file will not > be deleted and other tests will fail, because of the wrong configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612194#comment-16612194
]
Craig Condit edited comment on YARN-6456 at 9/12/18 2:14 PM:
-
[~eyang] wrote:
{quote}The new configuration looks like redundant configuration, but this is
runtime specific approach to grant image.
{quote}
This configuration is here for parity with allowed-runtimes. I don't believe
this is redundant, as container-executor configuration is meant to lock down
yarn-to-root escalation paths.
was (Author: ccondit-target):
[~eyang] wrote:
{quote}The new configuration looks like redundant configuration, but this is
runtime specific approach to grant image.
{quote}
This configuration is here for parity with allowed-runtimes. I don't believe
this is redundant, as container-executor configuration is meant to lock down
yarn->root escalation paths.
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
> Attachments: YARN-6456-ForceDockerRuntimeIfSupported.patch,
> YARN-6456.001.patch, YARN-6456.002.patch
>
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612194#comment-16612194
]
Craig Condit commented on YARN-6456:
[~eyang] wrote:
{quote}The new configuration looks like redundant configuration, but this is
runtime specific approach to grant image.
{quote}
This configuration is here for parity with allowed-runtimes. I don't believe
this is redundant, as container-executor configuration is meant to lock down
yarn->root escalation paths.
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
> Attachments: YARN-6456-ForceDockerRuntimeIfSupported.patch,
> YARN-6456.001.patch, YARN-6456.002.patch
>
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612193#comment-16612193
]
Szilard Nemeth commented on YARN-3525:
--
Hi [~zsiegl]!
Thanks for the patch, I like this deprecation approach.
Please use String constants for {{yarn.scheduler.fair.increment-allocation-mb
yarn.scheduler.fair.increment-allocation-vcores}} and
{{yarn.scheduler.fair.increment-allocation-mb
yarn.scheduler.fair.increment-allocation-vcores}} in the test where possible.
Otherwise the patch looks good to me.
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6456) Allow administrators to set a single ContainerRuntime for all containers
[
https://issues.apache.org/jira/browse/YARN-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612192#comment-16612192
]
Craig Condit commented on YARN-6456:
[~jlowe] wrote:
{quote}Should the docker runtime validate the allowed images against the docker
name pattern when initializing? That would let it fail fast if one or more of
the image names isn't going to be allowed by the docker image name pattern.
{quote}
My initial thought was that we already validate this at runtime, but I can
update the code to do the regex check here as well.
> Allow administrators to set a single ContainerRuntime for all containers
>
>
> Key: YARN-6456
> URL: https://issues.apache.org/jira/browse/YARN-6456
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Craig Condit
>Priority: Major
> Labels: Docker
> Attachments: YARN-6456-ForceDockerRuntimeIfSupported.patch,
> YARN-6456.001.patch, YARN-6456.002.patch
>
>
>
> With LCE, there are multiple ContainerRuntimes available for handling
> different types of containers; default, docker, java sandbox. Admins should
> have the ability to override the user decision and set a single global
> ContainerRuntime to be used for all containers.
> Original Description:
> {quote}One reason to use Docker containers is to be able to isolate different
> workloads, even, if they run as the same user.
> I have noticed some issues in the current design:
> 1. DockerLinuxContainerRuntime mounts containerLocalDirs
> {{nm-local-dir/usercache/user/appcache/application_1491598755372_0011/}} and
> userLocalDirs {{nm-local-dir/usercache/user/}}, so that a container can see
> and modify the files of another container. I think the application file cache
> directory should be enough for the container to run in most of the cases.
> 2. The whole cgroups directory is mounted. Would the container directory be
> enough?
> 3. There is no way to enforce exclusive use of Docker for all containers.
> There should be an option that it is not the user but the admin that requires
> to use Docker.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[
https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612150#comment-16612150
]
Hadoop QA commented on YARN-8234:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 9m
54s{color} | {color:red} Docker failed to build yetus/hadoop:c2d96dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8234 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939409/YARN-8234-branch-2.8.3.004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21823/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve RM system metrics publisher's performance by pushing events to
> timeline server in batch
> ---
>
> Key: YARN-8234
> URL: https://issues.apache.org/jira/browse/YARN-8234
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, timelineserver
>Affects Versions: 2.8.3
>Reporter: Hu Ziqian
>Assignee: Hu Ziqian
>Priority: Critical
> Attachments: YARN-8234-branch-2.8.3.001.patch,
> YARN-8234-branch-2.8.3.002.patch, YARN-8234-branch-2.8.3.003.patch,
> YARN-8234-branch-2.8.3.004.patch, YARN-8234.001.patch, YARN-8234.002.patch,
> YARN-8234.003.patch, YARN-8234.004.patch
>
>
> When system metrics publisher is enabled, RM will push events to timeline
> server via restful api. If the cluster load is heavy, many events are sent to
> timeline server and the timeline server's event handler thread locked.
> YARN-7266 talked about the detail of this problem. Because of the lock,
> timeline server can't receive event as fast as it generated in RM and lots of
> timeline event stays in RM's memory. Finally, those events will consume all
> RM's memory and RM will start a full gc (which cause an JVM stop-world and
> cause a timeout from rm to zookeeper) or even get an OOM.
> The main problem here is that timeline can't receive timeline server's event
> as fast as it generated. Now, RM system metrics publisher put only one event
> in a request, and most time costs on handling http header or some thing about
> the net connection on timeline side. Only few time is spent on dealing with
> the timeline event which is truly valuable.
> In this issue, we add a buffer in system metrics publisher and let publisher
> send events to timeline server in batch via one request. When sets the batch
> size to 1000, in out experiment the speed of the timeline server receives
> events has 100x improvement. We have implement this function int our product
> environment which accepts 2 app's in one hour and it works fine.
> We add following configuration:
> * yarn.resourcemanager.system-metrics-publisher.batch-size: the size of
> system metrics publisher sending events in one request. Default value is 1000
> * yarn.resourcemanager.system-metrics-publisher.buffer-size: the size of the
> event buffer in system metrics publisher.
> * yarn.resourcemanager.system-metrics-publisher.interval-seconds: When
> enable batch publishing, we must avoid that the publisher waits for a batch
> to be filled up and hold events in buffer for long time. So we add another
> thread which send event's in the buffer periodically. This config sets the
> interval of the cyclical sending thread. The default value is 60s.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[ https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Ziqian updated YARN-8234: Attachment: YARN-8234-branch-2.8.3.004.patch > Improve RM system metrics publisher's performance by pushing events to > timeline server in batch > --- > > Key: YARN-8234 > URL: https://issues.apache.org/jira/browse/YARN-8234 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, timelineserver >Affects Versions: 2.8.3 >Reporter: Hu Ziqian >Assignee: Hu Ziqian >Priority: Critical > Attachments: YARN-8234-branch-2.8.3.001.patch, > YARN-8234-branch-2.8.3.002.patch, YARN-8234-branch-2.8.3.003.patch, > YARN-8234-branch-2.8.3.004.patch, YARN-8234.001.patch, YARN-8234.002.patch, > YARN-8234.003.patch, YARN-8234.004.patch > > > When system metrics publisher is enabled, RM will push events to timeline > server via restful api. If the cluster load is heavy, many events are sent to > timeline server and the timeline server's event handler thread locked. > YARN-7266 talked about the detail of this problem. Because of the lock, > timeline server can't receive event as fast as it generated in RM and lots of > timeline event stays in RM's memory. Finally, those events will consume all > RM's memory and RM will start a full gc (which cause an JVM stop-world and > cause a timeout from rm to zookeeper) or even get an OOM. > The main problem here is that timeline can't receive timeline server's event > as fast as it generated. Now, RM system metrics publisher put only one event > in a request, and most time costs on handling http header or some thing about > the net connection on timeline side. Only few time is spent on dealing with > the timeline event which is truly valuable. > In this issue, we add a buffer in system metrics publisher and let publisher > send events to timeline server in batch via one request. When sets the batch > size to 1000, in out experiment the speed of the timeline server receives > events has 100x improvement. We have implement this function int our product > environment which accepts 2 app's in one hour and it works fine. > We add following configuration: > * yarn.resourcemanager.system-metrics-publisher.batch-size: the size of > system metrics publisher sending events in one request. Default value is 1000 > * yarn.resourcemanager.system-metrics-publisher.buffer-size: the size of the > event buffer in system metrics publisher. > * yarn.resourcemanager.system-metrics-publisher.interval-seconds: When > enable batch publishing, we must avoid that the publisher waits for a batch > to be filled up and hold events in buffer for long time. So we add another > thread which send event's in the buffer periodically. This config sets the > interval of the cyclical sending thread. The default value is 60s. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[ https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Ziqian updated YARN-8234: Attachment: YARN-8234.004.patch > Improve RM system metrics publisher's performance by pushing events to > timeline server in batch > --- > > Key: YARN-8234 > URL: https://issues.apache.org/jira/browse/YARN-8234 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, timelineserver >Affects Versions: 2.8.3 >Reporter: Hu Ziqian >Assignee: Hu Ziqian >Priority: Critical > Attachments: YARN-8234-branch-2.8.3.001.patch, > YARN-8234-branch-2.8.3.002.patch, YARN-8234-branch-2.8.3.003.patch, > YARN-8234.001.patch, YARN-8234.002.patch, YARN-8234.003.patch, > YARN-8234.004.patch > > > When system metrics publisher is enabled, RM will push events to timeline > server via restful api. If the cluster load is heavy, many events are sent to > timeline server and the timeline server's event handler thread locked. > YARN-7266 talked about the detail of this problem. Because of the lock, > timeline server can't receive event as fast as it generated in RM and lots of > timeline event stays in RM's memory. Finally, those events will consume all > RM's memory and RM will start a full gc (which cause an JVM stop-world and > cause a timeout from rm to zookeeper) or even get an OOM. > The main problem here is that timeline can't receive timeline server's event > as fast as it generated. Now, RM system metrics publisher put only one event > in a request, and most time costs on handling http header or some thing about > the net connection on timeline side. Only few time is spent on dealing with > the timeline event which is truly valuable. > In this issue, we add a buffer in system metrics publisher and let publisher > send events to timeline server in batch via one request. When sets the batch > size to 1000, in out experiment the speed of the timeline server receives > events has 100x improvement. We have implement this function int our product > environment which accepts 2 app's in one hour and it works fine. > We add following configuration: > * yarn.resourcemanager.system-metrics-publisher.batch-size: the size of > system metrics publisher sending events in one request. Default value is 1000 > * yarn.resourcemanager.system-metrics-publisher.buffer-size: the size of the > event buffer in system metrics publisher. > * yarn.resourcemanager.system-metrics-publisher.interval-seconds: When > enable batch publishing, we must avoid that the publisher waits for a batch > to be filled up and hold events in buffer for long time. So we add another > thread which send event's in the buffer periodically. This config sets the > interval of the cyclical sending thread. The default value is 60s. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-6949) Invalid event: LOCALIZATION_FAILED at LOCALIZED
[
https://issues.apache.org/jira/browse/YARN-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-6949:
Comment: was deleted
(was: I check the log and also find some NullPointerException:
{code:java}
ava.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.getPathForLocalization(LocalResourcesTrackerImpl.java:505)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.getPathForLocalization(ResourceLocalizationService.java:1131)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.processHeartbeat(ResourceLocalizationService.java:1093)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:720)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:355)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:48)
{code}
)
> Invalid event: LOCALIZATION_FAILED at LOCALIZED
> ---
>
> Key: YARN-6949
> URL: https://issues.apache.org/jira/browse/YARN-6949
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.8.0
>Reporter: lujie
>Priority: Major
>
> When job is running, I stop a nodemanager in one machine due to some reason,
> Then I check the logs to see the running state,I find many
> InvalidStateTransitionException:
> {code:java}
> rg.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> LOCALIZATION_FAILED at LOCALIZED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource.handle(LocalizedResource.java:198)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:58)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.processHeartbeat(ResourceLocalizationService.java:1058)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:720)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:355)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:48)
> at
> org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:63)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8759) Copy of "resource-types.xml" is not deleted if test fails, causes other test failures
[ https://issues.apache.org/jira/browse/YARN-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612031#comment-16612031 ] Antal Bálint Steinbach commented on YARN-8759: -- Hi [~sunilg] , thanks for the feedback. I just did not want to modify the test cases but I agree with you, your suggestion makes the code more straightforward. I uploaded a new patch with the changes. > Copy of "resource-types.xml" is not deleted if test fails, causes other test > failures > - > > Key: YARN-8759 > URL: https://issues.apache.org/jira/browse/YARN-8759 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Major > Attachments: YARN-8759.001.patch, YARN-8759.002.patch, > YARN-8759.003.patch > > > resource-types.xml is copied in several tests to the test machine, but it is > deleted only at the end of the test. In case the test fails the file will not > be deleted and other tests will fail, because of the wrong configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8759) Copy of "resource-types.xml" is not deleted if test fails, causes other test failures
[ https://issues.apache.org/jira/browse/YARN-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Bálint Steinbach updated YARN-8759: - Attachment: YARN-8759.003.patch > Copy of "resource-types.xml" is not deleted if test fails, causes other test > failures > - > > Key: YARN-8759 > URL: https://issues.apache.org/jira/browse/YARN-8759 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Major > Attachments: YARN-8759.001.patch, YARN-8759.002.patch, > YARN-8759.003.patch > > > resource-types.xml is copied in several tests to the test machine, but it is > deleted only at the end of the test. In case the test fails the file will not > be deleted and other tests will fail, because of the wrong configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612010#comment-16612010 ] Naganarasimha G R commented on YARN-3409: - All open items has been moved to YARN-8766, and other planned jiras will also be tracked as part of YARN-8766. > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R resolved YARN-3409. - Resolution: Fixed Fix Version/s: 3.2.0 Release Note: Node Attribute Feature has been added as part of this jira. > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8715) Make allocation tags in the placement spec optional for node-attributes
[
https://issues.apache.org/jira/browse/YARN-8715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-8715:
Parent Issue: YARN-8766 (was: YARN-3409)
> Make allocation tags in the placement spec optional for node-attributes
> ---
>
> Key: YARN-8715
> URL: https://issues.apache.org/jira/browse/YARN-8715
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> YARN-7863 adds support to specify constraints targeting to node-attributes,
> including the support in distributed shell, but it still needs to specify
> {{allocationTags=numOfContainers}} in the spec. We should make this optional
> as it is not required for node-attribute expressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8102) Retrospect on having enable and disable flag for Node Attribute
[ https://issues.apache.org/jira/browse/YARN-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8102: Parent Issue: YARN-8766 (was: YARN-3409) > Retrospect on having enable and disable flag for Node Attribute > --- > > Key: YARN-8102 > URL: https://issues.apache.org/jira/browse/YARN-8102 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > > Currently node attribute feature is by default enabled. We have to revisit on > the same. > Enabling by default means will try to create store for all cluster > installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8101) Add UT to verify node-attributes in RM nodes rest API
[ https://issues.apache.org/jira/browse/YARN-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8101: Parent Issue: YARN-8766 (was: YARN-3409) > Add UT to verify node-attributes in RM nodes rest API > - > > Key: YARN-8101 > URL: https://issues.apache.org/jira/browse/YARN-8101 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, restapi >Affects Versions: YARN-3409 >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > > YARN-8092 exposed node-attributes in RM rest api, this task is created to add > UT for that functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response
[ https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7877: Parent Issue: YARN-8766 (was: YARN-3409) > Node attributes error handling based on RM validation response > -- > > Key: YARN-7877 > URL: https://issues.apache.org/jira/browse/YARN-7877 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > When NM reports its distributed attributes to RM, RM needs to do proper > validation of the received attributes, if attributes were not valid or failed > to update, RM needs to notify NM about such failures. Such validation needs > to be also done in NM registration as well. > This JIRA tracks how NM re-act against RM response, probably we can display > some info through UI/metrics/log etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6857) Support REST for Node Attributes configurations
[ https://issues.apache.org/jira/browse/YARN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6857: Parent Issue: YARN-8766 (was: YARN-3409) > Support REST for Node Attributes configurations > --- > > Key: YARN-6857 > URL: https://issues.apache.org/jira/browse/YARN-6857 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-6857-YARN-3409.001.patch > > > This jira focusses on supporting mapping of Nodes to Attributes through REST -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8766: Description: This Jira will track all the improvements on the Node Attribute functionality developed in YARN-3409 > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > > This Jira will track all the improvements on the Node Attribute functionality > developed in YARN-3409 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8766) Support for Node Attribute Functionality, Phase 2
Naganarasimha G R created YARN-8766: --- Summary: Support for Node Attribute Functionality, Phase 2 Key: YARN-8766 URL: https://issues.apache.org/jira/browse/YARN-8766 Project: Hadoop YARN Issue Type: Bug Reporter: Naganarasimha G R -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R reassigned YARN-8766: --- Assignee: Naganarasimha G R > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8766) Support for Node Attribute Functionality, Phase 2
[ https://issues.apache.org/jira/browse/YARN-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8766: Issue Type: Improvement (was: Bug) > Support for Node Attribute Functionality, Phase 2 > - > > Key: YARN-8766 > URL: https://issues.apache.org/jira/browse/YARN-8766 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7840) Update PB for prefix support of node attributes
[ https://issues.apache.org/jira/browse/YARN-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611968#comment-16611968 ] Hudson commented on YARN-7840: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7840. Update PB for prefix support of node attributes. Contributed (sunilg: rev 4458b2772f481259453ab5472a476bdebb8c835b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java > Update PB for prefix support of node attributes > --- > > Key: YARN-7840 > URL: https://issues.apache.org/jira/browse/YARN-7840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Blocker > Fix For: 3.2.0 > > Attachments: YARN-7840-YARN-3409.001.patch, > YARN-7840-YARN-3409.002.patch, YARN-7840-YARN-3409.003.patch, > YARN-7840-YARN-3409.004.patch > > > We need to support prefix (namespace) for node attributes, this will add the > flexibility to provide ability to do proper ACL, avoid naming conflicts etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8100) Support API interface to query cluster attributes and attribute to nodes
[ https://issues.apache.org/jira/browse/YARN-8100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611980#comment-16611980 ] Hudson commented on YARN-8100: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8100. Support API interface to query cluster attributes and (sunilg: rev 0a01b1350d69dacaf101d52e54f67c91e28ab1ee) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodeAttributesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/DefaultClientRequestInterceptor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodeAttributesResponsePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodeAttributesRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/PassThroughClientRequestInterceptor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesRequest.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodeAttributesRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java > Support API interface to query cluster attributes and attribute to nodes >
[jira] [Updated] (YARN-7877) Node attributes error handling based on RM validation response
[ https://issues.apache.org/jira/browse/YARN-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7877: Fix Version/s: (was: YARN-3409) > Node attributes error handling based on RM validation response > -- > > Key: YARN-7877 > URL: https://issues.apache.org/jira/browse/YARN-7877 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > When NM reports its distributed attributes to RM, RM needs to do proper > validation of the received attributes, if attributes were not valid or failed > to update, RM needs to notify NM about such failures. Such validation needs > to be also done in NM registration as well. > This JIRA tracks how NM re-act against RM response, probably we can display > some info through UI/metrics/log etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8351) Node attribute manager logs are flooding RM logs
[
https://issues.apache.org/jira/browse/YARN-8351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611983#comment-16611983
]
Hudson commented on YARN-8351:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14931/])
YARN-8351. Node attribute manager logs are flooding RM logs. Contributed
(sunilg: rev 76183428b76524818fc250ae97bbf973dc910e1c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
> Node attribute manager logs are flooding RM logs
>
>
> Key: YARN-8351
> URL: https://issues.apache.org/jira/browse/YARN-8351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8351-YARN-3409.001.patch, YARN-8351.001.patch
>
>
> When distributed node attributes enabled, RM updates these attributes on each
> NM HB interval, and each time it creates a log like
> {noformat}
> REPLACE attributes on nodes: NM="xxx", attributes=""
> {noformat}
> this should be in DEBUG level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8104) Add API to fetch node to attribute mapping
[ https://issues.apache.org/jira/browse/YARN-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611981#comment-16611981 ] Hudson commented on YARN-8104: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8104. Add API to fetch node to attribute mapping. Contributed by (sunilg: rev 5dc7d6e0f3f46277f0615d61755970a7aa28b8d7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToAttributesResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/DefaultClientRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToAttributesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/PassThroughClientRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java > Add API to fetch node to attribute mapping > -- > > Key: YARN-8104 > URL: https://issues.apache.org/jira/browse/YARN-8104 > Project: Hadoop YARN >
[jira] [Commented] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611988#comment-16611988 ] Hudson commented on YARN-7865: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7865. Node attributes documentation. Contributed by Naganarasimha G (sunilg: rev 1e7d6e55a59e2d5265d0c99e7bb1856c6849d662) * (edit) hadoop-project/src/site/site.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeAttributes.md > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Labels: node-attributes > Fix For: 3.2.0 > > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611964#comment-16611964 ] Hudson commented on YARN-3409: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-6855. [YARN-3409] CLI Proto Modifications to support Node (sunilg: rev 1f42ce907a4987ea095ae0c237b89274c689ed99) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeToAttributes.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/rmadmin/PassThroughRMAdminRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingResponsePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodesToAttributesMappingRequest.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/AttributeMappingOperationType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/DefaultRMAdminRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttributeType.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodesToAttributesMappingResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java YARN-6856. [YARN-3409] Support CLI for Node Attributes Mapping. (sunilg: rev 2475fb0a1e2158aac11829c6e33eae3e02bf33ea) * (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeAttributesCLI.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestNodeAttributesCLI.java > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R >Priority: Major > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, > Node-Attributes-Requirements-Design-doc_v2.pdf, YARN-3409.WIP.001.patch > > > Specify only one label
[jira] [Commented] (YARN-7856) Validate Node Attributes from NM
[ https://issues.apache.org/jira/browse/YARN-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611971#comment-16611971 ] Hudson commented on YARN-7856: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7856. Validate Node Attributes from NM. Contributed by Weiwei Yang. (sunilg: rev ffcabd24c33fc8c663cc75b57d7562db4c199713) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ScriptBasedNodeAttributesProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestScriptBasedNodeAttributesProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelUtil.java > Validate Node Attributes from NM > > > Key: YARN-7856 > URL: https://issues.apache.org/jira/browse/YARN-7856 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, RM >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7856-YARN-3409.001.patch, > YARN-7856-YARN-3409.002.patch > > > NM needs to do proper validation about the attributes before sending them to > RM, this includes > # a valid prefix is presented > # no duplicate entries > # do not allow two attributes with same prefix/name but different types > This could be an utility class that can be used on both RM/NM sides. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611985#comment-16611985 ] Hudson commented on YARN-8574: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8574. Allow dot in attribute values. Contributed by Bibin A (sunilg: rev 8c947398ab2de642cb49a59e5e9508d196d4bc30) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelUtil.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestNodeLabelUtil.java > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Fix For: 3.2.0 > > Attachments: YARN-8574-YARN-3409.001.patch, > YARN-8574-YARN-3409.002.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8094) Support configuration based Node Attribute provider
[
https://issues.apache.org/jira/browse/YARN-8094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611975#comment-16611975
]
Hudson commented on YARN-8094:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14931/])
YARN-8094. Support configuration based Node Attribute provider. (sunilg: rev
440ff7f563df5e7db72dce020c3c3dc379f88c91)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestConfigurationNodeAttributesProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ScriptBasedNodeAttributesProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ConfigurationNodeAttributesProvider.java
> Support configuration based Node Attribute provider
> ---
>
> Key: YARN-8094
> URL: https://issues.apache.org/jira/browse/YARN-8094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8094-YARN-3409.001.patch,
> YARN-8094-YARN-3409.002.patch, YARN-8094-YARN-3409.003.patch,
> YARN-8094-YARN-3409.004.patch
>
>
> Fix TODO item in \{{ConfigurationNodeAttributesProvider}}, completely support
> configuration based node attribute provider.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8740) Clear node attribute path after each test run
[
https://issues.apache.org/jira/browse/YARN-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611990#comment-16611990
]
Hudson commented on YARN-8740:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14931/])
YARN-8740. Clear node attribute path after each test run. Contributed by
(sunilg: rev 5e64e62deeb467ac2969bc357a24810db855cffa)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestNodeAttributesManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestFileSystemNodeAttributeStore.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeAttributeTestUtils.java
> Clear node attribute path after each test run
> -
>
> Key: YARN-8740
> URL: https://issues.apache.org/jira/browse/YARN-8740
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8740-YARN-3409.001.patch,
> YARN-8740-YARN-3409.002.patch, YARN-8740-YARN-3409.003.patch,
> YARN-8740-YARN-3409.004.patch
>
>
> During CI run we observed that testcases are failing due to old attribute
> store
> Two updates are done as part of this jira
> # Skip sending scheduler event when update attributes map is empty
> # Clear Node attributes on testruns
> *CI logs*
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.internalUpdateAttributesOnNodes(NodeAttributesManagerImpl.java:223)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.processMapping(NodeAttributesManagerImpl.java:703)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.replaceNodeAttributes(NodeAttributesManagerImpl.java:671)
> at
> org.apache.hadoop.yarn.nodelabels.store.op.NodeAttributeMirrorOp.recover(NodeAttributeMirrorOp.java:56)
> at
> org.apache.hadoop.yarn.nodelabels.store.op.NodeAttributeMirrorOp.recover(NodeAttributeMirrorOp.java:35)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611984#comment-16611984 ] Hudson commented on YARN-8103: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8103. Add CLI interface to query node attributes. Contributed by (sunilg: rev eb08543c7a5d1fd97a1915dbc5a11a2ba2066ba1) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributeInfoPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeReport.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodeInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ClusterCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeAttributesCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestClusterCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestNodeAttributesCLI.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.004.patch, YARN-8103-YARN-3409.005.patch, > YARN-8103-YARN-3409.006.patch, YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6856) Support CLI for Node Attributes Mapping
[ https://issues.apache.org/jira/browse/YARN-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611966#comment-16611966 ] Hudson commented on YARN-6856: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-6856. [YARN-3409] Support CLI for Node Attributes Mapping. (sunilg: rev 2475fb0a1e2158aac11829c6e33eae3e02bf33ea) * (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeAttributesCLI.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestNodeAttributesCLI.java > Support CLI for Node Attributes Mapping > --- > > Key: YARN-6856 > URL: https://issues.apache.org/jira/browse/YARN-6856 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-6856-YARN-3409.001.patch, > YARN-6856-YARN-3409.002.patch, YARN-6856-YARN-3409.004.patch, > YARN-6856-yarn-3409.003.patch, YARN-6856-yarn-3409.004.patch > > > This focuses on the new CLI for the mapping of Node Attributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6855) CLI Proto Modifications to support Node Attributes
[ https://issues.apache.org/jira/browse/YARN-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611963#comment-16611963 ] Hudson commented on YARN-6855: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-6855. [YARN-3409] CLI Proto Modifications to support Node (sunilg: rev 1f42ce907a4987ea095ae0c237b89274c689ed99) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/DefaultRMAdminRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/AttributeMappingOperationType.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeToAttributes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingResponsePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodesToAttributesMappingRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingRequestPBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodesToAttributesMappingResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttributeType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/rmadmin/PassThroughRMAdminRequestInterceptor.java > CLI Proto Modifications to support Node Attributes > -- > > Key: YARN-6855 > URL: https://issues.apache.org/jira/browse/YARN-6855 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Labels: node-attributes > Fix For: 3.2.0 > > Attachments: YARN-6855-YARN-3409.001.patch, > YARN-6855-YARN-3409.002.patch, YARN-6855-YARN-3409.003.patch, > YARN-6855-YARN-3409.004.patch, YARN-6855-YARN-3409.005.patch, > YARN-6855-YARN-3409.006.patch, YARN-6855-yarn-3409.006.patch > > > This jira focuses only on the proto modifications required for the CLI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7842) PB changes to carry node-attributes in NM heartbeat
[ https://issues.apache.org/jira/browse/YARN-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611967#comment-16611967 ] Hudson commented on YARN-7842: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7842. PB changes to carry node-attributes in NM heartbeat. (sunilg: rev d9d93e392508da2e8b88faefe002128e245fc8f3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/protocolrecords/TestProtocolRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java > PB changes to carry node-attributes in NM heartbeat > --- > > Key: YARN-7842 > URL: https://issues.apache.org/jira/browse/YARN-7842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7842-YARN-3409.001.patch, > YARN-7842-YARN-3409.002.patch > > > PB changes to carry node-attributes in NM heartbeat. Split from a larger > patch for easier review. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8092) Expose Node Attributes info via RM nodes REST API
[ https://issues.apache.org/jira/browse/YARN-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611976#comment-16611976 ] Hudson commented on YARN-8092: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8092. Expose Node Attributes info via RM nodes REST API. (sunilg: rev 89b3ebd11ea93bf9d3521677f0fb9ab3c373a6ca) * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodeAttributesInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodeInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodeAttributeInfo.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java > Expose Node Attributes info via RM nodes REST API > - > > Key: YARN-8092 > URL: https://issues.apache.org/jira/browse/YARN-8092 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8092-YARN-3409.001.patch, > YARN-8092-YARN-3409.002.patch, YARN-8092-YARN-3409.003.patch, > node_attributes_via_config.png > > > Query node-attributes info via nodes REST API via RM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8739) Fix jenkins issues for Node Attributes branch
[ https://issues.apache.org/jira/browse/YARN-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611989#comment-16611989 ] Hudson commented on YARN-8739: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by (sunilg: rev c44088ac190e515b099183aeed4f9d6f8bee7da6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestFileSystemNodeAttributeStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeLabel.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestNodeLabelUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/MockResourceManagerFacade.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodesToAttributesMappingResponsePBImpl.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesStoreEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/RemoveClusterLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/NodeToLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodesToAttributesMappingResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeToAttributesPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/StoreOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/FSStoreOpHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/NodeLabelMirrorOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/AddClusterLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NonAppendableFSNodeLabelStore.java * (edit)
[jira] [Commented] (YARN-8721) Relax NE node-attribute check when attribute doesn't exist on a node
[ https://issues.apache.org/jira/browse/YARN-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611987#comment-16611987 ] Hudson commented on YARN-8721: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8721. Relax NE node-attribute check when attribute doesn't exist on (sunilg: rev 52194351e7df33b8438569c3a032f73d696c534d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/TestSingleConstraintAppPlacementAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintsUtil.java > Relax NE node-attribute check when attribute doesn't exist on a node > > > Key: YARN-8721 > URL: https://issues.apache.org/jira/browse/YARN-8721 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8721-YARN-3409.001.patch, > YARN-8721-YARN-3409.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6858) Attribute Manager to store and provide node attributes in RM
[ https://issues.apache.org/jira/browse/YARN-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611970#comment-16611970 ] Hudson commented on YARN-6858: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-6858. Attribute Manager to store and provide node attributes in RM. (sunilg: rev 2f7712be09d5b68978547f95b1be2352cdd46266) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/AttributeValue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeLabel.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesStoreEvent.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesStoreEventType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelUtil.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/AbstractLabel.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/StringAttributeValue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/AttributeExpressionOperation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java > Attribute Manager to store and provide node attributes in RM > > > Key: YARN-6858 > URL: https://issues.apache.org/jira/browse/YARN-6858 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, capacityscheduler, client >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-6858-YARN-3409.001.patch, > YARN-6858-YARN-3409.002.patch, YARN-6858-YARN-3409.003.patch, > YARN-6858-YARN-3409.004.patch, YARN-6858-YARN-3409.005.patch, > YARN-6858-YARN-3409.006.patch, YARN-6858-YARN-3409.007.patch, > YARN-6858-YARN-3409.008.patch, YARN-6858-YARN-3409.009.patch > > > Similar to CommonNodeLabelsManager we need to have a centralized manager for > Node Attributes too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7988) Refactor FSNodeLabelStore code for Node Attributes store support
[ https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611974#comment-16611974 ] Hudson commented on YARN-7988: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7988. Refactor FSNodeLabelStore code for Node Attributes store (sunilg: rev 6f4bc49c6d153e03edcc9b3a796fb1ccc95334b0) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/NodeToLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NonAppendableFSNodeLabelStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/FSNodeStoreLogOp.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/FSStoreOpHandler.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/StoreOp.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/NodeLabelMirrorOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NullRMNodeLabelsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/AddClusterLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RemoveFromClusterNodeLabelsRequestPBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/RemoveClusterLabelOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java > Refactor FSNodeLabelStore code for Node Attributes store support > > > Key: YARN-7988 > URL: https://issues.apache.org/jira/browse/YARN-7988 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7988-YARN-3409.002.patch, > YARN-7988-YARN-3409.003.patch, YARN-7988-YARN-3409.004.patch, > YARN-7988-YARN-3409.005.patch, YARN-7988-YARN-3409.006.patch, > YARN-7988-YARN-3409.007.patch, YARN-7988.001.patch > > > # Abstract out file FileSystemStore operation > # Define EditLog Operartions and Mirror operation > # Support compatibility with old nodelabel store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7757) Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers
[
https://issues.apache.org/jira/browse/YARN-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611969#comment-16611969
]
Hudson commented on YARN-7757:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14931/])
YARN-7757. Refactor NodeLabelsProvider to be more generic and reusable (sunilg:
rev d312b5cf9016432ec5a7e64b81ab7a8c31f3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestConfigurationNodeLabelsProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ScriptBasedNodeLabelsProvider.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeDescriptorsProvider.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeDescriptorsScriptRunner.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeAttributesProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ConfigurationNodeLabelsProvider.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestScriptBasedNodeAttributesProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestScriptBasedNodeLabelsProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (delete)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/AbstractNodeLabelsProvider.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/package-info.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/AbstractNodeDescriptorsProvider.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ScriptBasedNodeAttributesProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
> Refactor NodeLabelsProvider to be more generic and reusable for node
> attributes providers
> -
>
> Key: YARN-7757
> URL: https://issues.apache.org/jira/browse/YARN-7757
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: YARN-7757-YARN-3409.001.patch,
> YARN-7757-YARN-3409.002.patch, YARN-7757-YARN-3409.003.patch,
> YARN-7757-YARN-3409.004.patch, YARN-7757-YARN-3409.005.patch,
> YARN-7757-YARN-3409.006.patch,
> nodeLabelsProvider_refactor_class_hierarchy.pdf,
> nodeLabelsProvider_refactor_v2.pdf, nodeLabelsProvider_refactor_v3.pdf
>
>
> Propose to do refactor on {{NodeLabelsProvider}},
> {{AbstractNodeLabelsProvider}} to be more generic, so node attributes
> providers can reuse these interface/abstract classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8117) Fix TestRMWebServicesNodes test failure
[ https://issues.apache.org/jira/browse/YARN-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611978#comment-16611978 ] Hudson commented on YARN-8117: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8117. Fix TestRMWebServicesNodes test failure. Contributed by Bibin (sunilg: rev a6590c1f1f7cacd3843265f6e6227f1221205865) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java > Fix TestRMWebServicesNodes test failure > --- > > Key: YARN-8117 > URL: https://issues.apache.org/jira/browse/YARN-8117 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8117-YARN-3409.001.patch > > > Need to fix conflict in TestRMWebServices due to YARN-7792 and YARN-8092 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8033) CLI Integration with NodeAttributesManagerImpl
[ https://issues.apache.org/jira/browse/YARN-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611977#comment-16611977 ] Hudson commented on YARN-8033: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-8033. CLI Integration with NodeAttributesManagerImpl. Contributed (sunilg: rev 901e85238da08129374d44af80716b07f0b912e6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestNodeAttributesCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeAttributesCLI.java > CLI Integration with NodeAttributesManagerImpl > -- > > Key: YARN-8033 > URL: https://issues.apache.org/jira/browse/YARN-8033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8033-YARN-3409.001.patch, > YARN-8033-YARN-3409.002.patch, YARN-8033-YARN-3409.003.patch, > YARN-8033-YARN-3409.004.patch, YARN-8033-YARN-3409.WIP.patch > > > In continuation with YARN-6856 & YARN-6858, need to integrate CLI to the > NodeAttributeManager to complete the functionality of Centralised mapping of > Node Attributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7892) Revisit NodeAttribute class structure
[ https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611982#comment-16611982 ] Hudson commented on YARN-7892: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7892. Revisit NodeAttribute class structure. Contributed by (sunilg: rev 8cf6a9a2bde9d4ccad95a514885634792889ec7f) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributeInfoPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodeAttributesResponse.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributeKeyPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttribute.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttributeInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestNodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestFileSystemNodeAttributeStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestConfigurationNodeAttributesProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAttributesToNodesResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeAttributePBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeToAttributeValue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodeAttributesResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodeAttributeInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAttributesToNodesResponsePBImpl.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttributeKey.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * (edit)
[jira] [Commented] (YARN-7875) Node Attribute store for storing and recovering attributes
[ https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611979#comment-16611979 ] Hudson commented on YARN-7875: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7875. Node Attribute store for storing and recovering attributes. (sunilg: rev b9890d1f6604d572cd9f1cb719e2a4da08b31eff) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeAttribute.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/NodeAttributeMirrorOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributeStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/ReplaceNodeToAttributeLogOp.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestFileSystemNodeAttributeStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestNodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/FSStoreOpHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/AddNodeToAttributeLogOp.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/RemoveNodeToAttributeLogOp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/FSNodeStoreLogOp.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/op/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/FileSystemNodeAttributeStore.java > Node Attribute store for storing and recovering attributes > -- > > Key: YARN-7875 > URL: https://issues.apache.org/jira/browse/YARN-7875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7875-WIP.patch, YARN-7875-YARN-3409.001.patch, > YARN-7875-YARN-3409.002.patch, YARN-7875-YARN-3409.003.patch, > YARN-7875-YARN-3409.004.patch, YARN-7875-YARN-3409.005.patch, > YARN-7875-YARN-3409.006.patch, YARN-7875-YARN-3409.007.patch, > YARN-7875-YARN-3409.008.patch > > > Similar to NodeLabelStore need to support NodeAttributeStore for persisting > attributes mapping to Nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611986#comment-16611986 ] Hudson commented on YARN-7863: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7863. Modify placement constraints to support node attributes. (sunilg: rev 67ae81f0e0ac7f107261ee15f2eb4d189e3b1983) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/api/resource/TestPlacementConstraintParser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/pb/PlacementConstraintFromProtoConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraints.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/constraint/PlacementConstraintParser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/pb/PlacementConstraintToProtoConverter.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/NodeAttributesUpdateSchedulerEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalityAppPlacementAllocator.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeAttributeOpCode.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintsUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/PlacementSpec.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/SchedulerEventType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraint.java > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7965) NodeAttributeManager add/get API is not working properly
[ https://issues.apache.org/jira/browse/YARN-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611972#comment-16611972 ] Hudson commented on YARN-7965: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7965. NodeAttributeManager add/get API is not working properly. (sunilg: rev 86d024ef2a5844ac723c7b597689afa4b598ee27) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestNodeAttributesManager.java > NodeAttributeManager add/get API is not working properly > > > Key: YARN-7965 > URL: https://issues.apache.org/jira/browse/YARN-7965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: 3.2.0 > > Attachments: YARN-7965-YARN-3409.001.patch, > YARN-7965-YARN-3409.002.patch, YARN-7965-YARN-3409.003.patch, > YARN-7965-YARN-3409.004.patch > > > Fix following issues, > # After add node attributes to the manager, could not retrieve newly added > attributes > # Get cluster attributes API should return empty set when given prefix has > no match > # When an attribute is removed from all nodes, the manager did not remove > this mapping > and add UT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM
[ https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611973#comment-16611973 ] Hudson commented on YARN-7871: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14931/]) YARN-7871. Node attributes reporting from NM to RM. Contributed by (sunilg: rev 3b3b6efe2103244febfe6b4f61989e92bd7bb08a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdater.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/TestConfigurationNodeAttributesProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestNodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeAttributesManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/ConfigurationNodeAttributesProvider.java > Node attributes reporting from NM to RM > > > Key: YARN-7871 > URL: https://issues.apache.org/jira/browse/YARN-7871 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7871-YARN-3409.001.patch, > YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch, > YARN-7871-YARN-3409.004.patch > > > Support to initialize proper attribute provider based on user's configuration. > NM collects node attributes from a configured attribute provider and send > them to RM via HB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS
[ https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611957#comment-16611957 ] Naganarasimha G R commented on YARN-8670: - hi [~Sichen Zhao], please ensure to set target version for desired version to be handled and not Fix version, fix version is updated once its closed > Support scheduling request for SLS input and attributes for Node in SLS > --- > > Key: YARN-8670 > URL: https://issues.apache.org/jira/browse/YARN-8670 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Affects Versions: YARN-3409 >Reporter: Sichen zhao >Priority: Major > > YARN-3409 introduces placement constraint, Currently SLS does not support > specify placement constraint. > YARN-8007 support specifying placement constraint for task containers in SLS. > But there are still > some room for improvement: > # YARN-8007 only support placement constraint for the jobs level. In fact, > the more flexible way is support placement constraint for the tasks level. > # In most scenarios, node itself has some characteristics, called attribute, > which is not supported in SLS. So we can add the attribute on Nodes. > # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create > a large number of specific resource requests. We wanna create a new > schedulingrequest input format(like sis format) for the more authentic input. > We can add some field in sis format, and this is schedulingrequest input > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8670) Support scheduling request for SLS input and attributes for Node in SLS
[ https://issues.apache.org/jira/browse/YARN-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8670: Fix Version/s: (was: YARN-3409) > Support scheduling request for SLS input and attributes for Node in SLS > --- > > Key: YARN-8670 > URL: https://issues.apache.org/jira/browse/YARN-8670 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Affects Versions: YARN-3409 >Reporter: Sichen zhao >Priority: Major > > YARN-3409 introduces placement constraint, Currently SLS does not support > specify placement constraint. > YARN-8007 support specifying placement constraint for task containers in SLS. > But there are still > some room for improvement: > # YARN-8007 only support placement constraint for the jobs level. In fact, > the more flexible way is support placement constraint for the tasks level. > # In most scenarios, node itself has some characteristics, called attribute, > which is not supported in SLS. So we can add the attribute on Nodes. > # YARN-8007 use the SYNTH as schedulingrequest input. But SYNTH can't create > a large number of specific resource requests. We wanna create a new > schedulingrequest input format(like sis format) for the more authentic input. > We can add some field in sis format, and this is schedulingrequest input > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7988) Refactor FSNodeLabelStore code for Node Attributes store support
[ https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7988: Fix Version/s: (was: YARN-3409) 3.2.0 > Refactor FSNodeLabelStore code for Node Attributes store support > > > Key: YARN-7988 > URL: https://issues.apache.org/jira/browse/YARN-7988 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7988-YARN-3409.002.patch, > YARN-7988-YARN-3409.003.patch, YARN-7988-YARN-3409.004.patch, > YARN-7988-YARN-3409.005.patch, YARN-7988-YARN-3409.006.patch, > YARN-7988-YARN-3409.007.patch, YARN-7988.001.patch > > > # Abstract out file FileSystemStore operation > # Define EditLog Operartions and Mirror operation > # Support compatibility with old nodelabel store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7965) NodeAttributeManager add/get API is not working properly
[ https://issues.apache.org/jira/browse/YARN-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7965: Fix Version/s: (was: YARN-3409) 3.2.0 > NodeAttributeManager add/get API is not working properly > > > Key: YARN-7965 > URL: https://issues.apache.org/jira/browse/YARN-7965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: 3.2.0 > > Attachments: YARN-7965-YARN-3409.001.patch, > YARN-7965-YARN-3409.002.patch, YARN-7965-YARN-3409.003.patch, > YARN-7965-YARN-3409.004.patch > > > Fix following issues, > # After add node attributes to the manager, could not retrieve newly added > attributes > # Get cluster attributes API should return empty set when given prefix has > no match > # When an attribute is removed from all nodes, the manager did not remove > this mapping > and add UT -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7892) Revisit NodeAttribute class structure
[ https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7892: Fix Version/s: (was: YARN-3409) 3.2.0 > Revisit NodeAttribute class structure > - > > Key: YARN-7892 > URL: https://issues.apache.org/jira/browse/YARN-7892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7892-YARN-3409.001.patch, > YARN-7892-YARN-3409.002.patch, YARN-7892-YARN-3409.003.WIP.patch, > YARN-7892-YARN-3409.003.patch, YARN-7892-YARN-3409.004.patch, > YARN-7892-YARN-3409.005.patch, YARN-7892-YARN-3409.006.patch, > YARN-7892-YARN-3409.007.patch, YARN-7892-YARN-3409.008.patch, > YARN-7892-YARN-3409.009.patch, YARN-7892-YARN-3409.010.patch > > > In the existing structure, we had kept the type and value along with the > attribute which would create confusion to the user to understand the APIs as > they would not be clear as to what needs to be sent for type and value while > fetching the mappings for node(s). > As well as equals will not make sense when we compare only for prefix and > name where as values for them might be different. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7875) Node Attribute store for storing and recovering attributes
[ https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7875: Fix Version/s: (was: YARN-3409) 3.2.0 > Node Attribute store for storing and recovering attributes > -- > > Key: YARN-7875 > URL: https://issues.apache.org/jira/browse/YARN-7875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7875-WIP.patch, YARN-7875-YARN-3409.001.patch, > YARN-7875-YARN-3409.002.patch, YARN-7875-YARN-3409.003.patch, > YARN-7875-YARN-3409.004.patch, YARN-7875-YARN-3409.005.patch, > YARN-7875-YARN-3409.006.patch, YARN-7875-YARN-3409.007.patch, > YARN-7875-YARN-3409.008.patch > > > Similar to NodeLabelStore need to support NodeAttributeStore for persisting > attributes mapping to Nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7871) Node attributes reporting from NM to RM
[ https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7871: Fix Version/s: (was: YARN-3409) 3.2.0 > Node attributes reporting from NM to RM > > > Key: YARN-7871 > URL: https://issues.apache.org/jira/browse/YARN-7871 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7871-YARN-3409.001.patch, > YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch, > YARN-7871-YARN-3409.004.patch > > > Support to initialize proper attribute provider based on user's configuration. > NM collects node attributes from a configured attribute provider and send > them to RM via HB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7865) Node attributes documentation
[ https://issues.apache.org/jira/browse/YARN-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7865: Fix Version/s: (was: YARN-3409) 3.2.0 > Node attributes documentation > - > > Key: YARN-7865 > URL: https://issues.apache.org/jira/browse/YARN-7865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Major > Labels: node-attributes > Fix For: 3.2.0 > > Attachments: NodeAttributes.html, YARN-7865-YARN-3409.001.patch, > YARN-7865-YARN-3409.002.patch, YARN-7865-YARN-3409.003.patch, > YARN-7865-YARN-3409.004.patch, YARN-7865-YARN-3409.005.patch > > > We need proper docs to introduce how to enable node-attributes how to > configure providers, how to specify script paths, arguments in configuration, > what should be the proper permission of the script and who will run the > script. Also it would be good to add more info to the description of the > configuration properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7863: Fix Version/s: (was: YARN-3409) 3.2.0 > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7863-YARN-3409.002.patch, > YARN-7863-YARN-3409.003.patch, YARN-7863-YARN-3409.004.patch, > YARN-7863-YARN-3409.005.patch, YARN-7863-YARN-3409.006.patch, > YARN-7863-YARN-3409.007.patch, YARN-7863-YARN-3409.008.patch, > YARN-7863-YARN-3409.009.patch, YARN-7863-YARN-3409.010.patch, > YARN-7863-YARN-3409.011.patch, YARN-7863.v0.patch > > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7856) Validate Node Attributes from NM
[ https://issues.apache.org/jira/browse/YARN-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7856: Fix Version/s: (was: YARN-3409) 3.2.0 > Validate Node Attributes from NM > > > Key: YARN-7856 > URL: https://issues.apache.org/jira/browse/YARN-7856 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, RM >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7856-YARN-3409.001.patch, > YARN-7856-YARN-3409.002.patch > > > NM needs to do proper validation about the attributes before sending them to > RM, this includes > # a valid prefix is presented > # no duplicate entries > # do not allow two attributes with same prefix/name but different types > This could be an utility class that can be used on both RM/NM sides. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7842) PB changes to carry node-attributes in NM heartbeat
[ https://issues.apache.org/jira/browse/YARN-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7842: Fix Version/s: (was: YARN-3409) 3.2.0 > PB changes to carry node-attributes in NM heartbeat > --- > > Key: YARN-7842 > URL: https://issues.apache.org/jira/browse/YARN-7842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7842-YARN-3409.001.patch, > YARN-7842-YARN-3409.002.patch > > > PB changes to carry node-attributes in NM heartbeat. Split from a larger > patch for easier review. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7840) Update PB for prefix support of node attributes
[ https://issues.apache.org/jira/browse/YARN-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7840: Fix Version/s: (was: YARN-3409) 3.2.0 > Update PB for prefix support of node attributes > --- > > Key: YARN-7840 > URL: https://issues.apache.org/jira/browse/YARN-7840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Naganarasimha G R >Priority: Blocker > Fix For: 3.2.0 > > Attachments: YARN-7840-YARN-3409.001.patch, > YARN-7840-YARN-3409.002.patch, YARN-7840-YARN-3409.003.patch, > YARN-7840-YARN-3409.004.patch > > > We need to support prefix (namespace) for node attributes, this will add the > flexibility to provide ability to do proper ACL, avoid naming conflicts etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7757) Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers
[
https://issues.apache.org/jira/browse/YARN-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-7757:
Fix Version/s: (was: YARN-3409)
3.2.0
> Refactor NodeLabelsProvider to be more generic and reusable for node
> attributes providers
> -
>
> Key: YARN-7757
> URL: https://issues.apache.org/jira/browse/YARN-7757
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: YARN-7757-YARN-3409.001.patch,
> YARN-7757-YARN-3409.002.patch, YARN-7757-YARN-3409.003.patch,
> YARN-7757-YARN-3409.004.patch, YARN-7757-YARN-3409.005.patch,
> YARN-7757-YARN-3409.006.patch,
> nodeLabelsProvider_refactor_class_hierarchy.pdf,
> nodeLabelsProvider_refactor_v2.pdf, nodeLabelsProvider_refactor_v3.pdf
>
>
> Propose to do refactor on {{NodeLabelsProvider}},
> {{AbstractNodeLabelsProvider}} to be more generic, so node attributes
> providers can reuse these interface/abstract classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7031) Support distributed node attributes
[ https://issues.apache.org/jira/browse/YARN-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-7031: Fix Version/s: (was: YARN-3409) 3.2.0 > Support distributed node attributes > --- > > Key: YARN-7031 > URL: https://issues.apache.org/jira/browse/YARN-7031 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.2.0 > > Attachments: Distributed node attributes v1.pdf, > YARN-7031-YARN-3409.001.patch > > > Allow nodemanagers to push its attributes to RM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8094) Support configuration based Node Attribute provider
[
https://issues.apache.org/jira/browse/YARN-8094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated YARN-8094:
-
Fix Version/s: (was: YARN-3409)
3.2.0
> Support configuration based Node Attribute provider
> ---
>
> Key: YARN-8094
> URL: https://issues.apache.org/jira/browse/YARN-8094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8094-YARN-3409.001.patch,
> YARN-8094-YARN-3409.002.patch, YARN-8094-YARN-3409.003.patch,
> YARN-8094-YARN-3409.004.patch
>
>
> Fix TODO item in \{{ConfigurationNodeAttributesProvider}}, completely support
> configuration based node attribute provider.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7990) Node attribute prefix definition and validation
[ https://issues.apache.org/jira/browse/YARN-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-7990: - Fix Version/s: 3.2.0 > Node attribute prefix definition and validation > > > Key: YARN-7990 > URL: https://issues.apache.org/jira/browse/YARN-7990 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: Node Attribute Prefix Definition v1.pdf > > > Summary: > # Centralized: rm.yarn.io > # Distributed: nm.yarn.io > # System: *.yarn.io (yarn.io suffix is preserved for other yarn system set > node attributes) > # User-Defined: other form of prefixes except reserved ones > See detail in design doc [^Node Attribute Prefix Definition v1.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8092) Expose Node Attributes info via RM nodes REST API
[ https://issues.apache.org/jira/browse/YARN-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8092: - Fix Version/s: (was: YARN-3409) 3.2.0 > Expose Node Attributes info via RM nodes REST API > - > > Key: YARN-8092 > URL: https://issues.apache.org/jira/browse/YARN-8092 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8092-YARN-3409.001.patch, > YARN-8092-YARN-3409.002.patch, YARN-8092-YARN-3409.003.patch, > node_attributes_via_config.png > > > Query node-attributes info via nodes REST API via RM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
