[jira] [Comment Edited] (YARN-6164) Expose maximum-am-resource-percent in YarnClient
[
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898896#comment-15898896
]
Sunil G edited comment on YARN-6164 at 3/7/17 7:53 AM:
---
Thanks [~leftnoteasy] for sharing the thoughts.
It makes sense by seeing the diversity of params we have in QueueInfo. On the
same note in long run, we are looking for something like {{QueueCapacities}},
{{QueueConfigurations}}, and {{ResourceUsages}} inside *QueueInfo*. Correct?
and {{MaxAMPercentage}} could be part of {{QueueCapacities}} itself.
I guess this could generalized improvement task, and for simplicity we could
use *@Unstable/@Private* version of am percentage as given in current patch. Do
you see this as a fine approach at this point?
was (Author: sunilg):
Thanks [~leftnoteasy] for sharing the thoughts.
It makes sense by seeing the diversity of params we have in QueueInfo. On the
same note in long run, we are looking for something like {{QueueCapacities}},
{{QueueConfigurations}}, and {{ResourceUsages}} inside *QueueInfo*. Correct?
and {{MaxAMPercentage}} could be part of {{QueueCapacities}} itself.
I guess this could generalized improvement task, and for simplicity we could
use *@Unstable/@Private* version of am percentage. Do you see this as a fine
approach at this point?
> Expose maximum-am-resource-percent in YarnClient
>
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch,
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the
> [Cluster Scheduler
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
> but not through
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by
> default), it would be nice to expose `maximum-am-resource-percent` in
> YarnClient as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient
[
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898896#comment-15898896
]
Sunil G commented on YARN-6164:
---
Thanks [~leftnoteasy] for sharing the thoughts.
It makes sense by seeing the diversity of params we have in QueueInfo. On the
same note in long run, we are looking for something like {{QueueCapacities}},
{{QueueConfigurations}}, and {{ResourceUsages}} inside *QueueInfo*. Correct?
and {{MaxAMPercentage}} could be part of {{QueueCapacities}} itself.
I guess this could generalized improvement task, and for simplicity we could
use *@Unstable/@Private* version of am percentage. Do you see this as a fine
approach at this point?
> Expose maximum-am-resource-percent in YarnClient
>
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch,
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the
> [Cluster Scheduler
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
> but not through
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by
> default), it would be nice to expose `maximum-am-resource-percent` in
> YarnClient as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1589#comment-1589 ] Rohith Sharma K S commented on YARN-6209: - Going through whole discussion above, I would suggest to keep this JIRA only for notifying queue name into AM. Label related changes would discuss separately. One of the major difference I see is, changing queue name is *user operation* which has to be informed to ApplicationMaster i.e via push by YARN to AM. To the labels, we may need new first class API that gives full details about QueueInfo which could be a pull model. > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Description: When running a simple wordcount experiment on YARN, I noticed that the task failed to achieve data locality, even though there is no other job running on the cluster at the same time. The experiment was done in a 7-node (1 master, 6 data nodes/node managers) cluster and the input of the wordcount job (both Spark and MapReduce) is a single-block file in HDFS which is two-way replicated (replication factor = 2). I ran wordcount on YARN for 10 times. The results show that only 30% of tasks can achieve data locality, which seems like the result of a random placement of tasks. The experiment details are in the attachment, and feel free to reproduce the experiments. was: When I ran experiments with both Spark and MapReduce wordcount on YARN, I noticed that the task failed to achieve data locality, even though there is no other job running on the cluster. I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on YARN for 10 times with a single data block. The results show that only 30% of tasks can achieve data locality, it seems like random in the placement of tasks. the experiment details are in the attachment, you can reproduce the experiments. > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: Hadoop_Spark_Conf.zip, YARN-DataLocality.docx > > > When running a simple wordcount experiment on YARN, I noticed that the task > failed to achieve data locality, even though there is no other job running on > the cluster at the same time. The experiment was done in a 7-node (1 master, > 6 data nodes/node managers) cluster and the input of the wordcount job (both > Spark and MapReduce) is a single-block file in HDFS which is two-way > replicated (replication factor = 2). I ran wordcount on YARN for 10 times. > The results show that only 30% of tasks can achieve data locality, which > seems like the result of a random placement of tasks. The experiment details > are in the attachment, and feel free to reproduce the experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898807#comment-15898807 ] Sunil G commented on YARN-6148: --- I generally went through whole approach suggested here, and I still feel there are some more grey areas. Could [~varun_saxena] or [~naganarasimha...@apache.org] to put these in a small doc here. I feel this may be needed because there are small feature sets from node label / scheduler is playing impact on MR's blacklisting. Few of them are discussed here, but still its better to brainstorm. To summarize some features which has impact on blacklist are non-exclusive label, move app, default queue's label, preemption of non-exclusive label etc. In hindsight, I feel MR is going to get multiple data sets. Like per-label-node-count, node-report-on-label-change, app-queue-on-change, app-default-label-on-change etc in addition to existing MR blacklist. Its better to get a common consensus here on approach. I thought of looping [~jlowe] and [~leftnoteasy] as changes are proposed in MR and in YARN's interface end. > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-6209: --- Assignee: Rohith Sharma K S (was: Naganarasimha G R) > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-6209: --- Assignee: Naganarasimha G R (was: Rohith Sharma K S) > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6208) Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application
[
https://issues.apache.org/jira/browse/YARN-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898706#comment-15898706
]
Akira Ajisaka commented on YARN-6208:
-
Thanks [~dan...@cloudera.com] for the comment. I just thought it is too long
for logging. If you think it is better to log the reason, I'll update the patch.
> Improve the log when FinishAppEvent sent to the NodeManager which didn't run
> the application
>
>
> Key: YARN-6208
> URL: https://issues.apache.org/jira/browse/YARN-6208
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie, supportability
> Attachments: YARN-6208.01.patch
>
>
> When FinishAppEvent of an application is sent to a NodeManager and there are
> no applications of the application ran on the NodeManager, we can see the
> following log:
> {code}
> 2015-12-28 11:59:18,725 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1446103803043_9892
> {code}
> YARN-4520 made the log as follows:
> {code}
> LOG.warn("couldn't find application " + appID + " while processing"
> + " FINISH_APPS event");
> {code}
> and I'm thinking it can be improved.
> * Make the log WARN from INFO
> * Add why the NodeManager couldn't find the application. For example,
> "because there were no containers of the application ran on the NodeManager."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2904) Use linux cgroups to enhance container tear down
[ https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898688#comment-15898688 ] Feng Yuan edited comment on YARN-2904 at 3/7/17 3:50 AM: - In my humble opinion,Task process leak by some unknowable reason(YARN-6276) is first step,and result to when deleteCgroup() will get os panic(this issue),so i add the relation here.[~jlowe] was (Author: feng yuan): In my humble opinion,Task process leak but some unknowable reason(YARN-6276),and result to when deleteCgroup() will get os panic(this issue),so i add the relation here.[~jlowe] > Use linux cgroups to enhance container tear down > > > Key: YARN-2904 > URL: https://issues.apache.org/jira/browse/YARN-2904 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Nathan Roberts > > If we are launching yarn containers within cgroups, linux provides some > guarantees that can help completely tear down a container. Specifically, > linux guarantees that tasks can't escape a cgroup. We can use this fact to > tear down a yarn container without leaking tasks. > Today, a SIGTERM is sent to the session (normally lead by bash). When the > session leader exits, the LCE sees this and assumes all resources have been > given back to the system. This is not guaranteed. Example: YARN-2809 > implements a workaround that is only necessary because tasks are still > lingering within the cgroup when the nodemanager attempts to delete it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2904) Use linux cgroups to enhance container tear down
[ https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898688#comment-15898688 ] Feng Yuan edited comment on YARN-2904 at 3/7/17 3:49 AM: - In my humble opinion,Task process leak but some unknowable reason(YARN-6276),and result to when deleteCgroup() will get os panic(this issue),so i add the relation here.[~jlowe] was (Author: feng yuan): In my humble opinion,Task process leak but some unknowable reason(YARN-6276),and result to when deleteCgroup() will get os panic(this issue),so i add the relation here. > Use linux cgroups to enhance container tear down > > > Key: YARN-2904 > URL: https://issues.apache.org/jira/browse/YARN-2904 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Nathan Roberts > > If we are launching yarn containers within cgroups, linux provides some > guarantees that can help completely tear down a container. Specifically, > linux guarantees that tasks can't escape a cgroup. We can use this fact to > tear down a yarn container without leaking tasks. > Today, a SIGTERM is sent to the session (normally lead by bash). When the > session leader exits, the LCE sees this and assumes all resources have been > given back to the system. This is not guaranteed. Example: YARN-2809 > implements a workaround that is only necessary because tasks are still > lingering within the cgroup when the nodemanager attempts to delete it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2904) Use linux cgroups to enhance container tear down
[ https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898688#comment-15898688 ] Feng Yuan commented on YARN-2904: - In my humble opinion,Task process leak but some unknowable reason(YARN-6276),and result to when deleteCgroup() will get os panic(this issue),so i add the relation here. > Use linux cgroups to enhance container tear down > > > Key: YARN-2904 > URL: https://issues.apache.org/jira/browse/YARN-2904 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Nathan Roberts > > If we are launching yarn containers within cgroups, linux provides some > guarantees that can help completely tear down a container. Specifically, > linux guarantees that tasks can't escape a cgroup. We can use this fact to > tear down a yarn container without leaking tasks. > Today, a SIGTERM is sent to the session (normally lead by bash). When the > session leader exits, the LCE sees this and assumes all resources have been > given back to the system. This is not guaranteed. Example: YARN-2809 > implements a workaround that is only necessary because tasks are still > lingering within the cgroup when the nodemanager attempts to delete it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Attachment: Hadoop_Spark_Conf.zip > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: Hadoop_Spark_Conf.zip, YARN-DataLocality.docx > > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Issue Type: Bug (was: Improvement) > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: YARN-DataLocality.docx > > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Priority: Major (was: Minor) > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: YARN-DataLocality.docx > > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Attachment: YARN-DataLocality.docx > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-DataLocality.docx > > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Attachment: (was: YARN-6289.01.docx) > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Description: When I ran experiments with both Spark and MapReduce wordcount on YARN, I noticed that the task failed to achieve data locality, even though there is no other job running on the cluster. I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on YARN for 10 times with a single data block. The results show that only 30% of tasks can achieve data locality, it seems like random in the placement of tasks. the experiment details are in the attachment, you can reproduce the experiments. was: When I ran experiments with both Spark and MapReduce wordcount with yarn on the file, I noticed that the job did not get data locality every time. It was seemingly random in the placement of the tasks, even though there is no other job running on the cluster. I expected the task placement to always be on the single machine which is holding the data block, but that did not happen. I run the experiments with a 7 node cluster with 2x replication(1 master, 6 data nodes/node managers) , the experiment details are in the patch so you can recreate the result. In the experiments, I run Spark/MapReduce wordcount with yarn for 10 times in a single block and the results show that only 30% of tasks can satisfy data locality, it seems like random in the placement of tasks. Next,I will run two more experiments(7 node cluster with 2x replication with 2 blocks and 4 blocks) to verify the results and plan to do some optimization work (optimize the schedule policy) to improve data locality > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount on YARN, I > noticed that the task failed to achieve data locality, even though there is > no other job running on the cluster. > I adopted a 7 node (1 master, 6 data nodes/node managers) cluster and set 2x > replication for HDFS. In the experiments, I run Spark/MapReduce wordcount on > YARN for 10 times with a single data block. The results show that only 30% of > tasks can achieve data locality, it seems like random in the placement of > tasks. the experiment details are in the attachment, you can reproduce the > experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898619#comment-15898619 ] Huangkaixuan edited comment on YARN-6289 at 3/7/17 3:33 AM: The detailed environment and results of the experiments are shown in the attachment. was (Author: huangkx6810): The detail results of the experiments are shown in the patch > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. > Next,I will run two more experiments(7 node cluster with 2x replication > with 2 blocks and 4 blocks) to verify the results and plan to do some > optimization work (optimize the schedule policy) to improve data locality -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Summary: Fail to achieve data locality when runing MapReduce and Spark on HDFS (was: yarn got little data locality) > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. > Next,I will run two more experiments(7 node cluster with 2x replication > with 2 blocks and 4 blocks) to verify the results and plan to do some > optimization work (optimize the schedule policy) to improve data locality -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6293) Investigate Java 7 compatibility for new YARN UI
[
https://issues.apache.org/jira/browse/YARN-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898643#comment-15898643
]
Sunil G commented on YARN-6293:
---
cc/ [~Sreenath]
> Investigate Java 7 compatibility for new YARN UI
>
>
> Key: YARN-6293
> URL: https://issues.apache.org/jira/browse/YARN-6293
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Li Lu
>
> Right now when trying the YARN new UI with Java 7, I can get the following
> warning:
> {code}
> [INFO] --- maven-enforcer-plugin:1.4.1:enforce (dist-enforce) @
> hadoop-yarn-ui ---
> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed
> with message:
> Detected JDK Version: 1.7.0-67 is not in the allowed range [1.8,).
> {code}
> While right now this warning does not cause any troubles for trunk
> integration, when some users would like to package the new UI with some
> branch-2 based code, the JDK requirement would block the effort. So the
> question here is, is there any specific component in new UI codebase that
> prevent us using Java 7? I remember it should be a JS based implementation,
> right?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898619#comment-15898619 ] Huangkaixuan edited comment on YARN-6289 at 3/7/17 2:45 AM: The detail results of the experiments are shown in the patch was (Author: huangkx6810): The detail results of the experiment are shown in the patch > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. > Next,I will run two more experiments(7 node cluster with 2x replication > with 2 blocks and 4 blocks) to verify the results and plan to do some > optimization work (optimize the schedule policy) to improve data locality -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898630#comment-15898630 ] Sean Po commented on YARN-5703: --- I apologize for not responding, [~maniraj...@gmail.com] and [~Naganarasimha]. I haven't been checking my email for the last few weeks for personal reasons. Thank you both for contributing, and checking this in! > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > Attachments: YARN-5703.001.patch, YARN-5703.002.patch, > YARN-5703.003.patch, YARN-5703.004.patch, YARN-5703.005.patch, > YARN-5703.006.patch, YARN-5703.007.patch, YARN-5703.008.patch > > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po reassigned YARN-5881: - Assignee: Wangda Tan (was: Sean Po) > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Wangda Tan > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898625#comment-15898625 ] Sean Po commented on YARN-5881: --- Thanks Wangda, please take over the ticket. I have assigned it to you. > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898619#comment-15898619 ] Huangkaixuan commented on YARN-6289: The detail results of the experiment are shown in the patch > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. > Next,I will run two more experiments(7 node cluster with 2x replication > with 2 blocks and 4 blocks) to verify the results and plan to do some > optimization work (optimize the schedule policy) to improve data locality -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6234) Support multiple attempts on the node when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898613#comment-15898613
]
Giovanni Matteo Fumarola commented on YARN-6234:
The failed test is not related to the patch.
> Support multiple attempts on the node when AMRMProxy is enabled
> ---
>
> Key: YARN-6234
> URL: https://issues.apache.org/jira/browse/YARN-6234
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, federation, nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-6234-YARN-2915.v1.patch
>
>
> Currently {{AMRMProxy}} initializes an interceptor chain pipeline for every
> active AM in the node but it doesn't clean up & reinitialize correctly if
> there's a second attempt for any AM in the same node. This jira is to track
> the changes required to support multiple attempts on the node when AMRMProxy
> is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Dump scheduler and queue state information into FairScheduler DEBUG log
[
https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898609#comment-15898609
]
Hadoop QA commented on YARN-6042:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 8s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 40m
53s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6042 |
| GITHUB PR | https://github.com/apache/hadoop/pull/193 |
| Optional Tests | asflicense mvnsite unit compile javac javadoc
mvninstall findbugs checkstyle |
| uname | Linux a23fc4b1c24c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5e74196 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15185/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15185/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15185/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
[jira] [Updated] (YARN-5579) Resourcemanager should surface failed state store operation prominently
[
https://issues.apache.org/jira/browse/YARN-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated YARN-5579:
-
Description:
I found the following in Resourcemanager log when I tried to figure out why
application got stuck in NEW_SAVING state.
{code}
2016-08-29 18:14:23,486 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:runWithRetries(1242)) - Maxed out ZK retries. Giving up!
2016-08-29 18:14:23,486 ERROR recovery.RMStateStore
(RMStateStore.java:transition(205)) - Error storing app:
application_1470517915158_0001
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
AuthFailed
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$5.run(ZKRMStateStore.java:998)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$5.run(ZKRMStateStore.java:995)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1174)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1207)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:995)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:1009)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:1042)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:639)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:201)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:183)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:955)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1036)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1031)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:745)
2016-08-29 18:14:23,486 ERROR recovery.RMStateStore
(RMStateStore.java:notifyStoreOperationFailedInternal(987)) - State store
operation failed
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
AuthFailed
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$5.run(ZKRMStateStore.java:998)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$5.run(ZKRMStateStore.java:995)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1174)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1207)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:995)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:1009)
{code}
Resourcemanager should surface the above error prominently.
Likely subsequent application submission would encounter the same error.
was:
I found the following in Resourcemanager log when I tried to figure out why
application got stuck in NEW_SAVING state.
{code}
2016-08-29 18:14:23,486 INFO recovery.ZKRMStateStore
(ZKRMStateStore.java:runWithRetries(1242)) - Maxed out ZK retries. Giving up!
2016-08-29 18:14:23,486 ERROR recovery.RMStateStore
(RMStateStore.java:transition(205)) - Error storing app:
[jira] [Commented] (YARN-6234) Support multiple attempts on the node when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898543#comment-15898543
]
Hadoop QA commented on YARN-6234:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
45s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 23s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6234 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856390/YARN-6234-YARN-2915.v1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 57b29d81b5d9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 0c551c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15187/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15187/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15187/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support multiple attempts on the node when AMRMProxy is enabled
> ---
>
> Key: YARN-6234
>
[jira] [Commented] (YARN-6146) Add Builder methods for TimelineEntityFilters
[
https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898467#comment-15898467
]
Hadoop QA commented on YARN-6146:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
58s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
27s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 24 new
+ 38 unchanged - 6 fixed = 62 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
13s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-6146 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856376/YARN-6146-YARN-5355.01.patch
|
| Optional
[jira] [Updated] (YARN-6234) Support multiple attempts on the node when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-6234:
---
Attachment: YARN-6234-YARN-2915.v1.patch
> Support multiple attempts on the node when AMRMProxy is enabled
> ---
>
> Key: YARN-6234
> URL: https://issues.apache.org/jira/browse/YARN-6234
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, federation, nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-6234-YARN-2915.v1.patch
>
>
> Currently {{AMRMProxy}} initializes an interceptor chain pipeline for every
> active AM in the node but it doesn't clean up & reinitialize correctly if
> there's a second attempt for any AM in the same node. This jira is to track
> the changes required to support multiple attempts on the node when AMRMProxy
> is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6146) Add Builder methods for TimelineEntityFilters
[ https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6146: - Attachment: YARN-6146.01.patch > Add Builder methods for TimelineEntityFilters > - > > Key: YARN-6146 > URL: https://issues.apache.org/jira/browse/YARN-6146 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6146.01.patch, YARN-6146-YARN-5355.01.patch > > > The timeline filters are evolving and can be add more and more filters. It is > better to start using Builder methods rather than changing constructor every > time for adding new filters. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6146) Add Builder methods for TimelineEntityFilters
[ https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6146: - Attachment: YARN-6146-YARN-5355.01.patch Uploading a patch for branch YARN-5355 > Add Builder methods for TimelineEntityFilters > - > > Key: YARN-6146 > URL: https://issues.apache.org/jira/browse/YARN-6146 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6146-YARN-5355.01.patch > > > The timeline filters are evolving and can be add more and more filters. It is > better to start using Builder methods rather than changing constructor every > time for adding new filters. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Dump scheduler and queue state information into FairScheduler DEBUG log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898389#comment-15898389 ] Yufei Gu commented on YARN-6042: Thanks for the review [~rchiang], uploaded patch v9 to get logger name correctly. BTW, [~Tao Jie], continue the previous discussion actually no need to add a new link, you can find the new log file in RM webUI by going to "tool" -> "Local Logs" > Dump scheduler and queue state information into FairScheduler DEBUG log > --- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch, YARN-6042.007.patch, YARN-6042.008.patch, > YARN-6042.009.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6042) Dump scheduler and queue state information into FairScheduler DEBUG log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6042: --- Attachment: YARN-6042.009.patch > Dump scheduler and queue state information into FairScheduler DEBUG log > --- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch, YARN-6042.007.patch, YARN-6042.008.patch, > YARN-6042.009.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[
https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898375#comment-15898375
]
Hadoop QA commented on YARN-5948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
25s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
40s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 10s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856350/YARN-5948-YARN-5734.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 3e1e6daeb7aa 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5734 / 01ea2f3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (YARN-6208) Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application
[
https://issues.apache.org/jira/browse/YARN-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898372#comment-15898372
]
Daniel Templeton commented on YARN-6208:
Thanks for the patch, [~ajisakaa]. I like the additional comment. Any reason
not to make that the text of the log message?
> Improve the log when FinishAppEvent sent to the NodeManager which didn't run
> the application
>
>
> Key: YARN-6208
> URL: https://issues.apache.org/jira/browse/YARN-6208
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie, supportability
> Attachments: YARN-6208.01.patch
>
>
> When FinishAppEvent of an application is sent to a NodeManager and there are
> no applications of the application ran on the NodeManager, we can see the
> following log:
> {code}
> 2015-12-28 11:59:18,725 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1446103803043_9892
> {code}
> YARN-4520 made the log as follows:
> {code}
> LOG.warn("couldn't find application " + appID + " while processing"
> + " FINISH_APPS event");
> {code}
> and I'm thinking it can be improved.
> * Make the log WARN from INFO
> * Add why the NodeManager couldn't find the application. For example,
> "because there were no containers of the application ran on the NodeManager."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6294) ATS client should better handle Socket closed case
Junping Du created YARN-6294:
Summary: ATS client should better handle Socket closed case
Key: YARN-6294
URL: https://issues.apache.org/jira/browse/YARN-6294
Project: Hadoop YARN
Issue Type: Bug
Components: timelineclient
Reporter: Sumana Sathish
Assignee: Li Lu
Exception stack:
{noformat}
17/02/06 07:11:30 INFO distributedshell.ApplicationMaster: Container completed
successfully., containerId=container_1486362713048_0037_01_02
17/02/06 07:11:30 ERROR distributedshell.ApplicationMaster: Error in
RMCallbackHandler:
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException:
Socket closed
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:236)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:185)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:248)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at
com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
at
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPostingObject(TimelineWriter.java:154)
at
org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:115)
at
org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:112)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1833)
at
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)
at
org.apache.hadoop.yarn.client.api.impl.TimelineWriter.putEntities(TimelineWriter.java:92)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:346)
at
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishContainerEndEvent(ApplicationMaster.java:1145)
at
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.access$400(ApplicationMaster.java:169)
at
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$RMCallbackHandler.onContainersCompleted(ApplicationMaster.java:779)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:296)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 20 more
Exception in thread "AMRM Callback Handler Thread"
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
[ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898331#comment-15898331 ] Wangda Tan commented on YARN-6285: -- [~zhaoyunjiong], from your previous comment: bq. 1. Slowness in getApplications, below stack trace files shows it spend at least 2.25 seconds in getApplications. Does it include ResourceRequest? From our experiences, the there're lots of information in ResourceRequest/getLogAggregationReportsForApp and it should contribute to most of the data/size of get_applications responses. If it is possible, could you do a test to see, how much time will be spent if we don't include ResourceRequest/getLogAggregationReportsForApp in the get_applications response? This will be very helpful for us to make decisions. To solve the problem, instead of adding the limit of apps in the server side, I would prefer to optimize following items: 1) Add parameter to indicate if we should include ResourceRequest/getLogAggregationReportsForApp in the response, default is true to make it compatible. (Can be done if above experimental shows it really helps). 2) If required, use cache to store finished apps since their report will not change. (Can be done separately) In addition, following items can be optimized in client side: 3) Properly use filter to include only app with expected states (RUNNING/ACCEPT) to avoid including all apps. 4) Set limit when requesting app reports from client. Thoughts? > Add option to set max limit on ResourceManager for > ApplicationClientProtocol.getApplications > > > Key: YARN-6285 > URL: https://issues.apache.org/jira/browse/YARN-6285 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: YARN-6285.001.patch, YARN-6285.002.patch, > YARN-6285.003.patch > > > When users called ApplicationClientProtocol.getApplications, it will return > lots of data, and generate lots of garbage on ResourceManager which caused > long time GC. > For example, on one of our RM, when called rest API " http:// address:port>/ws/v1/cluster/apps" it can return 150MB data which have 944 > applications. > getApplications have limit parameter, but some user might not set it, and > then the limit will be Long.MAX_VALUE. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6287) RMCriticalThreadUncaughtExceptionHandler.rmContext should be final
[
https://issues.apache.org/jira/browse/YARN-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-6287:
--
Assignee: Corey Barker (was: Yufei Gu)
> RMCriticalThreadUncaughtExceptionHandler.rmContext should be final
> --
>
> Key: YARN-6287
> URL: https://issues.apache.org/jira/browse/YARN-6287
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: Corey Barker
>Priority: Minor
> Labels: newbie
>
> {code}
> private RMContext rmContext;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-516) TestContainerLocalizer.testContainerLocalizerMain is failing
[ https://issues.apache.org/jira/browse/YARN-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved YARN-516. -- Resolution: Later Resolving this old stale JIRA, unlikely we're going to do anything on HADOOP-9357. > TestContainerLocalizer.testContainerLocalizerMain is failing > > > Key: YARN-516 > URL: https://issues.apache.org/jira/browse/YARN-516 > Project: Hadoop YARN > Issue Type: Test >Reporter: Vinod Kumar Vavilapalli >Assignee: Andrew Wang > Attachments: YARN-516.txt > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5665) Enhance documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898272#comment-15898272 ] Hudson commented on YARN-5665: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11355 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11355/]) YARN-5665. Enhance documentation for (rchiang: rev d9dc444dc73fbe23f9e553d63baf83f12c636fa7) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > Enhance documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5665) Enhance documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898270#comment-15898270 ] Yufei Gu commented on YARN-5665: Thanks [~rchiang] for the review and commit. > Enhance documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5665) Enhance documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898270#comment-15898270 ] Yufei Gu edited comment on YARN-5665 at 3/6/17 10:27 PM: - Thanks [~rchiang] for the review and commit. Thanks [~miklos.szeg...@cloudera.com] for the review. was (Author: yufeigu): Thanks [~rchiang] for the review and commit. > Enhance documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898262#comment-15898262
]
Wangda Tan commented on YARN-6050:
--
[~rkanter],
The reason of "nodeA:0" means "nodeA:*" which will be used when admin want to
add labels to all NMs launched on the same host (YARN support launch multiple
NMs per node), or when admin is too lazy to specify NM-port. Since the
"wildcard NM" doesn't mean a physical active NM, so we didn't count it when
getActiveNMPerLabel.
To your question, you can either add a method to get active NMs, or just ignore
":0" NM.
And does the code in your latest comment handle #active-nms for each rack?
> AMs can't be scheduled on racks or nodes
>
>
> Key: YARN-6050
> URL: https://issues.apache.org/jira/browse/YARN-6050
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6050.001.patch, YARN-6050.002.patch,
> YARN-6050.003.patch, YARN-6050.004.patch, YARN-6050.005.patch,
> YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch
>
>
> Yarn itself supports rack/node aware scheduling for AMs; however, there
> currently are two problems:
> # To specify hard or soft rack/node requests, you have to specify more than
> one {{ResourceRequest}}. For example, if you want to schedule an AM only on
> "rackA", you have to create two {{ResourceRequest}}, like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS,
> true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more
> than one {{ResourceRequest}} in the {{ApplicationSubmissionContext}}. The
> current behavior is to either build one from {{getResource}} or directly from
> {{getAMContainerResourceRequest}}, depending on if
> {{getAMContainerResourceRequest}} is null or not. We'll need to add a third
> method, say {{getAMContainerResourceRequests}}, which takes a list of
> {{ResourceRequest}} so that clients can specify the multiple resource
> requests.
> # There are some places where things are hardcoded to overwrite what the
> client specifies. These are pretty straightforward to fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5665) Enhance documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898244#comment-15898244 ] Ray Chiang commented on YARN-5665: -- Sorry for the delay. +1. Committing soon. > Enhance documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[
https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898242#comment-15898242
]
Hadoop QA commented on YARN-5948:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
54s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
45s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 327 unchanged - 0 fixed = 328 total (was 327) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
7s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 56m
44s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856339/YARN-5948-YARN-5734.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 0783c9470f6c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5734 / 01ea2f3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Resolved] (YARN-6292) YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is specified in fs.defaultFS
[ https://issues.apache.org/jira/browse/YARN-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ang Zhang resolved YARN-6292. - Resolution: Duplicate Close as a duplicate of YARN-3269 > YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is > specified in fs.defaultFS > --- > > Key: YARN-6292 > URL: https://issues.apache.org/jira/browse/YARN-6292 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ang Zhang >Priority: Minor > > 2017-03-02 17:59:13,688 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: > Error in dispatcher thread > java.lang.IllegalArgumentException: Wrong FS: viewfs://ns-default/tmp/logs, > expected: hdfs://nameservice1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:321) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:445) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5665) Enhance documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5665: - Summary: Enhance documentation for yarn.resourcemanager.scheduler.class property (was: Improve documentation for yarn.resourcemanager.scheduler.class property) > Enhance documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5665) Improve documentation for yarn.resourcemanager.scheduler.class property
[ https://issues.apache.org/jira/browse/YARN-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5665: - Summary: Improve documentation for yarn.resourcemanager.scheduler.class property (was: Documentation does not mention package name requirement for yarn.resourcemanager.scheduler.class) > Improve documentation for yarn.resourcemanager.scheduler.class property > --- > > Key: YARN-5665 > URL: https://issues.apache.org/jira/browse/YARN-5665 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Trivial > Labels: doc, newbie > Attachments: YARN-5665.001.patch, YARN-5665.002.patch, > YARN-5665.003.patch > > > http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/ClusterSetup.html > refers to FairScheduler, when it documents the setting > yarn.resourcemanager.scheduler.class. What it forgets to mention is that the > user has to specify the full class path like > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > otherwise the system throws java.lang.ClassNotFoundException: FairScheduler. > It would be nice, if the documentation specified the full class path, so that > the user does not need to look it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6292) YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is specified in fs.defaultFS
[ https://issues.apache.org/jira/browse/YARN-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898225#comment-15898225 ] Ang Zhang commented on YARN-6292: - Yes [~jlowe], it is a duplicate of YARN-3269, thanks! > YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is > specified in fs.defaultFS > --- > > Key: YARN-6292 > URL: https://issues.apache.org/jira/browse/YARN-6292 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ang Zhang >Priority: Minor > > 2017-03-02 17:59:13,688 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: > Error in dispatcher thread > java.lang.IllegalArgumentException: Wrong FS: viewfs://ns-default/tmp/logs, > expected: hdfs://nameservice1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:321) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:445) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6293) Investigate Java 7 compatibility for new YARN UI
[
https://issues.apache.org/jira/browse/YARN-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898221#comment-15898221
]
Li Lu commented on YARN-6293:
-
Actually I just directly changed the ui module's pom.xml. I changed the parent
and the current module's version to 2.x. Right now the build passed. UI
experts, does this hide any potential issues? Thanks!
> Investigate Java 7 compatibility for new YARN UI
>
>
> Key: YARN-6293
> URL: https://issues.apache.org/jira/browse/YARN-6293
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Li Lu
>
> Right now when trying the YARN new UI with Java 7, I can get the following
> warning:
> {code}
> [INFO] --- maven-enforcer-plugin:1.4.1:enforce (dist-enforce) @
> hadoop-yarn-ui ---
> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed
> with message:
> Detected JDK Version: 1.7.0-67 is not in the allowed range [1.8,).
> {code}
> While right now this warning does not cause any troubles for trunk
> integration, when some users would like to package the new UI with some
> branch-2 based code, the JDK requirement would block the effort. So the
> question here is, is there any specific component in new UI codebase that
> prevent us using Java 7? I remember it should be a JS based implementation,
> right?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient
[
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898192#comment-15898192
]
Wangda Tan commented on YARN-6164:
--
Adding {{QueueConfigurations}} to QueueInfo is a good idea to me, but since it
is supposed to handle other options like capacity/preemption-disabled, etc. I
would prefer to continue existing approach of this patch, and think about how
to add QueueConfiguration separately.
In my opinion MaxAMPercentage is subject to be changed, so would prefer to make
it {{@Unstable/@Private}} for MaxAMPercentages.java (methods and class), and
QueueInfo#getMaxAMPercentages.
Thoughts?
> Expose maximum-am-resource-percent in YarnClient
>
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch,
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the
> [Cluster Scheduler
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
> but not through
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by
> default), it would be nice to expose `maximum-am-resource-percent` in
> YarnClient as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[ https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5948: Attachment: YARN-5948-YARN-5734.005.patch > Implement MutableConfigurationManager for handling storage into configuration > store > --- > > Key: YARN-5948 > URL: https://issues.apache.org/jira/browse/YARN-5948 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5948.001.patch, YARN-5948-YARN-5734.002.patch, > YARN-5948-YARN-5734.003.patch, YARN-5948-YARN-5734.004.patch, > YARN-5948-YARN-5734.005.patch > > > The MutableConfigurationManager will take REST calls with desired client > configuration changes and call YarnConfigurationStore methods to store these > changes in the backing store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5418) When partial log aggregation is enabled, display the list of aggregated files on the container log page
[ https://issues.apache.org/jira/browse/YARN-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898190#comment-15898190 ] Xuan Gong commented on YARN-5418: - uploaded a patch for this. > When partial log aggregation is enabled, display the list of aggregated files > on the container log page > --- > > Key: YARN-5418 > URL: https://issues.apache.org/jira/browse/YARN-5418 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: Screen Shot 2017-03-06 at 1.38.04 PM.png, > YARN-5418.1.patch > > > The container log pages lists all files. However, as soon as a file gets > aggregated - it's no longer available on this listing page. > It will be useful to list aggregated files as well as the current set of > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6293) Investigate Java 7 compatibility for new YARN UI
Li Lu created YARN-6293:
---
Summary: Investigate Java 7 compatibility for new YARN UI
Key: YARN-6293
URL: https://issues.apache.org/jira/browse/YARN-6293
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Right now when trying the YARN new UI with Java 7, I can get the following
warning:
{code}
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (dist-enforce) @ hadoop-yarn-ui
---
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed
with message:
Detected JDK Version: 1.7.0-67 is not in the allowed range [1.8,).
{code}
While right now this warning does not cause any troubles for trunk integration,
when some users would like to package the new UI with some branch-2 based code,
the JDK requirement would block the effort. So the question here is, is there
any specific component in new UI codebase that prevent us using Java 7? I
remember it should be a JS based implementation, right?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5418) When partial log aggregation is enabled, display the list of aggregated files on the container log page
[ https://issues.apache.org/jira/browse/YARN-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5418: Attachment: Screen Shot 2017-03-06 at 1.38.04 PM.png > When partial log aggregation is enabled, display the list of aggregated files > on the container log page > --- > > Key: YARN-5418 > URL: https://issues.apache.org/jira/browse/YARN-5418 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: Screen Shot 2017-03-06 at 1.38.04 PM.png, > YARN-5418.1.patch > > > The container log pages lists all files. However, as soon as a file gets > aggregated - it's no longer available on this listing page. > It will be useful to list aggregated files as well as the current set of > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5418) When partial log aggregation is enabled, display the list of aggregated files on the container log page
[ https://issues.apache.org/jira/browse/YARN-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5418: Attachment: YARN-5418.1.patch > When partial log aggregation is enabled, display the list of aggregated files > on the container log page > --- > > Key: YARN-5418 > URL: https://issues.apache.org/jira/browse/YARN-5418 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: YARN-5418.1.patch > > > The container log pages lists all files. However, as soon as a file gets > aggregated - it's no longer available on this listing page. > It will be useful to list aggregated files as well as the current set of > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[
https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898145#comment-15898145
]
Hadoop QA commented on YARN-5948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
14s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
17s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 327 unchanged - 0 fixed = 328 total (was 327) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 55s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856339/YARN-5948-YARN-5734.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux f6966d454cc9 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5734 / 01ea2f3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Assigned] (YARN-5418) When partial log aggregation is enabled, display the list of aggregated files on the container log page
[ https://issues.apache.org/jira/browse/YARN-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-5418: --- Assignee: Xuan Gong > When partial log aggregation is enabled, display the list of aggregated files > on the container log page > --- > > Key: YARN-5418 > URL: https://issues.apache.org/jira/browse/YARN-5418 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Xuan Gong > > The container log pages lists all files. However, as soon as a file gets > aggregated - it's no longer available on this listing page. > It will be useful to list aggregated files as well as the current set of > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6292) YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is specified in fs.defaultFS
[ https://issues.apache.org/jira/browse/YARN-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898091#comment-15898091 ] Jason Lowe commented on YARN-6292: -- What version is involved here? Is this a duplicate of YARN-3269? > YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is > specified in fs.defaultFS > --- > > Key: YARN-6292 > URL: https://issues.apache.org/jira/browse/YARN-6292 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ang Zhang >Priority: Minor > > 2017-03-02 17:59:13,688 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: > Error in dispatcher thread > java.lang.IllegalArgumentException: Wrong FS: viewfs://ns-default/tmp/logs, > expected: hdfs://nameservice1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:321) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:445) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[ https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898083#comment-15898083 ] Jason Lowe commented on YARN-6195: -- Thanks for the patch, [~benson.qiu]! I'm confused about how labels are supposed to interact with the JMX metrics. There's only one metric for used capacity in a queue even if there are labels? If the used proportion of a capacity is updated, it updates the same metric as for other labels or no label? The metric will always reflect the last update given at the time the metric is examined, meaning there could be wildly different results from reading to reading even if things aren't moving that much on the cluster in reality, e.g.: one reading gets the "fizgig" node label's usage while another gets the default/no label usage. MetricsRegistry.java: "mutable long float" and "mutable float integer" should both be "mutable float" The tabs flagged by the whitespace check need to be removed. > Export UsedCapacity and AbsoluteUsedCapacity to JMX > --- > > Key: YARN-6195 > URL: https://issues.apache.org/jira/browse/YARN-6195 > Project: Hadoop YARN > Issue Type: Improvement > Components: metrics, yarn >Affects Versions: 3.0.0-alpha3 >Reporter: Benson Qiu >Assignee: Benson Qiu > Attachments: YARN-6195.001.patch > > > `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6292) YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is specified in fs.defaultFS
Ang Zhang created YARN-6292: --- Summary: YARN log aggregation doesn't support HDFS/ViewFs namespace other than what is specified in fs.defaultFS Key: YARN-6292 URL: https://issues.apache.org/jira/browse/YARN-6292 Project: Hadoop YARN Issue Type: Bug Reporter: Ang Zhang Priority: Minor 2017-03-02 17:59:13,688 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread java.lang.IllegalArgumentException: Wrong FS: viewfs://ns-default/tmp/logs, expected: hdfs://nameservice1 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194) at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:194) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:321) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:445) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
[
https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897949#comment-15897949
]
yunjiong zhao commented on YARN-6285:
-
[~sunilg], Totally agree.
This patch is for short time purpose to give cluster admin a choice to prevent
RM spend to much time on GC.
After we deployed this patch and set a limit to 50, in the last two days, our
cluster's GC was doing good.
The top 10 worst case are:
{quote}
0.4960477
0.4992665
0.5180593
0.5804366
0.5876860
0.5885162
0.5900650
0.6041406
0.6474685
0.8865442
{quote}
And total time spend on GC is around 1.5%, compared to before it's much better.
> Add option to set max limit on ResourceManager for
> ApplicationClientProtocol.getApplications
>
>
> Key: YARN-6285
> URL: https://issues.apache.org/jira/browse/YARN-6285
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: YARN-6285.001.patch, YARN-6285.002.patch,
> YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return
> lots of data, and generate lots of garbage on ResourceManager which caused
> long time GC.
> For example, on one of our RM, when called rest API " http:// address:port>/ws/v1/cluster/apps" it can return 150MB data which have 944
> applications.
> getApplications have limit parameter, but some user might not set it, and
> then the limit will be Long.MAX_VALUE.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[
https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897947#comment-15897947
]
Hadoop QA commented on YARN-5948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
57s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
49s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 19 new + 327 unchanged - 0 fixed = 346 total (was 327) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 6s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855714/YARN-5948-YARN-5734.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 2df0e1ad5fa4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[ https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897906#comment-15897906 ] Jonathan Hung commented on YARN-5948: - 004 fixes checkstyle. > Implement MutableConfigurationManager for handling storage into configuration > store > --- > > Key: YARN-5948 > URL: https://issues.apache.org/jira/browse/YARN-5948 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5948.001.patch, YARN-5948-YARN-5734.002.patch, > YARN-5948-YARN-5734.003.patch, YARN-5948-YARN-5734.004.patch > > > The MutableConfigurationManager will take REST calls with desired client > configuration changes and call YarnConfigurationStore methods to store these > changes in the backing store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[ https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5948: Attachment: YARN-5948-YARN-5734.004.patch > Implement MutableConfigurationManager for handling storage into configuration > store > --- > > Key: YARN-5948 > URL: https://issues.apache.org/jira/browse/YARN-5948 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5948.001.patch, YARN-5948-YARN-5734.002.patch, > YARN-5948-YARN-5734.003.patch, YARN-5948-YARN-5734.004.patch > > > The MutableConfigurationManager will take REST calls with desired client > configuration changes and call YarnConfigurationStore methods to store these > changes in the backing store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5179) Issue of CPU usage of containers
[ https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897728#comment-15897728 ] Manikandan R commented on YARN-5179: [~kasha]/[~konstantinos], any thoughts? > Issue of CPU usage of containers > > > Key: YARN-5179 > URL: https://issues.apache.org/jira/browse/YARN-5179 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 > Environment: Both on Windows and Linux >Reporter: Zhongkai Mi > > // Multiply by 1000 to avoid losing data when converting to int >int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 > * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN); > This formula will not get right CPU usage based vcore if vcores != physical > cores. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient
[
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897695#comment-15897695
]
Sunil G commented on YARN-6164:
---
[~benson.qiu]
I personally feel that having {{QueueConfigurations}} inside {{QueueInfo}} is a
general approach here. And looks fine for me. [~leftnoteasy], do you mind add
some thoughts here.
> Expose maximum-am-resource-percent in YarnClient
>
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch,
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the
> [Cluster Scheduler
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
> but not through
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by
> default), it would be nice to expose `maximum-am-resource-percent` in
> YarnClient as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient
[
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897685#comment-15897685
]
Benson Qiu commented on YARN-6164:
--
[~sunilg] [~bibinchundatt]
I'm planning to set aside some time this week to work on creating
`QueueConfigurations`.
I would like to apply this patch to 2.8.0 as well.
If I complete the feature discussed above (add a per-label map of
{{QueueConfigurations}}, what is the likelihood that my patch will be accepted?
> Expose maximum-am-resource-percent in YarnClient
>
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch,
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the
> [Cluster Scheduler
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
> but not through
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by
> default), it would be nice to expose `maximum-am-resource-percent` in
> YarnClient as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897673#comment-15897673
]
Sunil G commented on YARN-6207:
---
+1 committing shortly.
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch, YARN-6207.005.patch,
> YARN-6207.006.patch, YARN-6207.007.patch, YARN-6207.008.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6237) Move UID constant into TimelineReaderUtils
[ https://issues.apache.org/jira/browse/YARN-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897476#comment-15897476 ] Varun Saxena commented on YARN-6237: Thanks [~rohithsharma] ! Straightforward fix. Will commit it after committing YARN-6256. > Move UID constant into TimelineReaderUtils > -- > > Key: YARN-6237 > URL: https://issues.apache.org/jira/browse/YARN-6237 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: newbie > Attachments: YARN-6237-YARN-5355.0001.patch > > > UID constant is kept in TimelineReader Manager. This can be moved to > TimelineReaderUtils which can keep track of all reader constants. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6256) Add FROM_ID info key for timeline entities in reader response.
[ https://issues.apache.org/jira/browse/YARN-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897473#comment-15897473 ] Varun Saxena commented on YARN-6256: Thanks [~rohithsharma] for the latest patch. Changes LGTM. Will wait for a day before committing to give others a chance to review. > Add FROM_ID info key for timeline entities in reader response. > --- > > Key: YARN-6256 > URL: https://issues.apache.org/jira/browse/YARN-6256 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6256-YARN-5355.0001.patch, > YARN-6256-YARN-5355.0002.patch, YARN-6256-YARN-5355.0003.patch > > > It is continuation with YARN-6027 to add FROM_ID key in all other timeline > entity responses which includes > # Flow run entity response. > # Application entity response > # Generic timeline entity response - Here we need to retrospect on idprefix > filter which is now separately provided. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6237) Move UID constant into TimelineReaderUtils
[ https://issues.apache.org/jira/browse/YARN-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-6237: -- Assignee: Rohith Sharma K S > Move UID constant into TimelineReaderUtils > -- > > Key: YARN-6237 > URL: https://issues.apache.org/jira/browse/YARN-6237 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: newbie > Attachments: YARN-6237-YARN-5355.0001.patch > > > UID constant is kept in TimelineReader Manager. This can be moved to > TimelineReaderUtils which can keep track of all reader constants. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6291) [YARN-3368] Introduce query parameters (sort, filter, etc.) for tables to keep state on refresh/navigation
[ https://issues.apache.org/jira/browse/YARN-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-6291: Attachment: YARN-6291.001.patch Patch #1 adds these query parameters to both "Applications" and "Nodes" tables: - searchText - sortColumnId - sortOrder - pageNum - rowCount First I tried to create a base class for all the Controllers with em-tables to contain all these fields, but that approach didn't work (see the discussion here: https://github.com/emberjs/ember.js/issues/12102), so I had to include them separately to the affected controllers. > [YARN-3368] Introduce query parameters (sort, filter, etc.) for tables to > keep state on refresh/navigation > -- > > Key: YARN-6291 > URL: https://issues.apache.org/jira/browse/YARN-6291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Novák >Assignee: Gergely Novák > Attachments: YARN-6291.001.patch > > > On Applications / Nodes pages the user can define filters, sort the list on > selected columns, etc. But all these settings are lost on a refresh, or > navigation, which is very inconvenient. > For example a possible use case might be watching the Applications table > while running a lot of YARN applications and continuously pressing Refresh to > see their progress. In this case, the user have to set any filter / sorting > manually after every single refresh. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[
https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897404#comment-15897404
]
Naganarasimha G R commented on YARN-6148:
-
bq. When default label for an application changes which will typically happen
when application moves across queues, we will consider only the blacklisting
failures on target queue's default label to ignore blacklisting.
The reason we need to consider {{"target queue's default label"}} for the
*threshold* is because we have *threshold* to ensure that we do not hang the
app by blacklisting all the nodes for this app, so once we have moved to the
new queue then if the container fails then the new request gets submitted to
the target queue's default label . so we need to consider the threshold for the
target queue's label only and we can ignore blacklist for the previous label,
Thoughts?
> NM node count reported to AM in Allocate Response should consider requested
> node label partitions.
> --
>
> Key: YARN-6148
> URL: https://issues.apache.org/jira/browse/YARN-6148
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-6148.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6276) Now container kill mechanism may lead process leak
[ https://issues.apache.org/jira/browse/YARN-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897373#comment-15897373 ] Jason Lowe commented on YARN-6276: -- Processes escaping from the session is a known problem. If that's the "leak" being discussed here then this seems like a duplicate of YARN-2904. > Now container kill mechanism may lead process leak > -- > > Key: YARN-6276 > URL: https://issues.apache.org/jira/browse/YARN-6276 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Feng Yuan >Assignee: Feng Yuan > > When kill bash process, YarnChild may didn`t response because fullgc, -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6291) [YARN-3368] Introduce query parameters (sort, filter, etc.) for tables to keep state on refresh/navigation
Gergely Novák created YARN-6291: --- Summary: [YARN-3368] Introduce query parameters (sort, filter, etc.) for tables to keep state on refresh/navigation Key: YARN-6291 URL: https://issues.apache.org/jira/browse/YARN-6291 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Novák Assignee: Gergely Novák On Applications / Nodes pages the user can define filters, sort the list on selected columns, etc. But all these settings are lost on a refresh, or navigation, which is very inconvenient. For example a possible use case might be watching the Applications table while running a lot of YARN applications and continuously pressing Refresh to see their progress. In this case, the user have to set any filter / sorting manually after every single refresh. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Description: When I ran experiments with both Spark and MapReduce wordcount with yarn on the file, I noticed that the job did not get data locality every time. It was seemingly random in the placement of the tasks, even though there is no other job running on the cluster. I expected the task placement to always be on the single machine which is holding the data block, but that did not happen. I run the experiments with a 7 node cluster with 2x replication(1 master, 6 data nodes/node managers) , the experiment details are in the patch so you can recreate the result. In the experiments, I run Spark/MapReduce wordcount with yarn for 10 times in a single block and the results show that only 30% of tasks can satisfy data locality, it seems like random in the placement of tasks. Next,I will run two more experiments(7 node cluster with 2x replication with 2 blocks and 4 blocks) to verify the results and plan to do some optimization work (optimize the schedule policy) to improve data locality was: When I ran experiments with both Spark and MapReduce wordcount with yarn on the file, I noticed that the job did not get data locality every time. It was seemingly random in the placement of the tasks, even though there is no other job running on the cluster. I expected the task placement to always be on the single machine which is holding the data block, but that did not happen. I run the experiments with a 7 node cluster with 2x replication(1 master, 6 data nodes/node managers) , the experiment details are in the patch so you can recreate the result. In the experiments, I run Spark/MapReduce wordcount with yarn for 10 times in a single block and the results show that only 30% of tasks can satisfy data locality, it seems like random in the placement of tasks. > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. > Next,I will run two more experiments(7 node cluster with 2x replication > with 2 blocks and 4 blocks) to verify the results and plan to do some > optimization work (optimize the schedule policy) to improve data locality -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in Queue pages
[ https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6182: --- Description: This patch fixes following issues: In Queues page: # Queue Capacities: Absolute Max Capacity should be aligned better. # Queue Information: State is coming empty # The queue tree graph is taking too much space. We should reduce both the vertical and horizontal spacing. # Queues tab becomes inactive while hovering on the queue. In application list page and per application page: # Change left nav label to ''Applications' in application list page. # Convert labels 'Master Node' and 'Master Node Expression' to headings in single app page. was: This patch fixes following issues: In Queues page: # Queue Capacities: Absolute Max Capacity should be aligned better. # Queue Information: State is coming empty # The queue tree graph is taking too much space. We should reduce both the vertical and horizontal spacing. # Queues tab becomes inactive while hovering on the queue. In application list page and per application page: # Change left nav label to ''Applications' in application list page # Convert labels 'Master Node' and 'Master Node Expression' to headings in single app page > [YARN-3368] Fix alignment issues and missing information in Queue pages > --- > > Key: YARN-6182 > URL: https://issues.apache.org/jira/browse/YARN-6182 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6182.001.patch, YARN-6182.002.patch, > YARN-6182.003.patch > > > This patch fixes following issues: > In Queues page: > # Queue Capacities: Absolute Max Capacity should be aligned better. > # Queue Information: State is coming empty > # The queue tree graph is taking too much space. We should reduce both the > vertical and horizontal spacing. > # Queues tab becomes inactive while hovering on the queue. > In application list page and per application page: > # Change left nav label to ''Applications' in application list page. > # Convert labels 'Master Node' and 'Master Node Expression' to headings in > single app page. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in Queue pages
[ https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6182: --- Description: This patch fixes following issues: In Queues page: # Queue Capacities: Absolute Max Capacity should be aligned better. # Queue Information: State is coming empty # The queue tree graph is taking too much space. We should reduce both the vertical and horizontal spacing. # Queues tab becomes inactive while hovering on the queue. In application list page and per application page: # Change left nav label to ''Applications' in application list page # Convert labels 'Master Node' and 'Master Node Expression' to headings in single app page was: This patch fixes following issues: In Queues page: # Queue Capacities: Absolute Max Capacity should be aligned better. # Queue Information: State is coming empty # The queue tree graph is taking too much space. We should reduce both the vertical and horizontal spacing. # Queues tab becomes inactive while hovering on the queue. In application list page and per application page: # Change left nav label to ''Applications' # Convert labels 'Master Node' and 'Master Node Expression' to headings > [YARN-3368] Fix alignment issues and missing information in Queue pages > --- > > Key: YARN-6182 > URL: https://issues.apache.org/jira/browse/YARN-6182 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6182.001.patch, YARN-6182.002.patch, > YARN-6182.003.patch > > > This patch fixes following issues: > In Queues page: > # Queue Capacities: Absolute Max Capacity should be aligned better. > # Queue Information: State is coming empty > # The queue tree graph is taking too much space. We should reduce both the > vertical and horizontal spacing. > # Queues tab becomes inactive while hovering on the queue. > In application list page and per application page: > # Change left nav label to ''Applications' in application list page > # Convert labels 'Master Node' and 'Master Node Expression' to headings in > single app page -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Attachment: (was: YARN-6289.01.docx) > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Attachment: YARN-6289.01.docx > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6289) yarn got little data locality
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huangkaixuan updated YARN-6289: --- Description: When I ran experiments with both Spark and MapReduce wordcount with yarn on the file, I noticed that the job did not get data locality every time. It was seemingly random in the placement of the tasks, even though there is no other job running on the cluster. I expected the task placement to always be on the single machine which is holding the data block, but that did not happen. I run the experiments with a 7 node cluster with 2x replication(1 master, 6 data nodes/node managers) , the experiment details are in the patch so you can recreate the result. In the experiments, I run Spark/MapReduce wordcount with yarn for 10 times in a single block and the results show that only 30% of tasks can satisfy data locality, it seems like random in the placement of tasks. was:When I ran this experiment with both Spark and MapReduce wordcount with yarn on the file, I noticed that the job did not get data locality every time. It was seemingly random in the placement of the tasks, even though there is no other job running on the cluster. I expected the task placement to always be on the single machine which is holding the data block, but that did not happen. > yarn got little data locality > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan >Priority: Minor > Attachments: YARN-6289.01.docx > > > When I ran experiments with both Spark and MapReduce wordcount with yarn > on the file, I noticed that the job did not get data locality every time. It > was seemingly random in the placement of the tasks, even though there is no > other job running on the cluster. I expected the task placement to always be > on the single machine which is holding the data block, but that did not > happen. > I run the experiments with a 7 node cluster with 2x replication(1 > master, 6 data nodes/node managers) , the experiment details are in the patch > so you can recreate the result. > In the experiments, I run Spark/MapReduce wordcount with yarn for 10 > times in a single block and the results show that only 30% of tasks can > satisfy data locality, it seems like random in the placement of tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897155#comment-15897155 ] Varun Saxena commented on YARN-6148: Me and Naga just had an offline discussion with [~sunilg]. Will update the summary. # We will report a default applicable label for application (if none is specified) which can either be the label in Application Submission context or queue default label, in both Register AM response and in Allocate Response. Latter because application can move across queues. # When the label for a node changes, we will also report this node as part of updated nodes report in Allocate Response. # We would also update Container Info in allocated containers list to include information such as label info including whether its a non exclusive label and the type of container i.e. guaranteed or opportunistic. # At a high level, MR AM will maintain a list of node to label information so as to ascertain which label should the node blacklist be accounted for. Implementation wise, this can probably be done by updating the assigned requests. # When default label for an application changes which will typically happen when application moves across queues, we will consider only the blacklisting failures on target queue's default label to ignore blacklisting. If application moves back and forth across queues, say y => x => y, we can reconstruct the applicable node count for blacklisting as we have both the blacklisted nodes and their mapping to a label. Thoughts? > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897123#comment-15897123 ] Varun Saxena commented on YARN-6209: My thoughts. Probably you meant the same in comment above. I think what we should report is default applicable label for the application. This can be either from ASC or default label of the queue (or whatever applicable default label is chosen due to future modifications). And this needs to be reported on both register AM response and on allocate response as the application can move across queues. Probably we can move this discussion over to YARN-6148 so that we can discuss in detail as this is more related to AM blacklisting. I will summarise the offline discussion we just had with you there as well > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6290) Fair scheduler page broken
Léopold Boudard created YARN-6290: - Summary: Fair scheduler page broken Key: YARN-6290 URL: https://issues.apache.org/jira/browse/YARN-6290 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.3 Reporter: Léopold Boudard Hello, I have an issue very similar to this one https://issues.apache.org/jira/browse/YARN-3478 not being able to access to scheduler interface in yarn webapp. Except that traceback is bit different (NullPointerException and could be be caused by stale queue configuration. I suspect QueueCapacitiesInfo.java initializing info object with null value for some reason. Below traceback: ``` 2017-03-06 10:20:00,945 ERROR webapp.Dispatcher (Dispatcher.java:service(162)) - error handling URI: /cluster/scheduler java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:614) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:573) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:95) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[
https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897069#comment-15897069
]
Sunil G commented on YARN-6209:
---
Ideally its better if we get an update from RM regarding the real AM' label.
Let it be coming from ASC or from default label of queue etc. Given that
information in {{RegisterApplicationMasterResponse}}, MR will have idea about
AM containers label. So rather passing as queue;s default label, it could be
passed as AM's label. Now other containers(map/reduce) label are available from
container info rt ?
I agree there are some more grey areas here
# change label of a node at runtime which will impact running containers as
well.
# container launched on a non-exclusive label
If per-label-node-count for requested partition is buffered at RM level, could
we NOT make it dirty when node label s changed in a node. Any way such
containers are prone for preemption when a real ask is available for new label.
As mentioned, I think you are not planning to add this count, correct?
> AM should get notified when application moved to new queue
> --
>
> Key: YARN-6209
> URL: https://issues.apache.org/jira/browse/YARN-6209
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Naganarasimha G R
>
> As Vinod pointed out in
> [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356],
> when application is moved to different queue, AM should get notified about
> the new queue.
> YARN-1623 adds up queue information in RegisterApplicationMasterResponse.
> The same functionality could be mirrored in AllocateResponse also when app is
> moved to new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896994#comment-15896994 ] Varun Saxena edited comment on YARN-6209 at 3/6/17 10:06 AM: - Basically in SchedulerUtils while normalizing ask after receiving it from AM, we will change the label expression in ask to default queue label expression or the one in ASC if none is specified. RM would store asks alongwith this label. Now when RM reports to AM the label to node count info in YARN-6148, it needs to correlate if this is the default queue label expression or the one in ASC or not to ascertain how to consider it while ignoring blacklisting. Thoughts? was (Author: varun_saxena): Basically in SchedulerUtils while normalizing ask after receiving it from AM, we will change the label expression in ask to default queue label expression or the one in ASC if none is specified. RM would store asks alongwith this label. Now when RM reports to AM the label to node count info in YARN-6148, it needs to correlate if this is the default queue label expression or not to ascertain how to consider it while ignoring blacklisting. Thoughts? > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896978#comment-15896978 ] Varun Saxena edited comment on YARN-6148 at 3/6/17 10:00 AM: - Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios which need to be handled. So following is the plan regarding what will be done. # We will send a label to node count map containing label to active NM count mapping for requested labels. # If labels are disabled, we will report cluster count as earlier and not send anything in label to active NM count mapping. AM can consider only the cluster node count reported earlier in such cases. This will also help us in handling the case for rolling downgrades. # Non Exclusive labels will not be reported in the label to node count unless explicitly requested by AM. # Mapreduce AM would consider both map and reduce label expression separately, if they are different, while deciding on ignoring AM blacklisting. We will check label to node count mapping to determine the applicable node counts for both map and reduce. # Queue can have a default label too. This needs to be reported to AM as well. Also needs to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and AM currently does not know anything about default label of queue. # Also we can report AM node label from Application Submission context in Register AM response. This is because RM can consider label in ASC if none is specified in AM ask. # Container info sent while reporting allocated containers would also contain label on which container was allocated on. Based on this info we will determine if the node assigned is a default label of queue, map/reduce partition or non exclusive label. We will not count non exclusive label towards ignoring blacklisting. Thoughts? was (Author: varun_saxena): Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios which need to be handled. So following is the plan regarding what will be done. # We will send a label to node count map containing label to active NM count mapping for requested labels. # If labels are disabled, we will report cluster count as earlier and not send anything in label to active NM count mapping. AM can consider only the cluster node count reported earlier in such cases. This will also help us in handling the case for rolling downgrades. # Non Exclusive labels will not be reported in the label to node count unless explicitly requested by AM. # Mapreduce AM would consider both map and reduce label expression separately, if they are different, while deciding on ignoring AM blacklisting. We will check label to node count mapping to determine the applicable node counts for both map and reduce. # Queue can have a default label too. This needs to be reported to AM as well. Also needs to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and AM currently does not know anything about default label of queue. # Container info sent while reporting allocated containers would also contain label on which container was allocated on. Based on this info we will determine if the node assigned is a default label of queue, map/reduce partition or non exclusive label. We will not count non exclusive label towards ignoring blacklisting. Thoughts? > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896994#comment-15896994 ] Varun Saxena commented on YARN-6209: Basically in SchedulerUtils while normalizing ask after receiving it from AM, we will change the label expression in ask to default queue label expression or the one in ASC if none is specified. RM would store asks alongwith this label. Now when RM reports to AM the label to node count info in YARN-6148, it needs to correlate if this is the default queue label expression or not to ascertain how to consider it while ignoring blacklisting. Thoughts? > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
[ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896984#comment-15896984 ] Sunil G commented on YARN-6285: --- Thanks [~zhaoyunjiong]. I understood the points made above. However I still feel that setting a max-downloadable limit will not help the problem mentioned here. Few reasons for this # apps are stored in RM in a map. Hence given any limit, server wont be able to assure any specific order for apps. In a heterogeneous cluster, you may feel like some pending/running apps are not in the list. # ideal solution is to provide paginated support to get apps from RM. This is kind of blocked due to point 1). As you see, completed apps which can be stored in RM is reduced to 1000. Servers like ATS where apps are stored in timeline, is having pagination support. Coming to the discussion here, i think its better to reduce the size of ApplicationReport. and also time computation to generate same. If we could hide ResourceRequests and LogAggreationStatus, size and time of ApplicationReport could be made better, Unfortunately we cannot make this as default behavior as these options are made default in hadoop-2.7. Hence we can add this params to hide/avoid computation to get ResourceRequests and LogAggreationStatus, and use it in above use case.. On same note, we can still push empty data to ResourceRequests and LogAggreationStatus, so that client code wont break. But they will get empty data. If some users are making use of this, it ll be broken from next release. Looping [~jianhe] here to add some more perspective. > Add option to set max limit on ResourceManager for > ApplicationClientProtocol.getApplications > > > Key: YARN-6285 > URL: https://issues.apache.org/jira/browse/YARN-6285 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: YARN-6285.001.patch, YARN-6285.002.patch, > YARN-6285.003.patch > > > When users called ApplicationClientProtocol.getApplications, it will return > lots of data, and generate lots of garbage on ResourceManager which caused > long time GC. > For example, on one of our RM, when called rest API " http:// address:port>/ws/v1/cluster/apps" it can return 150MB data which have 944 > applications. > getApplications have limit parameter, but some user might not set it, and > then the limit will be Long.MAX_VALUE. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896978#comment-15896978 ] Varun Saxena commented on YARN-6148: Offline. we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios which need to be handled. So following is the plan regarding what will be done. # We will send a label to node count map containing label to active NM count mapping for requested labels. # If labels are disabled, we will report cluster count as earlier and not send anything in label to active NM count mapping. AM can consider only the cluster node count reported earlier in such cases. This will also help us in handling the case for rolling downgrades. # Non Exclusive labels will not be reported in the label to node count unless explicitly requested by AM. # Mapreduce AM would consider both map and reduce label expression separately, if they are different, while deciding on ignoring AM blacklisting. We will check label to node count mapping to determine the applicable node counts for both map and reduce. # Queue can have a default label too. This needs to be reported to AM as well. Also needs to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and AM currently does not know anything about default label of queue. # Container info sent while reporting allocated containers would also contain label on which container was allocated on. Based on this info we will determine if the node assigned is a default label of queue, map/reduce partition or non exclusive label. We will not count non exclusive label towards ignoring blacklisting. Thoughts? > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896978#comment-15896978 ] Varun Saxena edited comment on YARN-6148 at 3/6/17 9:39 AM: Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios which need to be handled. So following is the plan regarding what will be done. # We will send a label to node count map containing label to active NM count mapping for requested labels. # If labels are disabled, we will report cluster count as earlier and not send anything in label to active NM count mapping. AM can consider only the cluster node count reported earlier in such cases. This will also help us in handling the case for rolling downgrades. # Non Exclusive labels will not be reported in the label to node count unless explicitly requested by AM. # Mapreduce AM would consider both map and reduce label expression separately, if they are different, while deciding on ignoring AM blacklisting. We will check label to node count mapping to determine the applicable node counts for both map and reduce. # Queue can have a default label too. This needs to be reported to AM as well. Also needs to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and AM currently does not know anything about default label of queue. # Container info sent while reporting allocated containers would also contain label on which container was allocated on. Based on this info we will determine if the node assigned is a default label of queue, map/reduce partition or non exclusive label. We will not count non exclusive label towards ignoring blacklisting. Thoughts? was (Author: varun_saxena): Offline. we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios which need to be handled. So following is the plan regarding what will be done. # We will send a label to node count map containing label to active NM count mapping for requested labels. # If labels are disabled, we will report cluster count as earlier and not send anything in label to active NM count mapping. AM can consider only the cluster node count reported earlier in such cases. This will also help us in handling the case for rolling downgrades. # Non Exclusive labels will not be reported in the label to node count unless explicitly requested by AM. # Mapreduce AM would consider both map and reduce label expression separately, if they are different, while deciding on ignoring AM blacklisting. We will check label to node count mapping to determine the applicable node counts for both map and reduce. # Queue can have a default label too. This needs to be reported to AM as well. Also needs to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and AM currently does not know anything about default label of queue. # Container info sent while reporting allocated containers would also contain label on which container was allocated on. Based on this info we will determine if the node assigned is a default label of queue, map/reduce partition or non exclusive label. We will not count non exclusive label towards ignoring blacklisting. Thoughts? > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: YARN-6148 > URL: https://issues.apache.org/jira/browse/YARN-6148 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6148.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896973#comment-15896973 ] Sunil G edited comment on YARN-6209 at 3/6/17 9:37 AM: --- Sorry to ask this question. I didnt really get this :( bq.default label of the queue Why this is needed at MR side. MR just need to know its AM label expression and map/reducer label expression rt. was (Author: sunilg): Sorry to ask this question. bq.default label of the queue Why this is needed at MR side. MR just need to know its AM label expression and map/reducer label expression rt. > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896973#comment-15896973 ] Sunil G commented on YARN-6209: --- Sorry to ask this question. bq.default label of the queue Why this is needed at MR side. MR just need to know its AM label expression and map/reducer label expression rt. > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896967#comment-15896967 ] Varun Saxena commented on YARN-6209: Probably you mean that we report default label of the queue to which job has been submitted to in this JIRA. And also report it when application moves to a new queue. +1 to doing that in this JIRA. But I think MAPREDUCE related changes with regards to blacklisting for it can be done separately. Thoughts? > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896947#comment-15896947 ] Varun Saxena edited comment on YARN-6209 at 3/6/17 9:20 AM: bq. Along with queue we have to send label too when default queue label is specified. AM side blacklisting based on partition same will be required. Yes. But this JIRA is slightly different in intention. We will do that in YARN-6148 or some other JIRA which will have to be raised. Right? Basically label information will have to passed for each Container info which is sent upon allocation. was (Author: varun_saxena): bq. Along with queue we have to send label too when default queue label is specified. AM side blacklisting based on partition same will be required. Yes. But this JIRA is slightly different in intention. We will do that in YARN-6148 or some other JIRA which will have to be raised. Right? > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896947#comment-15896947 ] Varun Saxena commented on YARN-6209: bq. Along with queue we have to send label too when default queue label is specified. AM side blacklisting based on partition same will be required. Yes. But this JIRA is slightly different in intention. We will do that in YARN-6148 or some other JIRA which will have to be raised. Right? > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6209) AM should get notified when application moved to new queue
[ https://issues.apache.org/jira/browse/YARN-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R reassigned YARN-6209: --- Assignee: Naganarasimha G R > AM should get notified when application moved to new queue > -- > > Key: YARN-6209 > URL: https://issues.apache.org/jira/browse/YARN-6209 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Naganarasimha G R > > As Vinod pointed out in > [comment|https://issues.apache.org/jira/browse/YARN-5068?focusedCommentId=15867356=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15867356], > when application is moved to different queue, AM should get notified about > the new queue. > YARN-1623 adds up queue information in RegisterApplicationMasterResponse. > The same functionality could be mirrored in AllocateResponse also when app is > moved to new queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
