[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043981#comment-16043981 ] Rohith Sharma K S commented on YARN-6638: - I think it is NOT required since it is mainly used by Web UI since Reader server does not host any UI by itself. +1 lgtm for 03 patch. > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch, YARN-6638-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6608) Backport all SLS improvements from trunk to branch-2
[ https://issues.apache.org/jira/browse/YARN-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043967#comment-16043967 ] Carlo Curino edited comment on YARN-6608 at 6/9/17 5:54 AM: [~yufeigu], that is a good idea. In the latest patch I: # fixed Java 1.7 compilation issues. # rebased to branch-2 and included all trunk patches on SLS. # created the following list of changes we backport (by comparing trunk and branch-2 edits that hit SLS, and then see if they were completely or partially backported). Here is what I got: b65100c YARN-6547. fcbdecc YARN-6111. 40e6a85 YARN-6560. 101852c YARN-6535. 5cb6e3e YARN-6473. 2ba9903 HADOOP-14400. (partial) 3082552 YARN-6522. 475f933 YARN-6423. de69d6e YARN-6363. 7e075a50 YARN-6365. 6a5516c YARN-5654. cc938e9 YARN-6326. 1598fd3 YARN-6275. b32ffa2 YARN-4779. 5ed63e3 YARN-4899. 3bcfe3a YARN-5706. 209e805 HADOOP-13506. (partial) de01327 MAPREDUCE-6796. (partial) 5877f20 HADOOP-10075. (partial) 2cc841f HADOOP-13678 (partial) 58ed4fa HADOOP-13341. (partial) 96fa0f8 HADOOP-12709 (partial) 3c5c57a HADOOP-13142. (partial) 63e5412 HDFS-9427. (partial) 9c32f87 HDFS-10266. (partial) 0a74610 HADOOP-11393. (partial) 7381550 HADOOP-12857. (partial) 73b9c7b HADOOP-10787. (partial) 18727c6 HADOOP-12133 (partial) c5afe44 HDFS-8349. (partial) 618ba70 MAPREDUCE-6057. (partial) dce8b9c HADOOP-11637. (partial) b6fc1f3 HADOOP-11593. (partial) f990e9d HADOOP-11346. d989ac0 MAPREDUCE-5972. (partial) 22cbcd1 HADOOP-10717. (partial) bae5837 HADOOP-10717. (partial) Note that many of the older partial patches were .sh / pom.xml file edits. was (Author: curino): [~yufeigu], that is a good idea. In the latest patch I: # fixed Java 1.7 compilation issues. # rebased to branch-2 and included all trunk patches on SLS. # created the following list of changes we backport (by comparing trunk and branch-2 edits that hit SLS, and then see if they were completely or partially backported). Here is what I got: b65100c YARN-6547. fcbdecc YARN-6111. 40e6a85 YARN-6560. 101852c YARN-6535. 5cb6e3e YARN-6473. 2ba9903 HADOOP-14400. (partial) 3082552 YARN-6522. 475f933 YARN-6423. de69d6e YARN-6363. 7e075a50 YARN-6365. 6a5516c YARN-5654. cc938e9 YARN-6326. 1598fd3 YARN-6275. b32ffa2 YARN-4779. 5ed63e3 YARN-4899. 3bcfe3a YARN-5706. 209e805 HADOOP-13506. (partial) de01327 MAPREDUCE-6796. (partial) 5877f20 HADOOP-10075. (partial) 2cc841f HADOOP-13678 (partial) 58ed4fa HADOOP-13341. (partial) 96fa0f8 HADOOP-12709 (partial) 3c5c57a HADOOP-13142. (partial) 63e5412 HDFS-9427. (partial) 9c32f87 HDFS-10266. (partial) 0a74610 HADOOP-11393. (partial) 7381550 HADOOP-12857. (partial) 73b9c7b HADOOP-10787. (partial) 18727c6 HADOOP-12133 (partial) c5afe44 HDFS-8349. (partial) 618ba70 MAPREDUCE-6057. (partial) dce8b9c HADOOP-11637. (partial) b6fc1f3 HADOOP-11593. (partial) f990e9d HADOOP-11346. d989ac0 MAPREDUCE-5972. (partial) 22cbcd1 HADOOP-10717. (partial) bae5837 HADOOP-10717. (partial) 28db28d YARN-1311. (partial) Note that many of the older partial patches were .sh / pom.xml file edits. > Backport all SLS improvements from trunk to branch-2 > > > Key: YARN-6608 > URL: https://issues.apache.org/jira/browse/YARN-6608 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6608-branch-2.v0.patch, > YARN-6608-branch-2.v1.patch, YARN-6608-branch-2.v2.patch, > YARN-6608-branch-2.v3.patch > > > The SLS has received lots of attention in trunk, but only some of it made it > back to branch-2. This patch is a "raw" fork-lift of the trunk development > from hadoop-tools/hadoop-sls. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6608) Backport all SLS improvements from trunk to branch-2
[ https://issues.apache.org/jira/browse/YARN-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043967#comment-16043967 ] Carlo Curino edited comment on YARN-6608 at 6/9/17 5:53 AM: [~yufeigu], that is a good idea. In the latest patch I: # fixed Java 1.7 compilation issues. # rebased to branch-2 and included all trunk patches on SLS. # created the following list of changes we backport (by comparing trunk and branch-2 edits that hit SLS, and then see if they were completely or partially backported). Here is what I got: b65100c YARN-6547. fcbdecc YARN-6111. 40e6a85 YARN-6560. 101852c YARN-6535. 5cb6e3e YARN-6473. 2ba9903 HADOOP-14400. (partial) 3082552 YARN-6522. 475f933 YARN-6423. de69d6e YARN-6363. 7e075a50 YARN-6365. 6a5516c YARN-5654. cc938e9 YARN-6326. 1598fd3 YARN-6275. b32ffa2 YARN-4779. 5ed63e3 YARN-4899. 3bcfe3a YARN-5706. 209e805 HADOOP-13506. (partial) de01327 MAPREDUCE-6796. (partial) 5877f20 HADOOP-10075. (partial) 2cc841f HADOOP-13678 (partial) 58ed4fa HADOOP-13341. (partial) 96fa0f8 HADOOP-12709 (partial) 3c5c57a HADOOP-13142. (partial) 63e5412 HDFS-9427. (partial) 9c32f87 HDFS-10266. (partial) 0a74610 HADOOP-11393. (partial) 7381550 HADOOP-12857. (partial) 73b9c7b HADOOP-10787. (partial) 18727c6 HADOOP-12133 (partial) c5afe44 HDFS-8349. (partial) 618ba70 MAPREDUCE-6057. (partial) dce8b9c HADOOP-11637. (partial) b6fc1f3 HADOOP-11593. (partial) f990e9d HADOOP-11346. d989ac0 MAPREDUCE-5972. (partial) 22cbcd1 HADOOP-10717. (partial) bae5837 HADOOP-10717. (partial) 28db28d YARN-1311. (partial) Note that many of the older partial patches were .sh / pom.xml file edits. was (Author: curino): [~yufeigu], that is a good idea. In the latest patch I: # fixed Java 1.7 compilation issues. # rebased to branch-2 and included all trunk patches on SLS. # created the following list of changes we backport (by comparing trunk and branch-2 edits that hit SLS, and then see if they were completely or partially backported). Here is what I got: b65100c YARN-6547. fcbdecc YARN-6111. 40e6a85 YARN-6560. 101852c YARN-6535. 5cb6e3e YARN-6473. 2ba9903 HADOOP-14400. (partial) 3082552 YARN-6522. 475f933 YARN-6423. de69d6e YARN-6363. 7e075a50 YARN-6365. 6a5516c YARN-5654. cc938e9 YARN-6326. 1598fd3 YARN-6275. b32ffa2 YARN-4779. 5ed63e3 YARN-4899. 3bcfe3a YARN-5706. 209e805 HADOOP-13506. (partial) de01327 MAPREDUCE-6796. (partial) 5877f20 HADOOP-10075. (partial) 2cc841f HADOOP-13678 (partial) 58ed4fa HADOOP-13341. (partial) 96fa0f8 HADOOP-12709 (partial) 3c5c57a HADOOP-13142. (partial) 63e5412 HDFS-9427. (partial) 9c32f87 HDFS-10266. (partial) 0a74610 HADOOP-11393. (partial) 7381550 HADOOP-12857. (partial) 73b9c7b HADOOP-10787. (partial) 18727c6 HADOOP-12133 (partial) c5afe44 HDFS-8349. (partial) 618ba70 MAPREDUCE-6057. (partial) dce8b9c HADOOP-11637. (partial) b6fc1f3 HADOOP-11593. (partial) f990e9d HADOOP-11346. d989ac0 MAPREDUCE-5972. (partial) 4625792 YARN-1726. b0c2c85 YARN-1726. 22cbcd1 HADOOP-10717. (partial) bae5837 HADOOP-10717. (partial) 28db28d YARN-1311. (partial) Note that many of the older partial patches were .sh / pom.xml file edits. > Backport all SLS improvements from trunk to branch-2 > > > Key: YARN-6608 > URL: https://issues.apache.org/jira/browse/YARN-6608 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6608-branch-2.v0.patch, > YARN-6608-branch-2.v1.patch, YARN-6608-branch-2.v2.patch, > YARN-6608-branch-2.v3.patch > > > The SLS has received lots of attention in trunk, but only some of it made it > back to branch-2. This patch is a "raw" fork-lift of the trunk development > from hadoop-tools/hadoop-sls. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6608) Backport all SLS improvements from trunk to branch-2
[ https://issues.apache.org/jira/browse/YARN-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6608: --- Attachment: YARN-6608-branch-2.v3.patch > Backport all SLS improvements from trunk to branch-2 > > > Key: YARN-6608 > URL: https://issues.apache.org/jira/browse/YARN-6608 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6608-branch-2.v0.patch, > YARN-6608-branch-2.v1.patch, YARN-6608-branch-2.v2.patch, > YARN-6608-branch-2.v3.patch > > > The SLS has received lots of attention in trunk, but only some of it made it > back to branch-2. This patch is a "raw" fork-lift of the trunk development > from hadoop-tools/hadoop-sls. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6608) Backport all SLS improvements from trunk to branch-2
[ https://issues.apache.org/jira/browse/YARN-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043967#comment-16043967 ] Carlo Curino commented on YARN-6608: [~yufeigu], that is a good idea. In the latest patch I: # fixed Java 1.7 compilation issues. # rebased to branch-2 and included all trunk patches on SLS. # created the following list of changes we backport (by comparing trunk and branch-2 edits that hit SLS, and then see if they were completely or partially backported). Here is what I got: b65100c YARN-6547. fcbdecc YARN-6111. 40e6a85 YARN-6560. 101852c YARN-6535. 5cb6e3e YARN-6473. 2ba9903 HADOOP-14400. (partial) 3082552 YARN-6522. 475f933 YARN-6423. de69d6e YARN-6363. 7e075a50 YARN-6365. 6a5516c YARN-5654. cc938e9 YARN-6326. 1598fd3 YARN-6275. b32ffa2 YARN-4779. 5ed63e3 YARN-4899. 3bcfe3a YARN-5706. 209e805 HADOOP-13506. (partial) de01327 MAPREDUCE-6796. (partial) 5877f20 HADOOP-10075. (partial) 2cc841f HADOOP-13678 (partial) 58ed4fa HADOOP-13341. (partial) 96fa0f8 HADOOP-12709 (partial) 3c5c57a HADOOP-13142. (partial) 63e5412 HDFS-9427. (partial) 9c32f87 HDFS-10266. (partial) 0a74610 HADOOP-11393. (partial) 7381550 HADOOP-12857. (partial) 73b9c7b HADOOP-10787. (partial) 18727c6 HADOOP-12133 (partial) c5afe44 HDFS-8349. (partial) 618ba70 MAPREDUCE-6057. (partial) dce8b9c HADOOP-11637. (partial) b6fc1f3 HADOOP-11593. (partial) f990e9d HADOOP-11346. d989ac0 MAPREDUCE-5972. (partial) 4625792 YARN-1726. b0c2c85 YARN-1726. 22cbcd1 HADOOP-10717. (partial) bae5837 HADOOP-10717. (partial) 28db28d YARN-1311. (partial) Note that many of the older partial patches were .sh / pom.xml file edits. > Backport all SLS improvements from trunk to branch-2 > > > Key: YARN-6608 > URL: https://issues.apache.org/jira/browse/YARN-6608 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6608-branch-2.v0.patch, > YARN-6608-branch-2.v1.patch, YARN-6608-branch-2.v2.patch > > > The SLS has received lots of attention in trunk, but only some of it made it > back to branch-2. This patch is a "raw" fork-lift of the trunk development > from hadoop-tools/hadoop-sls. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6428) Queue AM limit is not honored in CS always
[ https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043885#comment-16043885 ] Sunil G commented on YARN-6428: --- I guess adding new apis may more complicate here. cc [~leftnoteasy] who was also part of offline discussion. > Queue AM limit is not honored in CS always > --- > > Key: YARN-6428 > URL: https://issues.apache.org/jira/browse/YARN-6428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6428.0001.patch, YARN-6428.0002.patch > > > Steps to reproduce > > Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each. > Configure 100% to default queue as capacity and max am limit as 10 % > Minimum scheduler memory and vcore as 512,1 > *Expected* > AM limit 4096 and 4 vores > *Actual* > AM limit 4096+512 and 4+1 vcore -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6681) Eliminate double-copy of child queues in canAssignToThisQueue
[
https://issues.apache.org/jira/browse/YARN-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043852#comment-16043852
]
Naganarasimha G R commented on YARN-6681:
-
hi [~daryn], mostly looks good to me, but is it not good enough that leaf
queue returns false and parent queue returns true ?
Is there any scenario where we need to do this check ?
> Eliminate double-copy of child queues in canAssignToThisQueue
> -
>
> Key: YARN-6681
> URL: https://issues.apache.org/jira/browse/YARN-6681
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: YARN-6681.2.branch-2.8.patch,
> YARN-6681.2.branch-2.patch, YARN-6681.2.trunk.patch,
> YARN-6681.branch-2.8.patch, YARN-6681.branch-2.patch, YARN-6681.trunk.patch
>
>
> 20% of the time in {{AbstractCSQueue#canAssignToThisQueue}} is spent
> performing two duplications a treemap of child queues into a list - once to
> test for null, second to see if it's empty. Eliminating the dups reduces the
> overhead to 2%.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6701) Add application type to capacity scheduler logs
[ https://issues.apache.org/jira/browse/YARN-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung resolved YARN-6701. - Resolution: Won't Do > Add application type to capacity scheduler logs > --- > > Key: YARN-6701 > URL: https://issues.apache.org/jira/browse/YARN-6701 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Jonathan Hung > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6701) Add application type to capacity scheduler logs
[ https://issues.apache.org/jira/browse/YARN-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043750#comment-16043750 ] Jonathan Hung commented on YARN-6701: - Sorry, never mind. I see YARN-3085 contains the appropriate functionality. > Add application type to capacity scheduler logs > --- > > Key: YARN-6701 > URL: https://issues.apache.org/jira/browse/YARN-6701 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Jonathan Hung > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6701) Add application type to capacity scheduler logs
[ https://issues.apache.org/jira/browse/YARN-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reassigned YARN-6701: --- Assignee: (was: Jonathan Hung) > Add application type to capacity scheduler logs > --- > > Key: YARN-6701 > URL: https://issues.apache.org/jira/browse/YARN-6701 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6680) Avoid locking overhead for NO_LABEL lookups
[ https://issues.apache.org/jira/browse/YARN-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043716#comment-16043716 ] Naganarasimha G R commented on YARN-6680: - [~daryn], I am kind of +0 on this, so i would wait for another +1 to go ahead on this . bq. Cloning the resource would be very bad. Resource object allocation overhead is already very high. Well i was referring to not on the read part but on the write/modification part. > Avoid locking overhead for NO_LABEL lookups > --- > > Key: YARN-6680 > URL: https://issues.apache.org/jira/browse/YARN-6680 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-6680.patch > > > Labels are managed via a hash that is protected with a read lock. The lock > acquire and release are each just as expensive as the hash lookup itself - > resulting in a 3X slowdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6682) Improve performance of AssignmentInformation datastructures
[
https://issues.apache.org/jira/browse/YARN-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043709#comment-16043709
]
Naganarasimha G R commented on YARN-6682:
-
Thanks for the patch [~daryn],
Looks like a straight forward patch and improvement. Will commit it shortly !
> Improve performance of AssignmentInformation datastructures
> ---
>
> Key: YARN-6682
> URL: https://issues.apache.org/jira/browse/YARN-6682
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: YARN-6682.branch-2.8.patch, YARN-6682.branch-2.patch,
> YARN-6682.trunk.patch
>
>
> {{AssignmentInformation}} is inefficient and creates lots of garbage that
> increase gc pressure. It creates 3 hashmaps that each contain only 2
> enum-based keys. This requires wrapper node objects, boxing/unboxing of ints,
> and more expensive lookups than simply using primitive arrays indexed by enum
> ordinal.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6428) Queue AM limit is not honored in CS always
[ https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043705#comment-16043705 ] Naganarasimha G R commented on YARN-6428: - Thanks [~bibinchundatt] Earlier too in our offline discussion i had preferred for opting modification of the existing methods rather than adding a new interface if it suffices for all the callers. [~sunilg], I think its fine to modify the existing interface which is done in 2nd patch rather than having a workaround, whats your opinion ? If no issues i plan to go ahead and commit the 2nd patch. > Queue AM limit is not honored in CS always > --- > > Key: YARN-6428 > URL: https://issues.apache.org/jira/browse/YARN-6428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6428.0001.patch, YARN-6428.0002.patch > > > Steps to reproduce > > Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each. > Configure 100% to default queue as capacity and max am limit as 10 % > Minimum scheduler memory and vcore as 512,1 > *Expected* > AM limit 4096 and 4 vores > *Actual* > AM limit 4096+512 and 4+1 vcore -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations
[
https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043683#comment-16043683
]
Hadoop QA commented on YARN-5953:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
53s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
25s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
14s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5734 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
26s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 4s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 40m
58s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 54s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}183m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-5953 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872141/YARN-5953-YARN-5734.003.patch
|
| Optional Tests | asflicense shellcheck shelldocs mvnsite unit compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 3a0da6cd9fb4 3.13.0-106-generic
[jira] [Resolved] (YARN-650) User guide for preemption
[ https://issues.apache.org/jira/browse/YARN-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved YARN-650. Resolution: Won't Fix Documentation was added in YARN-4492 > User guide for preemption > - > > Key: YARN-650 > URL: https://issues.apache.org/jira/browse/YARN-650 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Chris Douglas >Priority: Minor > Attachments: Y650-0.patch > > > YARN-45 added a protocol for the RM to ask back resources. The docs on > writing YARN applications should include a section on how to interpret this > message. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6127) Add support for work preserving NM restart when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043635#comment-16043635
]
Botong Huang commented on YARN-6127:
Unit test failures are not related:
TestMiniYarnClusterNodeUtilization open in
https://issues.apache.org/jira/browse/YARN-4453
TestContainerManagerSecurity being discussed in
https://issues.apache.org/jira/browse/YARN-5655
> Add support for work preserving NM restart when AMRMProxy is enabled
>
>
> Key: YARN-6127
> URL: https://issues.apache.org/jira/browse/YARN-6127
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, nodemanager
>Reporter: Subru Krishnan
>Assignee: Botong Huang
> Attachments: YARN-6127.v1.patch, YARN-6127.v2.patch
>
>
> YARN-1336 added the ability to restart NM without loosing any running
> containers. In a Federated YARN environment, there's additional state in the
> {{AMRMProxy}} to allow for spanning across multiple sub-clusters, so we need
> to enhance {{AMRMProxy}} to support work-preserving restart.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5655) TestContainerManagerSecurity#testNMTokens is asserting
[
https://issues.apache.org/jira/browse/YARN-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043620#comment-16043620
]
Botong Huang edited comment on YARN-5655 at 6/8/17 11:29 PM:
-
Hi [~rkanter] and [~templedf], I am also getting NPE (most of the time) here.
Using latest trunk: $ mvn -Dtest=TestContainerManagerSecurity test
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 36.821 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 20.467 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 16.074 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
I've also hit timeout on it with yetus two times in a row:
https://builds.apache.org/job/PreCommit-YARN-Build/16159/testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testContainerManager_0_/
https://builds.apache.org/job/PreCommit-YARN-Build/16149/testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testContainerManager_1_/
Can you please take another look? Thanks!
was (Author: botong):
Hi [~rkanter] and [~templedf], I am also getting lots of NPE here. Using latest
trunk: $ mvn -Dtest=TestContainerManagerSecurity test
Can you please take another look? Thanks!
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 36.821 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 20.467 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 16.074 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
> TestContainerManagerSecurity#testNMTokens is asserting
> --
>
> Key: YARN-5655
> URL: https://issues.apache.org/jira/browse/YARN-5655
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: Robert Kanter
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: mvn_test.out, YARN-5655.001.patch
>
>
> TestContainerManagerSecurity has been failing recently in 2.8:
> {noformat}
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 80.928 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 44.478 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:394)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:337)
> at
>
[jira] [Commented] (YARN-5655) TestContainerManagerSecurity#testNMTokens is asserting
[
https://issues.apache.org/jira/browse/YARN-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043620#comment-16043620
]
Botong Huang commented on YARN-5655:
Hi [~rkanter] and [~templedf], I am also getting lots of NPE here. Using latest
trunk: $ mvn -Dtest=TestContainerManagerSecurity test
Can you please take another look? Thanks!
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 36.821 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 20.467 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 16.074 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:398)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:341)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:158)
> TestContainerManagerSecurity#testNMTokens is asserting
> --
>
> Key: YARN-5655
> URL: https://issues.apache.org/jira/browse/YARN-5655
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: Robert Kanter
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: mvn_test.out, YARN-5655.001.patch
>
>
> TestContainerManagerSecurity has been failing recently in 2.8:
> {noformat}
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 80.928 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 44.478 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:394)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:337)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 34.964 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:333)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6701) Add application type to capacity scheduler logs
Jonathan Hung created YARN-6701: --- Summary: Add application type to capacity scheduler logs Key: YARN-6701 URL: https://issues.apache.org/jira/browse/YARN-6701 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: Jonathan Hung Assignee: Jonathan Hung -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043470#comment-16043470 ] Varun Saxena commented on YARN-6638: Any need to load StaticUserWebFilter by default? It can be configured if required. Right? > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch, YARN-6638-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6698) Backport YARN-5121 to branch-2.7
[ https://issues.apache.org/jira/browse/YARN-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043467#comment-16043467 ] Chris Douglas commented on YARN-6698: - I just skimmed the backport and compared with YARN-5121, but lgtm. +1 > Backport YARN-5121 to branch-2.7 > > > Key: YARN-6698 > URL: https://issues.apache.org/jira/browse/YARN-6698 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: YARN-6698-branch-2.7-01.patch, > YARN-6698-branch-2.7-test.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Attachment: YARN-5953-YARN-5734.003.patch > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043413#comment-16043413 ] Jonathan Hung commented on YARN-5953: - Re-attached the 003 patch with the correct files added... > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Attachment: (was: YARN-5953-YARN-5734.003.patch) > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6127) Add support for work preserving NM restart when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043405#comment-16043405
]
Hadoop QA commented on YARN-6127:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 203 unchanged - 4 fixed = 203 total (was 207)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} hadoop-yarn-server-tests in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
43s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 27s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| | hadoop.yarn.server.TestContainerManagerSecurity |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6127 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872129/YARN-6127.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux
[jira] [Commented] (YARN-6355) Preprocessor framework for AM and Client interactions with the RM
[
https://issues.apache.org/jira/browse/YARN-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043388#comment-16043388
]
Subru Krishnan commented on YARN-6355:
--
Thanks [~asuresh] for working on this. I looked at the patch and have a few
high-level comments:
* Can you add a simple design doc with one diagram as the approach is
non-trivial to grok from only the patch. I had to have an offline discussions
with you to understand the nuances.
* I feel we should break up the jiras as there are multiple changes happening
here:
** Refactoring of the {{ApplicationMasterService}} to allow for the
filter/interceptor extension. This should be the baseline (status-quo) without
no regressions.
** Define the filter/interceptor extension. Consider using an private
interface to simplify the class hierarchy.
** Refactor the {{OpportunisticContainerAllocatorAMService}} to follow the
above model.
* The patch currently covers AM - RM protocol, we should reuse the work from
YARN-5411 for client - RM protocol.
> Preprocessor framework for AM and Client interactions with the RM
> -
>
> Key: YARN-6355
> URL: https://issues.apache.org/jira/browse/YARN-6355
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Labels: amrmproxy, resourcemanager
> Attachments: YARN-6355.001.patch, YARN-6355.002.patch
>
>
> Currently on the NM, we have the {{AMRMProxy}} framework to intercept the AM
> <-> RM communication and enforce policies. This is used both by YARN
> federation (YARN-2915) as well as Distributed Scheduling (YARN-2877).
> This JIRA proposes to introduce a similar framework on the the RM side, so
> that pluggable policies can be enforced on ApplicationMasterService centrally
> as well.
> This would be similar in spirit to a Java Servlet Filter Chain. Where the
> order of the interceptors can declared externally.
> Once possible usecase would be:
> the {{OpportunisticContainerAllocatorAMService}} is implemented as a wrapper
> over the {{ApplicationMasterService}}. It would probably be better to
> implement it as an Interceptor.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[
https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043387#comment-16043387
]
Hadoop QA commented on YARN-6638:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
24s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872126/YARN-6638-YARN-5355.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux baf7c685c65f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 79dae62 |
| Default Java | 1.8.0_131 |
| findbugs
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[
https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043360#comment-16043360
]
Hadoop QA commented on YARN-6638:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
36s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872126/YARN-6638-YARN-5355.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e6756d50ca70 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 79dae62 |
| Default Java | 1.8.0_131 |
| findbugs
[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations
[
https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043351#comment-16043351
]
Hadoop QA commented on YARN-5953:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
7s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
28s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5734 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
50s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
36s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 36s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
38s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
21s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
26s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 28 new +
6405 unchanged - 0 fixed = 6433 total (was 6405) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 21 new + 875 unchanged - 0 fixed = 896 total (was 875) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red}
[jira] [Updated] (YARN-6127) Add support for work preserving NM restart when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Botong Huang updated YARN-6127:
---
Attachment: YARN-6127.v2.patch
> Add support for work preserving NM restart when AMRMProxy is enabled
>
>
> Key: YARN-6127
> URL: https://issues.apache.org/jira/browse/YARN-6127
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, nodemanager
>Reporter: Subru Krishnan
>Assignee: Botong Huang
> Attachments: YARN-6127.v1.patch, YARN-6127.v2.patch
>
>
> YARN-1336 added the ability to restart NM without loosing any running
> containers. In a Federated YARN environment, there's additional state in the
> {{AMRMProxy}} to allow for spanning across multiple sub-clusters, so we need
> to enhance {{AMRMProxy}} to support work-preserving restart.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043321#comment-16043321 ] Wangda Tan commented on YARN-6679: -- [~daryn]/[~templedf], I just tried to test the patch using SLS: (a generated trace which consists of 8K nodes, each node has 128GB memory, 400 concurrent running apps, size of each app varies). >From my test, YARN scheduler can allocate 2700-3100 containers / sec on latest >trunk with and without the patch, I didn't see obvious regression however I >didn't see obvious improvement either. Not sure if it is the same for >branch-2.8 as we have already fixed some performance issues on trunk. I don't have chance to look into the issue since theoretically this patch should improve performance. Not sure if we hit any other bottleneck, [~daryn], could you share size / allocation-throughput of your cluster? > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6655) We should validate yarn native services application submission side to ensure that the hostname should be less than 63 characters
[ https://issues.apache.org/jira/browse/YARN-6655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-6655: - Attachment: YARN-6655-yarn-native-services.002.patch Updated patch that only performs the validation if RegistryDNS is enabled. > We should validate yarn native services application submission side to > ensure that the hostname should be less than 63 characters > -- > > Key: YARN-6655 > URL: https://issues.apache.org/jira/browse/YARN-6655 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: yarn-native-services >Reporter: Karam Singh >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-6655-yarn-native-services.001.patch, > YARN-6655-yarn-native-services.002.patch > > > According to RFC 1035 the length of a FQDN is limited to 255 characters, and > each label (node delimited by a dot in the hostname) is limited to 63 > characters > So We should validate yarn native services application submission side to > ensure that the hostname should be less than 63 characters -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6638: --- Attachment: YARN-6638-YARN-5355.03.patch > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch, YARN-6638-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5947) Create LeveldbConfigurationStore class using Leveldb as backing store
[
https://issues.apache.org/jira/browse/YARN-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043311#comment-16043311
]
Hadoop QA commented on YARN-5947:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
34s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
27s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
6s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} YARN-5734 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 30 new + 286 unchanged - 0 fixed = 316 total (was 286) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 4 new + 875 unchanged - 0 fixed = 879 total (was 875) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 35s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.LeveldbConfigurationStore.initialize(Configuration,
Configuration) At
LeveldbConfigurationStore.java:org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.LeveldbConfigurationStore.initialize(Configuration,
Configuration) At LeveldbConfigurationStore.java:[line 77] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.LeveldbConfigurationStore.initialize(Configuration,
Configuration):in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.LeveldbConfigurationStore.initialize(Configuration,
Configuration): new String(byte[]) At LeveldbConfigurationStore.java:[line
77] |
| | Found reliance on default encoding in
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[
https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043299#comment-16043299
]
Hadoop QA commented on YARN-6638:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 5 new +
2 unchanged - 2 fixed = 7 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
0s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872117/YARN-6638-YARN-5355.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 924eab153e78 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 79dae62 |
| Default Java | 1.8.0_131 |
| findbugs |
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043290#comment-16043290 ] Jian He commented on YARN-6638: --- To Rohith's question, the StaticUserWebFilter was always added by default in insecure mode, do we need this ? > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[
https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043280#comment-16043280
]
Hadoop QA commented on YARN-6638:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 5 new +
2 unchanged - 2 fixed = 7 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
42s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in
the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
18s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872117/YARN-6638-YARN-5355.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 102006f79627 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 79dae62 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1
[jira] [Created] (YARN-6700) Local directory health checks should not only find problems at the local dir level but also at the usercache and filecache level
Grant Sohn created YARN-6700: Summary: Local directory health checks should not only find problems at the local dir level but also at the usercache and filecache level Key: YARN-6700 URL: https://issues.apache.org/jira/browse/YARN-6700 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Grant Sohn Priority: Minor Chmod 400 the usercache directory. The node manager doesn't notice that the usercache is no longer usable. Since the usercache and filecache are part of the local dir, we should extend the checks on these too. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6699) [ATSv2 Security] Delegation token support for Timeline reader
Varun Saxena created YARN-6699: -- Summary: [ATSv2 Security] Delegation token support for Timeline reader Key: YARN-6699 URL: https://issues.apache.org/jira/browse/YARN-6699 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Saxena Assignee: Varun Saxena -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6691) Update YARN daemon startup/shutdown scripts to include Router service
[
https://issues.apache.org/jira/browse/YARN-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043224#comment-16043224
]
Hadoop QA commented on YARN-6691:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
35s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
22s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
11s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872113/YARN-6691-YARN-2915.v4.patch
|
| Optional Tests | asflicense shellcheck shelldocs mvnsite unit |
| uname | Linux 60a3ea77534a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / e3ed636 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16153/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16153/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update YARN daemon startup/shutdown scripts to include Router service
> -
>
> Key: YARN-6691
> URL: https://issues.apache.org/jira/browse/YARN-6691
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-6691-YARN-2915.v1.patch,
> YARN-6691-YARN-2915.v2.patch, YARN-6691-YARN-2915.v3.patch,
> YARN-6691-YARN-2915.v4.patch
>
>
> YARN-5410 introduce a new YARN service, i.e. Router. This jira proposes to
> update YARN daemon startup/shutdown scripts to include Router service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Attachment: (was: YARN-5953-YARN-5734.003.patch) > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043217#comment-16043217 ] Jonathan Hung commented on YARN-5953: - Thanks [~xgong] for the review! Attached 003 which moves these classes to oahy.webapp.dao, fixed findbugs, fixed whitespace, fixed checkstyle. > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Attachment: YARN-5953-YARN-5734.003.patch > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Attachment: YARN-5953-YARN-5734.003.patch > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043216#comment-16043216 ] Varun Saxena commented on YARN-6638: I have added a TimelineReaderAuthenticationFilterInitializer instead of using AuthenticationFilterInitializer to use timeline service specific configurations so that when we move to TimelineAuthenticationFilterInitializer for DT support, the configurations used for kerberos authentication remain the same. > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6585) RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels.
[
https://issues.apache.org/jira/browse/YARN-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043215#comment-16043215
]
Sunil G commented on YARN-6585:
---
Thanks [~eepayne]
bq. I feel that these issues should be fixed as part of a separate JIRA since
they are confusing things here and are not related to fixing this problem.
I think its fine to take this changes in separate ticket. Ideally
{{initNodeLabels}} changes are enough to support for breakage sceanrios. Also
thanks you for sharing the test case which I think will help here.
I ll update a patch shortwhile.
> RM fails to start when upgrading from 2.7 to 2.8 for clusters with node
> labels.
> ---
>
> Key: YARN-6585
> URL: https://issues.apache.org/jira/browse/YARN-6585
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Payne
>Assignee: Sunil G
>Priority: Blocker
> Attachments: YARN-6585.0001.patch, YARN-6585.0002.patch
>
>
> {noformat}
> Caused by: java.io.IOException: Not all labels being replaced contained by
> known label collections, please check, new labels=[abc]
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.checkReplaceLabelsOnNode(CommonNodeLabelsManager.java:718)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.replaceLabelsOnNode(CommonNodeLabelsManager.java:737)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.replaceLabelsOnNode(RMNodeLabelsManager.java:189)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.loadFromMirror(FileSystemNodeLabelsStore.java:181)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:208)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:251)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6638: --- Attachment: YARN-6638-YARN-5355.02.patch > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043202#comment-16043202 ] Sean Po commented on YARN-6528: --- There are three check style failures: The first two are unavoidable without making a dramatic API change to InMemoryPlan. The last one is about about setting ReservationQueueMetrics to be a final class. This cannot be avoided because final classes cannot be mocked, and the following tests will fail: # hadoop.yarn.server.resourcemanager.reservation.planning.TestSimpleCapacityReplanner # hadoop.yarn.server.resourcemanager.reservation.TestInMemoryPlan # hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent # hadoop.yarn.server.resourcemanager.TestRMRestart # hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy # hadoop.yarn.server.resourcemanager.reservation.planning.TestAlignedPlanner # hadoop.yarn.server.resourcemanager.reservation.TestNoOverCommitPolicy > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6691) Update YARN daemon startup/shutdown scripts to include Router service
[ https://issues.apache.org/jira/browse/YARN-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043156#comment-16043156 ] Giovanni Matteo Fumarola commented on YARN-6691: Thanks [~aw]. I updated the patch with your comment (V4). > Update YARN daemon startup/shutdown scripts to include Router service > - > > Key: YARN-6691 > URL: https://issues.apache.org/jira/browse/YARN-6691 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6691-YARN-2915.v1.patch, > YARN-6691-YARN-2915.v2.patch, YARN-6691-YARN-2915.v3.patch, > YARN-6691-YARN-2915.v4.patch > > > YARN-5410 introduce a new YARN service, i.e. Router. This jira proposes to > update YARN daemon startup/shutdown scripts to include Router service. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6691) Update YARN daemon startup/shutdown scripts to include Router service
[ https://issues.apache.org/jira/browse/YARN-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6691: --- Attachment: YARN-6691-YARN-2915.v4.patch > Update YARN daemon startup/shutdown scripts to include Router service > - > > Key: YARN-6691 > URL: https://issues.apache.org/jira/browse/YARN-6691 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6691-YARN-2915.v1.patch, > YARN-6691-YARN-2915.v2.patch, YARN-6691-YARN-2915.v3.patch, > YARN-6691-YARN-2915.v4.patch > > > YARN-5410 introduce a new YARN service, i.e. Router. This jira proposes to > update YARN daemon startup/shutdown scripts to include Router service. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043128#comment-16043128 ] Hudson commented on YARN-6679: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11844 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11844/]) YARN-6679. Reduce Resource instance overhead via non-PBImpl (Contributed (templedf: rev a062374c396ba2bddbbe4d3806cc2896293b4e0b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/protocolrecords/TestProtocolRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ReservationRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/ContainerTokenIdentifierForTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/records/impl/pb/ContainerStartDataPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NMContainerStatusPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceOptionPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceAllocationRequestPBImpl.java > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043120#comment-16043120 ] Daniel Templeton commented on YARN-6679: Sure. I just figured since the patch has been posted for 2 days, that I'd go ahead and get it in. > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6679: --- Fix Version/s: 2.9.0 Committed to branch-2 as well. > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043109#comment-16043109 ] Wangda Tan commented on YARN-6679: -- [~templedf] thanks for reviewing the patch, I just realized you have already committed the patch, will do perf tests for the patch anyway. Could you wait at least 1 day before committing non trivial patches next time? > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043096#comment-16043096 ] Wangda Tan commented on YARN-6679: -- [~daryn], thanks for the patch, I'm trying to use SLS to test performance of the patch, will share test results shortly. > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5947) Create LeveldbConfigurationStore class using Leveldb as backing store
[ https://issues.apache.org/jira/browse/YARN-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5947: Attachment: YARN-5947-YARN-5734.002.patch > Create LeveldbConfigurationStore class using Leveldb as backing store > - > > Key: YARN-5947 > URL: https://issues.apache.org/jira/browse/YARN-5947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5947.001.patch, YARN-5947-YARN-5734.001.patch, > YARN-5947-YARN-5734.002.patch > > > LeveldbConfigurationStore will extend YarnConfigurationStore for storing > scheduler configuration in a LevelDB. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5947) Create LeveldbConfigurationStore class using Leveldb as backing store
[ https://issues.apache.org/jira/browse/YARN-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043087#comment-16043087 ] Jonathan Hung commented on YARN-5947: - Fixed compilation error... > Create LeveldbConfigurationStore class using Leveldb as backing store > - > > Key: YARN-5947 > URL: https://issues.apache.org/jira/browse/YARN-5947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5947.001.patch, YARN-5947-YARN-5734.001.patch, > YARN-5947-YARN-5734.002.patch > > > LeveldbConfigurationStore will extend YarnConfigurationStore for storing > scheduler configuration in a LevelDB. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043001#comment-16043001 ] Daniel Templeton commented on YARN-6679: LGTM +1! > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042993#comment-16042993 ] Xuan Gong commented on YARN-5953: - Overall looks good. Some nits: 1. Could we move QueueConfigInfo.java and SchedConfUpdateInfo.java to package: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/dao ? 2. Also please fix the findBug issue. > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042972#comment-16042972 ] Daniel Templeton commented on YARN-6679: Sorry, I've been out sick. Back now. I'll take a look. > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6682) Improve performance of AssignmentInformation datastructures
[
https://issues.apache.org/jira/browse/YARN-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042953#comment-16042953
]
Daryn Sharp commented on YARN-6682:
---
This is a simple/self-contained change. Does anyone have time to review?
> Improve performance of AssignmentInformation datastructures
> ---
>
> Key: YARN-6682
> URL: https://issues.apache.org/jira/browse/YARN-6682
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: YARN-6682.branch-2.8.patch, YARN-6682.branch-2.patch,
> YARN-6682.trunk.patch
>
>
> {{AssignmentInformation}} is inefficient and creates lots of garbage that
> increase gc pressure. It creates 3 hashmaps that each contain only 2
> enum-based keys. This requires wrapper node objects, boxing/unboxing of ints,
> and more expensive lookups than simply using primitive arrays indexed by enum
> ordinal.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6681) Eliminate double-copy of child queues in canAssignToThisQueue
[
https://issues.apache.org/jira/browse/YARN-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042950#comment-16042950
]
Daryn Sharp commented on YARN-6681:
---
[~Ying Zhang], are you ok with the current patch?
> Eliminate double-copy of child queues in canAssignToThisQueue
> -
>
> Key: YARN-6681
> URL: https://issues.apache.org/jira/browse/YARN-6681
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: YARN-6681.2.branch-2.8.patch,
> YARN-6681.2.branch-2.patch, YARN-6681.2.trunk.patch,
> YARN-6681.branch-2.8.patch, YARN-6681.branch-2.patch, YARN-6681.trunk.patch
>
>
> 20% of the time in {{AbstractCSQueue#canAssignToThisQueue}} is spent
> performing two duplications a treemap of child queues into a list - once to
> test for null, second to see if it's empty. Eliminating the dups reduces the
> overhead to 2%.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6680) Avoid locking overhead for NO_LABEL lookups
[ https://issues.apache.org/jira/browse/YARN-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042949#comment-16042949 ] Daryn Sharp commented on YARN-6680: --- Any feedback? > Avoid locking overhead for NO_LABEL lookups > --- > > Key: YARN-6680 > URL: https://issues.apache.org/jira/browse/YARN-6680 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-6680.patch > > > Labels are managed via a hash that is protected with a read lock. The lock > acquire and release are each just as expensive as the hash lookup itself - > resulting in a 3X slowdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042946#comment-16042946 ] Daryn Sharp commented on YARN-6679: --- [~dan...@cloudera.com], have I addressed your concerns? > Reduce Resource instance overhead via non-PBImpl > > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6691) Update YARN daemon startup/shutdown scripts to include Router service
[
https://issues.apache.org/jira/browse/YARN-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042793#comment-16042793
]
Allen Wittenauer commented on YARN-6691:
{code}
+# Specify the max heapsize for the Router. If no units are
+# given, it will be assumed to be in MB.
+# This value will be overridden by an Xmx setting specified in either
+# HADOOP_OPTS and/or YARN_ROUTER_OPTS.
+# Default is the same as HADOOP_HEAPSIZE_MAX.
+#export YARN_ROUTER_HEAPSIZE=
+
{code}
The \_HEAPSIZE vars are only used for backward compatibility. All new daemons
in 3.x use _OPTS to set heap.
> Update YARN daemon startup/shutdown scripts to include Router service
> -
>
> Key: YARN-6691
> URL: https://issues.apache.org/jira/browse/YARN-6691
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-6691-YARN-2915.v1.patch,
> YARN-6691-YARN-2915.v2.patch, YARN-6691-YARN-2915.v3.patch
>
>
> YARN-5410 introduce a new YARN service, i.e. Router. This jira proposes to
> update YARN daemon startup/shutdown scripts to include Router service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5814) Add druid as storage backend in YARN Timeline Service
[ https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042759#comment-16042759 ] daemon commented on YARN-5814: -- [~BINGXUE QIU] hi, bingxue，can you upload your patch? Your patch is very useful for me! > Add druid as storage backend in YARN Timeline Service > -- > > Key: YARN-5814 > URL: https://issues.apache.org/jira/browse/YARN-5814 > Project: Hadoop YARN > Issue Type: New Feature > Components: ATSv2 >Affects Versions: 3.0.0-alpha2 >Reporter: Bingxue Qiu > Attachments: Add-Druid-in-YARN-Timeline-Service.pdf > > > h3. Introduction > I propose to add druid as storage backend in YARN Timeline Service. > We run more than 6000 applications and generate 450 million metrics daily in > Alibaba Clusters with thousands of nodes. We need to collect and store > meta/events/metrics data, online analyze the utilization reports of various > dimensions and display the trends of allocation/usage resources for cluster > by joining and aggregating data. It helps us to manage and optimize the > cluster by tracking resource utilization. > To achieve our goal we have changed to use druid as the storage instead of > HBase and have achieved sub-second OLAP performance in our production > environment for few months. > h3. Analysis > Currently YARN Timeline Service only supports aggregating metrics at a) flow > level by FlowRunCoprocessor and b) application level metrics aggregating by > AppLevelTimelineCollector, offline (time-based periodic) aggregation for > flows/users/queues for reporting and analysis is planned but not yet > implemented. YARN Timeline Service chooses Apache HBase as the primary > storage backend. As we all know that HBase doesn't fit for OLAP. > For arbitrary exploration of data,such as online analyze the utilization > reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by > joining and aggregating data, Druid's custom column format enables ad-hoc > queries without pre-computation. The format also enables fast scans on > columns, which is important for good aggregation performance. > To achieve our goal that support to online analyze the utilization reports of > various dimensions, display the variation trends of allocation/usage > resources for cluster, and arbitrary exploration of data, we propose to add > druid storage and implement DruidWriter /DruidReader in YARN Timeline Service. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6602) Impersonation does not work if standby RM is contacted first
[
https://issues.apache.org/jira/browse/YARN-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042652#comment-16042652
]
Kiran N edited comment on YARN-6602 at 6/8/17 1:42 PM:
---
Is this applicable to 2.7.4.
If applicable can you please help in backporting the patch to 2.7.4.
was (Author: kwx417842):
Is this applicable to 2.7.4.
If applicable can you backport the patch to 2.7.4.
> Impersonation does not work if standby RM is contacted first
>
>
> Key: YARN-6602
> URL: https://issues.apache.org/jira/browse/YARN-6602
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-6602.001.patch, YARN-6602.002.patch
>
>
> When RM HA is enabled, impersonation does not work correctly if the Yarn
> Client connects to the standby RM first. When this happens, the
> impersonation is "lost" and the client does things on behalf of the
> impersonator user. We saw this with the OOZIE-1770 Oozie on Yarn feature.
> I need to investigate this some more, but it appears to be related to
> delegation tokens. When this issue occurs, the tokens have the owner as
> "oozie" instead of the actual user. On a hunch, we found a workaround that
> explicitly adding a correct RM HA delegation token fixes the problem:
> {code:java}
> org.apache.hadoop.yarn.api.records.Token token =
> yarnClient.getRMDelegationToken(ClientRMProxy.getRMDelegationTokenService(conf));
> org.apache.hadoop.security.token.Token token2 = new
> org.apache.hadoop.security.token.Token(token.getIdentifier().array(),
> token.getPassword().array(), new Text(token.getKind()), new
> Text(token.getService()));
> UserGroupInformation.getCurrentUser().addToken(token2);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6638: --- Summary: [ATSv2 Security] Timeline reader side changes for loading auth filters and principals (was: [Security] Timeline reader side changes for loading auth filters and principals) > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3591) Resource localization on a bad disk causes subsequent containers failure
[ https://issues.apache.org/jira/browse/YARN-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042656#comment-16042656 ] Kiran N commented on YARN-3591: --- Can this be backported to 2.7.4..? > Resource localization on a bad disk causes subsequent containers failure > - > > Key: YARN-3591 > URL: https://issues.apache.org/jira/browse/YARN-3591 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Lavkesh Lahngir >Assignee: Lavkesh Lahngir > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-3591.1.patch, 0001-YARN-3591.patch, > YARN-3591.2.patch, YARN-3591.3.patch, YARN-3591.4.patch, YARN-3591.5.patch, > YARN-3591.6.patch, YARN-3591.7.patch, YARN-3591.8.patch, YARN-3591.9.patch > > > It happens when a resource is localised on the disk, after localising that > disk has gone bad. NM keeps paths for localised resources in memory. At the > time of resource request isResourcePresent(rsrc) will be called which calls > file.exists() on the localised path. > In some cases when disk has gone bad, inodes are stilled cached and > file.exists() returns true. But at the time of reading, file will not open. > Note: file.exists() actually calls stat64 natively which returns true because > it was able to find inode information from the OS. > A proposal is to call file.list() on the parent path of the resource, which > will call open() natively. If the disk is good it should return an array of > paths with length at-least 1. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6602) Impersonation does not work if standby RM is contacted first
[
https://issues.apache.org/jira/browse/YARN-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042652#comment-16042652
]
Kiran N commented on YARN-6602:
---
Is this applicable to 2.7.4.
If applicable can you backport the patch to 2.7.4.
> Impersonation does not work if standby RM is contacted first
>
>
> Key: YARN-6602
> URL: https://issues.apache.org/jira/browse/YARN-6602
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-6602.001.patch, YARN-6602.002.patch
>
>
> When RM HA is enabled, impersonation does not work correctly if the Yarn
> Client connects to the standby RM first. When this happens, the
> impersonation is "lost" and the client does things on behalf of the
> impersonator user. We saw this with the OOZIE-1770 Oozie on Yarn feature.
> I need to investigate this some more, but it appears to be related to
> delegation tokens. When this issue occurs, the tokens have the owner as
> "oozie" instead of the actual user. On a hunch, we found a workaround that
> explicitly adding a correct RM HA delegation token fixes the problem:
> {code:java}
> org.apache.hadoop.yarn.api.records.Token token =
> yarnClient.getRMDelegationToken(ClientRMProxy.getRMDelegationTokenService(conf));
> org.apache.hadoop.security.token.Token token2 = new
> org.apache.hadoop.security.token.Token(token.getIdentifier().array(),
> token.getPassword().array(), new Text(token.getKind()), new
> Text(token.getService()));
> UserGroupInformation.getCurrentUser().addToken(token2);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6601) Allow service to be started as System Services during serviceapi start up
[
https://issues.apache.org/jira/browse/YARN-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042528#comment-16042528
]
Hadoop QA commented on YARN-6601:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
17s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
yarn-native-services has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
29s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 29s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 7 new + 206 unchanged - 1 fixed = 213 total (was 207) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
46s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s{color}
| {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6601 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872029/YARN-6601-yarn-native-services.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 2f5af056ee39 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / c9d9c94 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16151/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
[jira] [Updated] (YARN-6601) Allow service to be started as System Services during serviceapi start up
[ https://issues.apache.org/jira/browse/YARN-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated YARN-6601: -- Attachment: YARN-6601-yarn-native-services.003.patch > Allow service to be started as System Services during serviceapi start up > - > > Key: YARN-6601 > URL: https://issues.apache.org/jira/browse/YARN-6601 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S > Attachments: SystemServices.pdf, > YARN-6601-yarn-native-services.001.patch, > YARN-6601-yarn-native-services.002.patch, > YARN-6601-yarn-native-services.003.patch > > > This is extended from YARN-1593 focusing only on system services. System > services are started during boot up of daemon or admin can be configurable > and started at any point of time. These services have special characteristics > which need to be respected. The document covers details about system services > characteristics. > This JIRA is focusing on configuring services using a json template and > placing in a shared filesystem. During YARN REST server( native-service-api) > start up read services details from shared location and start those services. > If there are services already configured than skip those services and > continue to start up the services. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042434#comment-16042434 ] Varun Saxena commented on YARN-6638: bq. How about reusing reusing RM timeline APIs for reader server? But one potential issue I see is reader server can be any number deployed in cluster, so RM will issue token to which reader server? I think the reader instance itself should issue delegation token. We would have to address recovery of tokens as well if we do it. Once we have work for offline timeline collector in, even collectors may need to recover token to have them across restarts. So I had thought we can add DT support in reader then. I will change the patch to use Authentication Filter as of now. And add back the Timeline Auth filter once the support for delegation token is added. We do not have a JIRA for it as of now. Will raise it. I think we can have it in the next iteration. We do not need it immediately. Or for your use case scenario, do you need it immediately? If yes, we can do it in this iteration itself. > [Security] Timeline reader side changes for loading auth filters and > principals > --- > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6698) Backport YARN-5121 to branch-2.7
[
https://issues.apache.org/jira/browse/YARN-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042412#comment-16042412
]
Hadoop QA commented on YARN-6698:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
32s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
12s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
3s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
33s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
54s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
16s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
16s{color} | {color:green} root-jdk1.8.0_131 with JDK v1.8.0_131 generated 0
new + 31 unchanged - 3 fixed = 31 total (was 34) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
11s{color} | {color:green} root-jdk1.7.0_131 with JDK v1.7.0_131 generated 0
new + 41 unchanged - 3 fixed = 41 total (was 44) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 4590 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 2m
18s{color} | {color:red} The patch 37 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
29s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
52s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 26s{color}
| {color:red} root in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_131 Failed junit tests | hadoop.util.bloom.TestBloomFilters |
| JDK v1.8.0_131 Timed out junit tests |
org.apache.hadoop.conf.TestConfiguration |
| JDK v1.7.0_131 Failed junit tests | hadoop.util.bloom.TestBloomFilters |
| JDK v1.7.0_131 Timed out junit tests |
org.apache.hadoop.conf.TestConfiguration |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:67e87c9 |
| JIRA Issue | YARN-6698 |
| JIRA Patch URL |
[jira] [Commented] (YARN-6683) Invalid event: COLLECTOR_UPDATE at KILLED
[
https://issues.apache.org/jira/browse/YARN-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042294#comment-16042294
]
Rohith Sharma K S commented on YARN-6683:
-
FYI to ATSv2 folks, this JIRA need not to backport to YARN-5355 branch. As part
of YARN-5638, fix is given with same approach as this jira.
> Invalid event: COLLECTOR_UPDATE at KILLED
> -
>
> Key: YARN-6683
> URL: https://issues.apache.org/jira/browse/YARN-6683
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Rohith Sharma K S
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6683.001.patch, YARN-6683.001.patch
>
>
> {code}
> 2017-06-01 20:01:22,686 ERROR rmapp.RMAppImpl (RMAppImpl.java:handle(905)) -
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> COLLECTOR_UPDATE at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:903)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:904)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:888)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> {code}
> Below code already gets the RMApp instance and then send an event to RMApp to
> update the collector address. Instead of updating via event, it could just
> update via a method of RMApp. This also avoids state-machine changes.
> Also, is there any implications that COLLECTOR_UPDATE happened at KILLED
> state ?
> {code}
> } else {
> String previousCollectorAddr = rmApp.getCollectorAddr();
> if (previousCollectorAddr == null
> || !previousCollectorAddr.equals(collectorAddr)) {
> // sending collector update event.
> RMAppCollectorUpdateEvent event =
> new RMAppCollectorUpdateEvent(appId, collectorAddr);
> rmContext.getDispatcher().getEventHandler().handle(event);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6698) Backport YARN-5121 to branch-2.7
[ https://issues.apache.org/jira/browse/YARN-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042284#comment-16042284 ] Akira Ajisaka commented on YARN-6698: - I ran all the tests in yarn-server-nodemanager module on macOS 10.12.5 and all of them passed except TestNMProxy. HADOOP-11212 and YARN-4916 are related and they should be backported to fix this test failure. > Backport YARN-5121 to branch-2.7 > > > Key: YARN-6698 > URL: https://issues.apache.org/jira/browse/YARN-6698 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: YARN-6698-branch-2.7-01.patch, > YARN-6698-branch-2.7-test.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042282#comment-16042282 ] Rohith Sharma K S commented on YARN-6638: - Reg the patch, overall looks good. I see loading static web app filter is being removed. doesn't it required? > [Security] Timeline reader side changes for loading auth filters and > principals > --- > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6638) [Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042279#comment-16042279 ] Rohith Sharma K S commented on YARN-6638: - bq. I had thought, for reader, merely Kerberos authentication should be enough. So an authentication filter should also be enough. this would lead every HTTP request is authenticated with kerberos which will impact delay in response time. As you said, as of now we can go ahead with it, but this one would need to address it. How about reusing reusing RM timeline APIs for reader server? But one potential issue I see is reader server can be any number deployed in cluster, so RM will issue token to which reader server!? Is there any JIRA for reader server DT implementation? If no, lets raise and discuss this point over there. > [Security] Timeline reader side changes for loading auth filters and > principals > --- > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6638-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6698) Backport YARN-5121 to branch-2.7
[ https://issues.apache.org/jira/browse/YARN-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-6698: Attachment: YARN-6698-branch-2.7-01.patch Backported YARN-4017. The patch can be applied cleanly. > Backport YARN-5121 to branch-2.7 > > > Key: YARN-6698 > URL: https://issues.apache.org/jira/browse/YARN-6698 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: YARN-6698-branch-2.7-01.patch, > YARN-6698-branch-2.7-test.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6686) Support for adding and removing queue mappings
[
https://issues.apache.org/jira/browse/YARN-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042267#comment-16042267
]
Jonathan Hung commented on YARN-6686:
-
Currently it seems difficult to add native support for adding/removing queue
mappings since it is configurable. Since the UserGroupMappingPlacementRule uses
csv to configure queue mappings, I'm thinking we can add support for
adding/removing from a configuration which uses csv. This way a configuration
like accessible-node-labels can also use it. We are also planning on adding our
own PlacementRule which uses csv as well.
On a somewhat related note, it seems it's not very easy to configure queue acls
via REST/CLI, since the format (user1,user2 group1,group2) is unique to the
acl_submit_applications / acl_administer_queue configurations (as far as I
know). Whatever route we end up taking here, it probably makes sense to have
some endpoint to get a configuration value (right now the endpoint is
write-only).
> Support for adding and removing queue mappings
> --
>
> Key: YARN-6686
> URL: https://issues.apache.org/jira/browse/YARN-6686
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>
> Right now capacity scheduler uses UserGroupMappingPlacementRule to determine
> queue mappings. This rule stores mappings in
> {{yarn.scheduler.capacity.queue-mappings}}. For users with a large number of
> mappings, adding or removing queue mappings becomes infeasible.
> Need to come up with a way to add/remove individual mappings, for any/all
> different configured placement rules.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6697) Catch an exception, exception information needs to be recorded, even if the probability of occurrence is very low
[ https://issues.apache.org/jira/browse/YARN-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042246#comment-16042246 ] ZhangBing Lin commented on YARN-6697: - [~templedf],thank you for your answer carefully, I think you're right, deletionService not necessary logging,the ResourceLocalizationService logging is not clear,I just think in the code specification that this is a problem, did not think so much. I think it adding a comment may be better. > Catch an exception, exception information needs to be recorded, even if the > probability of occurrence is very low > - > > Key: YARN-6697 > URL: https://issues.apache.org/jira/browse/YARN-6697 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Attachments: YARN-6697.001.patch > > > Catch an exception, exception information needs to be recorded, even if the > probability of occurrence is very low.I think a good programming habit is > needed to do this, it's just what I think -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
