[jira] [Updated] (YARN-4630) Remove useless boxing/unboxing code
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated YARN-4630: Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Summary: Remove useless boxing/unboxing code (was: Remove useless boxing/unboxing code (Hadoop YARN)) > Remove useless boxing/unboxing code > --- > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4630.0.patch, YARN-4630.1.patch, YARN-4630.2.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234519#comment-15234519 ] Akira AJISAKA commented on YARN-4630: - +1, the test failures seem to be unrelated. I ran all the failed tests and they passed except TestNodeLabelContainerAllocation, which is tracked by YARN-4890. Committing this. > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch, YARN-4630.1.patch, YARN-4630.2.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234508#comment-15234508
]
Bibin A Chundatt commented on YARN-3971:
[~Naganarasimha]
Thank you for review comment. Attached patch after handling the same
> Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel
> recovery
> --
>
> Key: YARN-3971
> URL: https://issues.apache.org/jira/browse/YARN-3971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-3971.patch, 0002-YARN-3971.patch,
> 0003-YARN-3971.patch, 0004-YARN-3971.patch,
> 0005-YARN-3971.001.addendum.patch, 0005-YARN-3971.addendum.patch,
> 0005-YARN-3971.patch
>
>
> Steps to reproduce
> # Create label x,y
> # Delete label x,y
> # Create label x,y add capacity scheduler xml for labels x and y too
> # Restart RM
>
> Both RM will become Standby.
> Since below exception is thrown on {{FileSystemNodeLabelsStore#recover}}
> {code}
> 2015-07-23 14:03:33,627 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager failed in
> state STARTED; cause: java.io.IOException: Cannot remove label=x, because
> queue=a1 is using this label. Please remove label on queue before remove the
> label
> java.io.IOException: Cannot remove label=x, because queue=a1 is using this
> label. Please remove label on queue before remove the label
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.checkRemoveFromClusterNodeLabelsOfQueue(RMNodeLabelsManager.java:104)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.removeFromClusterNodeLabels(RMNodeLabelsManager.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:221)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:232)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:245)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:587)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3971:
---
Attachment: 0005-YARN-3971.001.addendum.patch
> Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel
> recovery
> --
>
> Key: YARN-3971
> URL: https://issues.apache.org/jira/browse/YARN-3971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-3971.patch, 0002-YARN-3971.patch,
> 0003-YARN-3971.patch, 0004-YARN-3971.patch,
> 0005-YARN-3971.001.addendum.patch, 0005-YARN-3971.addendum.patch,
> 0005-YARN-3971.patch
>
>
> Steps to reproduce
> # Create label x,y
> # Delete label x,y
> # Create label x,y add capacity scheduler xml for labels x and y too
> # Restart RM
>
> Both RM will become Standby.
> Since below exception is thrown on {{FileSystemNodeLabelsStore#recover}}
> {code}
> 2015-07-23 14:03:33,627 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager failed in
> state STARTED; cause: java.io.IOException: Cannot remove label=x, because
> queue=a1 is using this label. Please remove label on queue before remove the
> label
> java.io.IOException: Cannot remove label=x, because queue=a1 is using this
> label. Please remove label on queue before remove the label
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.checkRemoveFromClusterNodeLabelsOfQueue(RMNodeLabelsManager.java:104)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.removeFromClusterNodeLabels(RMNodeLabelsManager.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:221)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:232)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:245)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:587)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3971) Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery
[
https://issues.apache.org/jira/browse/YARN-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234503#comment-15234503
]
Naganarasimha G R commented on YARN-3971:
-
Hi [~bibinchundatt], thanks for providing the addendum patch, was able to apply
only the test case from the patch and reproduce the issue . I am fine with the
approach taken. Just one small nit.
* Instead of creating a subclass just to verify whether the method was called
or not we can make use of the Mockito.spy on the actual object and then try to
use Mockito.verify method
{code}
lmgr = spy(new RMNodeLabelsManager());
verify(lmgr,times(0)).checkRemoveFromClusterNodeLabelsOfQueue(any(Collection.class));
{code}
> Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel
> recovery
> --
>
> Key: YARN-3971
> URL: https://issues.apache.org/jira/browse/YARN-3971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-3971.patch, 0002-YARN-3971.patch,
> 0003-YARN-3971.patch, 0004-YARN-3971.patch, 0005-YARN-3971.addendum.patch,
> 0005-YARN-3971.patch
>
>
> Steps to reproduce
> # Create label x,y
> # Delete label x,y
> # Create label x,y add capacity scheduler xml for labels x and y too
> # Restart RM
>
> Both RM will become Standby.
> Since below exception is thrown on {{FileSystemNodeLabelsStore#recover}}
> {code}
> 2015-07-23 14:03:33,627 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager failed in
> state STARTED; cause: java.io.IOException: Cannot remove label=x, because
> queue=a1 is using this label. Please remove label on queue before remove the
> label
> java.io.IOException: Cannot remove label=x, because queue=a1 is using this
> label. Please remove label on queue before remove the label
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.checkRemoveFromClusterNodeLabelsOfQueue(RMNodeLabelsManager.java:104)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager.removeFromClusterNodeLabels(RMNodeLabelsManager.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:221)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:232)
> at
> org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:245)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:587)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:832)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:422)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4168) Test TestLogAggregationService.testLocalFileDeletionOnDiskFull failing
[
https://issues.apache.org/jira/browse/YARN-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234482#comment-15234482
]
Hadoop QA commented on YARN-4168:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 20s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 53s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797956/YARN-4168.3.patch |
| JIRA Issue | YARN-4168 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 25207f975dad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1b78b2b |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |
[jira] [Commented] (YARN-4562) YARN WebApp ignores the configuration passed to it for keystore settings
[
https://issues.apache.org/jira/browse/YARN-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234479#comment-15234479
]
Hudson commented on YARN-4562:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9589 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9589/])
YARN-4562. YARN WebApp ignores the configuration passed to it for (vvasudev:
rev 07920acc165f17e8fe3ca9d21283a2123223f438)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
> YARN WebApp ignores the configuration passed to it for keystore settings
>
>
> Key: YARN-4562
> URL: https://issues.apache.org/jira/browse/YARN-4562
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.9.0
>
> Attachments: YARN-4562.patch
>
>
> The conf can be passed to WebApps builder, however the following code in
> WebApps.java that builds the HttpServer2 object:
> {noformat}
> if (httpScheme.equals(WebAppUtils.HTTPS_PREFIX)) {
> WebAppUtils.loadSslConfiguration(builder);
> }
> {noformat}
> ...results in loadSslConfiguration creating a new Configuration object; the
> one that is passed in is ignored, as far as the keystore/etc. settings are
> concerned. loadSslConfiguration has another overload with Configuration
> parameter that should be used instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-857) Localization failures should be available in container diagnostics
[
https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234478#comment-15234478
]
Hadoop QA commented on YARN-857:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 8s {color} |
{color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 37s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797201/YARN-857-20160405.txt
|
| JIRA
[jira] [Updated] (YARN-4855) Should check if node exists when replace nodelabels
[ https://issues.apache.org/jira/browse/YARN-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4855: -- Attachment: YARN-4855.002.patch > Should check if node exists when replace nodelabels > --- > > Key: YARN-4855 > URL: https://issues.apache.org/jira/browse/YARN-4855 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: YARN-4855.001.patch, YARN-4855.002.patch > > > Today when we add nodelabels to nodes, it would succeed even if nodes are not > existing NodeManger in cluster without any message. > It could be like this: > When we use *yarn rmadmin -replaceLabelsOnNode "node1=label1"*, it would be > denied if node does not exist. > When we use *yarn rmadmin -replaceLabelsOnNode -force "node1=label1"* would > add nodelabels no matter whether node exists -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4168) Test TestLogAggregationService.testLocalFileDeletionOnDiskFull failing
[
https://issues.apache.org/jira/browse/YARN-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-4168:
--
Attachment: YARN-4168.3.patch
The latest patch almost looks good to me.
Making trivial changes to try for a little longer and keeping the retries per
log-file.
Will check this in if Jenkins says okay.
> Test TestLogAggregationService.testLocalFileDeletionOnDiskFull failing
> --
>
> Key: YARN-4168
> URL: https://issues.apache.org/jira/browse/YARN-4168
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Takashi Ohnishi
>Priority: Critical
> Attachments: YARN-4168.1.patch, YARN-4168.2.patch, YARN-4168.3.patch
>
>
> {{TestLogAggregationService.testLocalFileDeletionOnDiskFull}} failing on
> [Jenkins build
> 1136|https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/1136/testReport/junit/org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation/TestLogAggregationService/testLocalFileDeletionOnDiskFull/]
> {code}
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.verifyLocalFileDeletion(TestLogAggregationService.java:229)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLocalFileDeletionOnDiskFull(TestLogAggregationService.java:285)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4794) Deadlock in NMClientImpl
[
https://issues.apache.org/jira/browse/YARN-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234414#comment-15234414
]
Rohith Sharma K S commented on YARN-4794:
-
Yes, this can be optimized!
> Deadlock in NMClientImpl
>
>
> Key: YARN-4794
> URL: https://issues.apache.org/jira/browse/YARN-4794
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-4794.1.patch
>
>
> Distributed shell app gets stuck on stopping containers after App completes
> with the following exception
> {code:title = app log}
> 15/12/10 14:52:20 INFO distributedshell.ApplicationMaster: Application
> completed. Stopping running containers
> 15/12/10 14:52:20 WARN ipc.Client: Exception encountered while connecting to
> the server : java.nio.channels.ClosedByInterruptException
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4935) TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234365#comment-15234365 ] Karthik Kambatla commented on YARN-4935: If the test is capacity scheduler specific, should it be in the capacity scheduler tests? If the goal is to check the YarnClient behavior, may be the name of the test should reflect it is testing only capacity scheduler? > TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler > > > Key: YARN-4935 > URL: https://issues.apache.org/jira/browse/YARN-4935 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-4935.001.patch > > > This test case introduced by YARN-3131 works well on CapacityScheduler but > not on FairScheduler, since CS doesn't allow dynamically create a queue, > while FS supports it. So if you give a random queue name, CS will reject it, > but FS will create a new queue for it by default. > One simple solution is to specific CS in this test case. /cc [~lichangleo]. I > was thinking about creating another test case for FS. But for the code > introduced by YARN-3131, it may be not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[
https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234332#comment-15234332
]
Hadoop QA commented on YARN-4630:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 10s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed with
JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 1s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 10s {color}
| {color:red} hadoop-yarn-client in the
[jira] [Commented] (YARN-4935) TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler
[
https://issues.apache.org/jira/browse/YARN-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234299#comment-15234299
]
Hadoop QA commented on YARN-4935:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 7s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 22s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| | hadoop.yarn.client.api.impl.TestAMRMProxy |
| JDK v1.8.0_77 Timed out junit tests |
org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| | hadoop.yarn.client.api.impl.TestAMRMProxy |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| |
[jira] [Updated] (YARN-4935) TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-4935: --- Attachment: YARN-4935.001.patch > TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler > > > Key: YARN-4935 > URL: https://issues.apache.org/jira/browse/YARN-4935 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-4935.001.patch > > > This test case introduced by YARN-3131 works well on CapacityScheduler but > not on FairScheduler, since CS doesn't allow dynamically create a queue, > while FS supports it. So if you give a random queue name, CS will reject it, > but FS will create a new queue for it by default. > One simple solution is to specific CS in this test case. /cc [~lichangleo]. I > was thinking about creating another test case for FS. But for the code > introduced by YARN-3131, it may be not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234207#comment-15234207 ] Akira AJISAKA commented on YARN-4630: - No problem. Thank you for the update. > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch, YARN-4630.1.patch, YARN-4630.2.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-4630: - Attachment: YARN-4630.2.patch [~ajisakaa] Sorry again. Now I've confirmed that compilation passed on my machine. > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch, YARN-4630.1.patch, YARN-4630.2.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-4630: - Attachment: (was: YARN-4630.2.patch) > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch, YARN-4630.1.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4938) MiniYarnCluster should not request transitionToActive to RM on non-HA environment
[
https://issues.apache.org/jira/browse/YARN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234193#comment-15234193
]
Hudson commented on YARN-4938:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9588 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9588/])
YARN-4938. MiniYarnCluster should not request transitionToActive to RM
(aajisaka: rev 1b78b2ba17cffdd19ae056555dd7f89b7a3eb06a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
> MiniYarnCluster should not request transitionToActive to RM on non-HA
> environment
> -
>
> Key: YARN-4938
> URL: https://issues.apache.org/jira/browse/YARN-4938
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Eric Badger
> Fix For: 2.8.0, 2.7.3
>
> Attachments: MAPREDUCE-6658.001.patch
>
>
> TestMRJobs#testJobWithChangePriority fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobs
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 446.855 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
> testJobWithChangePriority(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 21.477 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobWithChangePriority(TestMRJobs.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[
https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234190#comment-15234190
]
Akira AJISAKA commented on YARN-4630:
-
{code}
+int result = this.getApplicationAttemptId().compareTo(
+other.getApplicationAttemptId())
{code}
Hi [~sarutak], would you add {{;}}?
> Remove useless boxing/unboxing code (Hadoop YARN)
> -
>
> Key: YARN-4630
> URL: https://issues.apache.org/jira/browse/YARN-4630
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
> Attachments: YARN-4630.0.patch, YARN-4630.1.patch, YARN-4630.2.patch
>
>
> There are lots of places where useless boxing/unboxing occur.
> To avoid performance issue, let's remove them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2883) Queuing of container requests in the NM
[
https://issues.apache.org/jira/browse/YARN-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234184#comment-15234184
]
Karthik Kambatla commented on YARN-2883:
Thanks for refactoring the patch, [~kkaranasos]. This definitely feels more
intuitive. I had to review over multiple sittings, so might have missed some.
Here are some comments (both high-level and nits). Will take a closer look on
the next iteration.
# Nit: A few lines introduced in the patch are over 80 chars.
# Nit: When referring to fields in comments, please use tags so
the comments can keep up with code changes.
# Nit: When evaluating boolean conditions in if or while statements, it makes
for better reading if the entire sub-condition is on one line. e.g. I would
prefer the latter. There are other places in the code that could similar minor
adjustments.
{code}
if (resourcesToFreeUp.getPhysicalMemory() <= 0 && resourcesToFreeUp
.getVirtualMemory() <= 0 && resourcesToFreeUp.getCPU() <= 0.0f) {
}
{code}
{code}
if (resourcesToFreeUp.getPhysicalMemory() <= 0 &&
resourcesToFreeUp.getVirtualMemory() <= 0 &&
resourcesToFreeUp.getCPU() <= 0.0f) {
{code}
# We seem to be overriding both {{performContainerStart}} and
{{startContainerInternal}}
## It seems simpler to override only one. If we keep {{startContainerInternal}}
private, the additional changes there can be moved to {{performContainerStart}}
## Alternatively, to keep the changes symmetric with {{stopContainerInternal}},
it might make sense to just override {{startContainerInternal}} and not have a
{{performContainerStart}} method at all.
# {{QueuingContainerManagerImpl}}
## While the abbreviated variable names do make space, they make reading the
code slightly harder. Can we reconsider using full names? I am comfortable with
using alloc for allocated. Also, there seems to be a mismatch in variable names
ending in containers and requests. How about using guaranteedAllocations,
opportunisticAllocations, queuedGuaranteedAllocations and
queuedOpporunisticAllocations?
## Use {{ConcurrentLinkedQueue}} instead of
{{Collections.synchronizedList(LinkedList)}}?
## I still think we should never queue guaranteed containers. Is there a good
reason for queuing them? Since we are killing opportunistic containers anyway,
why not do that synchronously and start the guaranteed containers? The killing
opportunistic containers should likely take a kill context, so we can handle it
appropriately in YARN-1011 where we kill when the utilization goes over a
threshold. Thinking more about this, I wonder if we should just defer to the
kill when over a threshold utilization anyway once an opportunistic container
has started.
## Are we queuing guaranteed containers, so a single call to kill could free up
resources for potentially multiple guaranteed containers?
## Nit: Rename {{removeContainerFromQueues}} to {{removeQueuedContainer}}?
## Does {{startContainersFromQueue}} require {{resourcesAvailable}} flag to be
passed?
## Nit: In {{startPendingContainers}}, the code might read better as follows:
{code}
if (resourcesAvailable) {
startContainersFromQueue(queuedOpportRequests);
}
## {{killOpportContainers}} seems to not kill opportunistic containers if they
started running. Doesn’t this mess with guarantees on the guaranteed
containers? Am I missing something? May be, we should just iterate in reverse
through the queuedOpportunisticContainers and spawned opportunistic containers
and kill as many as required?
## QueuingApplicationEventDispatcher#handle
### Nit: the if condition could use better formatting
### Nit: The comment mentions starting pending containers *if resources
available*, but the code there doesn’t directly reflect it.
### I wonder if we are better off using a separate thread for starting pending
containers periodically, and would wait between attempts. The code here could
just notify that thread? That would likely work better when YARN-1011 wants to
use a queue length > 1.
# {{QueuingContainersMonitorImpl}}
## When queuing containers, depending on whether oversubscription is enabled or
not, shouldn't we be checking utilization vs allocation respectively?
## hasAllocatedResourcesAvailable: rename to hasResourcesAvailable if we are
not differentiating between allocation and utilization.
## hasAllocatedResourcesAvailable should check vmem only when vmem check is
enabled?
## {{resourcesToFreeUp}} takes {{opportContainersToKill}}, but I don't see any
container ever being added here. Am I missing something or do we not need it
and all related fields and method parameters can be removed?
## If we are using utilization to decide whether to start an opportunistic
container, why not just look at the aggregate containers’ utilization? Granted,
we ll be able to make a decision about only one container at a time, but we
don’t have to iterate through container lists
[jira] [Commented] (YARN-4938) MiniYarnCluster should not request transitionToActive to RM on non-HA environment
[
https://issues.apache.org/jira/browse/YARN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234177#comment-15234177
]
Hadoop QA commented on YARN-4938:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 44s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 52s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797375/MAPREDUCE-6658.001.patch
|
| JIRA Issue | YARN-4938 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c5c005fa437b
[jira] [Updated] (YARN-4771) Some containers can be skipped during log aggregation after NM restart
[ https://issues.apache.org/jira/browse/YARN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-4771: -- Target Version/s: 2.7.4 (was: 2.7.3) bq. However it may have issues with very long-running apps that churn a lot of containers, since the container state won't be released until the application completes. This is going to be problematic, impacting NM memory usage. I think the right solution is to decouple log-aggregation state completely from the rest of the container-state, and persist that separately in state-store etc irrespective of container / application state. IAC, I am dropping this off 2.7.3 given that this is a long standing issue and as I'd to proceed with that release now. > Some containers can be skipped during log aggregation after NM restart > -- > > Key: YARN-4771 > URL: https://issues.apache.org/jira/browse/YARN-4771 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: YARN-4771.001.patch, YARN-4771.002.patch > > > A container can be skipped during log aggregation after a work-preserving > nodemanager restart if the following events occur: > # Container completes more than > yarn.nodemanager.duration-to-track-stopped-containers milliseconds before the > restart > # At least one other container completes after the above container and before > the restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4938) MiniYarnCluster should not request transitionToActive to RM on non-HA environment
[
https://issues.apache.org/jira/browse/YARN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234167#comment-15234167
]
Akira AJISAKA commented on YARN-4938:
-
Moved to YARN project because the fix is in MiniYarnCluster.
> MiniYarnCluster should not request transitionToActive to RM on non-HA
> environment
> -
>
> Key: YARN-4938
> URL: https://issues.apache.org/jira/browse/YARN-4938
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6658.001.patch
>
>
> TestMRJobs#testJobWithChangePriority fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobs
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 446.855 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
> testJobWithChangePriority(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 21.477 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobWithChangePriority(TestMRJobs.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4938) MiniYarnCluster should not request transitionToActive to RM on non-HA environment
[
https://issues.apache.org/jira/browse/YARN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-4938:
Target Version/s: 2.7.3
Hadoop Flags: Reviewed
Summary: MiniYarnCluster should not request transitionToActive to
RM on non-HA environment (was: TestMRJobs fails)
> MiniYarnCluster should not request transitionToActive to RM on non-HA
> environment
> -
>
> Key: YARN-4938
> URL: https://issues.apache.org/jira/browse/YARN-4938
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6658.001.patch
>
>
> TestMRJobs#testJobWithChangePriority fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobs
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 446.855 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
> testJobWithChangePriority(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 21.477 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobWithChangePriority(TestMRJobs.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (YARN-4938) TestMRJobs fails
[
https://issues.apache.org/jira/browse/YARN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA moved MAPREDUCE-6658 to YARN-4938:
Component/s: (was: test)
test
Key: YARN-4938 (was: MAPREDUCE-6658)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> TestMRJobs fails
>
>
> Key: YARN-4938
> URL: https://issues.apache.org/jira/browse/YARN-4938
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6658.001.patch
>
>
> TestMRJobs#testJobWithChangePriority fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobs
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 446.855 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
> testJobWithChangePriority(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 21.477 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobWithChangePriority(TestMRJobs.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4311) Removing nodes from include and exclude lists will not remove them from decommissioned nodes list
[ https://issues.apache.org/jira/browse/YARN-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234023#comment-15234023 ] Kuhu Shukla commented on YARN-4311: --- Requesting [~jlowe] for reviews on branch-2.7 version of the patch. > Removing nodes from include and exclude lists will not remove them from > decommissioned nodes list > - > > Key: YARN-4311 > URL: https://issues.apache.org/jira/browse/YARN-4311 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0 > > Attachments: YARN-4311-branch-2.7.001.patch, > YARN-4311-branch-2.7.002.patch, YARN-4311-branch-2.7.003.patch, > YARN-4311-branch-2.7.004.patch, YARN-4311-v1.patch, YARN-4311-v10.patch, > YARN-4311-v11.patch, YARN-4311-v11.patch, YARN-4311-v12.patch, > YARN-4311-v13.patch, YARN-4311-v13.patch, YARN-4311-v14.patch, > YARN-4311-v2.patch, YARN-4311-v3.patch, YARN-4311-v4.patch, > YARN-4311-v5.patch, YARN-4311-v6.patch, YARN-4311-v7.patch, > YARN-4311-v8.patch, YARN-4311-v9.patch > > > In order to fully forget about a node, removing the node from include and > exclude list is not sufficient. The RM lists it under Decomm-ed nodes. The > tricky part that [~jlowe] pointed out was the case when include lists are not > used, in that case we don't want the nodes to fall off if they are not active. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4311) Removing nodes from include and exclude lists will not remove them from decommissioned nodes list
[
https://issues.apache.org/jira/browse/YARN-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234020#comment-15234020
]
Hadoop QA commented on YARN-4311:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 1s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in branch-2.7 has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-tools/hadoop-sls in branch-2.7 has 11 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 13s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 58s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 30s
{color} | {color:red} root: patch generated 11 new + 1187 unchanged - 0 fixed =
1198 total (was 1187) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 9295 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 4m 24s
{color} | {color:red} The patch has 440 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green}
[jira] [Commented] (YARN-4935) TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233982#comment-15233982 ] Yufei Gu commented on YARN-4935: [~lichangleo], thanks for the quick reply and background of YARN-3131. > TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler > > > Key: YARN-4935 > URL: https://issues.apache.org/jira/browse/YARN-4935 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > > This test case introduced by YARN-3131 works well on CapacityScheduler but > not on FairScheduler, since CS doesn't allow dynamically create a queue, > while FS supports it. So if you give a random queue name, CS will reject it, > but FS will create a new queue for it by default. > One simple solution is to specific CS in this test case. /cc [~lichangleo]. I > was thinking about creating another test case for FS. But for the code > introduced by YARN-3131, it may be not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4935) TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-4935: --- Description: This test case introduced by YARN-3131 works well on CapacityScheduler but not on FairScheduler, since CS doesn't allow dynamically create a queue, while FS supports it. So if you give a random queue name, CS will reject it, but FS will create a new queue for it by default. One simple solution is to specific CS in this test case. /cc [~lichangleo]. I was thinking about creating another test case for FS. But for the code introduced by YARN-3131, it may be not necessary. was: This test case introduced by YARN-3131 works well on CapacityScheduler but not on FairScheduler, since CS doesn't allow dynamically create a queue, but FS support it. So if you give a random queue name, CS will reject it, but FS will create a new queue for it by default. One simple solution is to specific CS in this test case. /cc [~lichangleo]. I was thinking about creating another test case for FS. But for the code introduced by YARN-3131, it may be not necessary. > TestYarnClient#testSubmitIncorrectQueue fails with FairScheduler > > > Key: YARN-4935 > URL: https://issues.apache.org/jira/browse/YARN-4935 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > > This test case introduced by YARN-3131 works well on CapacityScheduler but > not on FairScheduler, since CS doesn't allow dynamically create a queue, > while FS supports it. So if you give a random queue name, CS will reject it, > but FS will create a new queue for it by default. > One simple solution is to specific CS in this test case. /cc [~lichangleo]. I > was thinking about creating another test case for FS. But for the code > introduced by YARN-3131, it may be not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4921) Remove deprecated "yarn.timeline-service.hostname" from yarn-default
[
https://issues.apache.org/jira/browse/YARN-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233972#comment-15233972
]
Hadoop QA commented on YARN-4921:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 11s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 39s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s
{color} | {color:green} hadoop-yarn-site in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 55s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-yarn-site in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.yarn.client.api.impl.TestTimelineClientForATS1_5 |
| | hadoop.yarn.client.api.impl.TestTimelineClient |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.client.api.impl.TestTimelineClientForATS1_5 |
| |
[jira] [Commented] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233960#comment-15233960 ] Sunil G commented on YARN-3933: --- Thanks. I feel this can go in independently and for other refactoring we can do in YARN-4809 > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4921) Remove deprecated "yarn.timeline-service.hostname" from yarn-default
[
https://issues.apache.org/jira/browse/YARN-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated YARN-4921:
-
Attachment: YARN-4921.01.patch
YARN-2460 tried to remove the configuration but kept it as it was because for
future use case. {{yarn.timeline-service.hostname}} cannot be configured even
now and other configurations such as {{yarn.timeline-service.address}} that
seemed to depend on {{yarn.timeline-service.hostname}} does not use
{{yarn.timeline-service.hostname}}. So we can change the documentation as well
to reflect actual implementation.
> Remove deprecated "yarn.timeline-service.hostname" from yarn-default
>
>
> Key: YARN-4921
> URL: https://issues.apache.org/jira/browse/YARN-4921
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Li Lu
> Attachments: YARN-4921.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-4921) Remove deprecated "yarn.timeline-service.hostname" from yarn-default
[ https://issues.apache.org/jira/browse/YARN-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki reassigned YARN-4921: Assignee: Kai Sasaki > Remove deprecated "yarn.timeline-service.hostname" from yarn-default > > > Key: YARN-4921 > URL: https://issues.apache.org/jira/browse/YARN-4921 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Li Lu >Assignee: Kai Sasaki > Attachments: YARN-4921.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4794) Deadlock in NMClientImpl
[
https://issues.apache.org/jira/browse/YARN-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233917#comment-15233917
]
Vinod Kumar Vavilapalli commented on YARN-4794:
---
The patch looks fine to me too.
Not related to this patch but I see both startedContainers.putIfAbsent() and
startedContainers.put() in addStartingContainer(), that is duplicated and
unnecesary?
> Deadlock in NMClientImpl
>
>
> Key: YARN-4794
> URL: https://issues.apache.org/jira/browse/YARN-4794
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-4794.1.patch
>
>
> Distributed shell app gets stuck on stopping containers after App completes
> with the following exception
> {code:title = app log}
> 15/12/10 14:52:20 INFO distributedshell.ApplicationMaster: Application
> completed. Stopping running containers
> 15/12/10 14:52:20 WARN ipc.Client: Exception encountered while connecting to
> the server : java.nio.channels.ClosedByInterruptException
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4311) Removing nodes from include and exclude lists will not remove them from decommissioned nodes list
[
https://issues.apache.org/jira/browse/YARN-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated YARN-4311:
--
Attachment: YARN-4311-branch-2.7.004.patch
TestResourceTrackerService#testNodeRemoval passes locally but is flaky since
the wait period was set to the minimum wait time which may get exceeded during
removal. I have added {{waitForNodeRemoval}} that waits for the node to get
removed before asserting. This fix should be added to the test in trunk
although it was not seen in the precommits. I will do so after some review
comments on this.
> Removing nodes from include and exclude lists will not remove them from
> decommissioned nodes list
> -
>
> Key: YARN-4311
> URL: https://issues.apache.org/jira/browse/YARN-4311
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 2.8.0
>
> Attachments: YARN-4311-branch-2.7.001.patch,
> YARN-4311-branch-2.7.002.patch, YARN-4311-branch-2.7.003.patch,
> YARN-4311-branch-2.7.004.patch, YARN-4311-v1.patch, YARN-4311-v10.patch,
> YARN-4311-v11.patch, YARN-4311-v11.patch, YARN-4311-v12.patch,
> YARN-4311-v13.patch, YARN-4311-v13.patch, YARN-4311-v14.patch,
> YARN-4311-v2.patch, YARN-4311-v3.patch, YARN-4311-v4.patch,
> YARN-4311-v5.patch, YARN-4311-v6.patch, YARN-4311-v7.patch,
> YARN-4311-v8.patch, YARN-4311-v9.patch
>
>
> In order to fully forget about a node, removing the node from include and
> exclude list is not sufficient. The RM lists it under Decomm-ed nodes. The
> tricky part that [~jlowe] pointed out was the case when include lists are not
> used, in that case we don't want the nodes to fall off if they are not active.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
