[jira] [Updated] (YARN-1919) Log yarn.resourcemanager.cluster-id is required for HA instead of throwing NPE
[
https://issues.apache.org/jira/browse/YARN-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1919:
---
Summary: Log yarn.resourcemanager.cluster-id is required for HA instead of
throwing NPE (was: NPE in Resource Manager if we don't set
'yarn.resourcemanager.cluster-id' for HA)
> Log yarn.resourcemanager.cluster-id is required for HA instead of throwing NPE
> --
>
> Key: YARN-1919
> URL: https://issues.apache.org/jira/browse/YARN-1919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Devaraj K
>Assignee: Karthik Kambatla
>Priority: Minor
>
> {code:xml}
> 2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (YARN-1919) NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for HA
[
https://issues.apache.org/jira/browse/YARN-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla reopened YARN-1919:
Assignee: Karthik Kambatla
> NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for
> HA
>
>
> Key: YARN-1919
> URL: https://issues.apache.org/jira/browse/YARN-1919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Devaraj K
>Assignee: Karthik Kambatla
>Priority: Minor
>
> {code:xml}
> 2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1919) NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for HA
[
https://issues.apache.org/jira/browse/YARN-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964982#comment-13964982
]
Karthik Kambatla commented on YARN-1919:
Indeed, that makes more sense.
> NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for
> HA
>
>
> Key: YARN-1919
> URL: https://issues.apache.org/jira/browse/YARN-1919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Devaraj K
>Priority: Minor
>
> {code:xml}
> 2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1919) NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for HA
[
https://issues.apache.org/jira/browse/YARN-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964972#comment-13964972
]
Jian He commented on YARN-1919:
---
Does it make sense to print a help message instead of throwing NPE ?
> NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for
> HA
>
>
> Key: YARN-1919
> URL: https://issues.apache.org/jira/browse/YARN-1919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Devaraj K
>Priority: Minor
>
> {code:xml}
> 2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1920) TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows
[
https://issues.apache.org/jira/browse/YARN-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964921#comment-13964921
]
Hadoop QA commented on YARN-1920:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639515/YARN-1920.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3541//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3541//console
This message is automatically generated.
> TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData
> fails in windows
> ---
>
> Key: YARN-1920
> URL: https://issues.apache.org/jira/browse/YARN-1920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Attachments: YARN-1920.txt
>
>
> Though this was only failing in Windows, after debugging, I realized that the
> test fails because we are leaking a file-handle in the history service.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1914) Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows
[
https://issues.apache.org/jira/browse/YARN-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964914#comment-13964914
]
Hadoop QA commented on YARN-1914:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639511/apache-yarn-1914.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3540//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3540//console
This message is automatically generated.
> Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows
>
>
> Key: YARN-1914
> URL: https://issues.apache.org/jira/browse/YARN-1914
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-1914.0.patch, apache-yarn-1914.1.patch
>
>
> The TestFSDownload.testDownloadPublicWithStatCache test in hadoop-yarn-common
> consistently fails on Windows environments.
> The root cause is that the test checks for execute permission for all users
> on every ancestor of the target directory. In windows, by default, group
> "Everyone" has no permissions on any directory in the install drive. It's
> unreasonable to expect this test to pass and we should skip it on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-996) REST API support for node resource configuration
[ https://issues.apache.org/jira/browse/YARN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-996: Assignee: Kenji Kikushima (was: Junping Du) > REST API support for node resource configuration > > > Key: YARN-996 > URL: https://issues.apache.org/jira/browse/YARN-996 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Kenji Kikushima > Attachments: YARN-996-sample.patch > > > Besides admin protocol and CLI, REST API should also be supported for node > resource configuration -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (YARN-996) REST API support for node resource configuration
[ https://issues.apache.org/jira/browse/YARN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reassigned YARN-996: --- Assignee: Junping Du > REST API support for node resource configuration > > > Key: YARN-996 > URL: https://issues.apache.org/jira/browse/YARN-996 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-996-sample.patch > > > Besides admin protocol and CLI, REST API should also be supported for node > resource configuration -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-996) REST API support for node resource configuration
[ https://issues.apache.org/jira/browse/YARN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964906#comment-13964906 ] Junping Du commented on YARN-996: - Hi [~kj-ki], Thanks for delivering a patch for this. I already assign this JIRA to you and will review your patch later. Thx! > REST API support for node resource configuration > > > Key: YARN-996 > URL: https://issues.apache.org/jira/browse/YARN-996 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Kenji Kikushima > Attachments: YARN-996-sample.patch > > > Besides admin protocol and CLI, REST API should also be supported for node > resource configuration -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1920) TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows
[ https://issues.apache.org/jira/browse/YARN-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1920: -- Attachment: YARN-1920.txt Here's a patch that fixes the leak and passes the test on Windows. > TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData > fails in windows > --- > > Key: YARN-1920 > URL: https://issues.apache.org/jira/browse/YARN-1920 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-1920.txt > > > Though this was only failing in Windows, after debugging, I realized that the > test fails because we are leaking a file-handle in the history service. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1920) TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows
Vinod Kumar Vavilapalli created YARN-1920: - Summary: TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows Key: YARN-1920 URL: https://issues.apache.org/jira/browse/YARN-1920 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Though this was only failing in Windows, after debugging, I realized that the test fails because we are leaking a file-handle in the history service. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1914) Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows
[ https://issues.apache.org/jira/browse/YARN-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-1914: Attachment: apache-yarn-1914.1.patch Attached new patch fixing root cause in FSDownload.java. The patch should fail the HadoopQA because it fixes a currently failing test on Windows. Thank you [~bikassaha] and [~ivanmi]. > Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows > > > Key: YARN-1914 > URL: https://issues.apache.org/jira/browse/YARN-1914 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1914.0.patch, apache-yarn-1914.1.patch > > > The TestFSDownload.testDownloadPublicWithStatCache test in hadoop-yarn-common > consistently fails on Windows environments. > The root cause is that the test checks for execute permission for all users > on every ancestor of the target directory. In windows, by default, group > "Everyone" has no permissions on any directory in the install drive. It's > unreasonable to expect this test to pass and we should skip it on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1910) TestAMRMTokens fails on windows
[ https://issues.apache.org/jira/browse/YARN-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964887#comment-13964887 ] Hudson commented on YARN-1910: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5485 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5485/]) YARN-1910. Fixed a race condition in TestAMRMTokens that causes the test to fail more often on Windows. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586192) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestAMRMTokens.java > TestAMRMTokens fails on windows > --- > > Key: YARN-1910 > URL: https://issues.apache.org/jira/browse/YARN-1910 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.4.0 > > Attachments: YARN-1910.1.patch, YARN-1910.2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1910) TestAMRMTokens fails on windows
[ https://issues.apache.org/jira/browse/YARN-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964882#comment-13964882 ] Vinod Kumar Vavilapalli commented on YARN-1910: --- Alright, looks good to me too. +1. The test fails without the patch and passes with on Windows. It passes on Linux before and after the patch. Checking this in. > TestAMRMTokens fails on windows > --- > > Key: YARN-1910 > URL: https://issues.apache.org/jira/browse/YARN-1910 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.4.0 > > Attachments: YARN-1910.1.patch, YARN-1910.2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-996) REST API support for node resource configuration
[
https://issues.apache.org/jira/browse/YARN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenji Kikushima updated YARN-996:
-
Attachment: YARN-996-sample.patch
Hi, I'm trying to implement REST API. Here is a sample patch.
- We can get Resource infomation from getting "/ws/v1/cluster/nodes/{nodeid}"
- We can update Resources. For example, putting
"/ws/v1/cluster/nodes/{nodeid}/vcores/10" means updates to 10 vcores
- At present, Resource Manager REST API only
- This patch needs YARN-1911.patch in YARN-1911 to avoid NPE
Please comment if you have interest. Thanks.
> REST API support for node resource configuration
>
>
> Key: YARN-996
> URL: https://issues.apache.org/jira/browse/YARN-996
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, scheduler
>Reporter: Junping Du
> Attachments: YARN-996-sample.patch
>
>
> Besides admin protocol and CLI, REST API should also be supported for node
> resource configuration
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (YARN-1919) NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for HA
[
https://issues.apache.org/jira/browse/YARN-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla resolved YARN-1919.
Resolution: Not a Problem
I understand this is an inconvenience, but this is by design. One of those
bug-or-feature cases. See
https://issues.apache.org/jira/browse/YARN-1029?focusedCommentId=13861990&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13861990
> NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for
> HA
>
>
> Key: YARN-1919
> URL: https://issues.apache.org/jira/browse/YARN-1919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Devaraj K
>Priority: Minor
>
> {code:xml}
> 2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1914) Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows
[ https://issues.apache.org/jira/browse/YARN-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964326#comment-13964326 ] Ivan Mitic commented on YARN-1914: -- Thanks Bikas for looping us in. Varun, HADOOP-8731 might be able to help. > Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows > > > Key: YARN-1914 > URL: https://issues.apache.org/jira/browse/YARN-1914 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1914.0.patch > > > The TestFSDownload.testDownloadPublicWithStatCache test in hadoop-yarn-common > consistently fails on Windows environments. > The root cause is that the test checks for execute permission for all users > on every ancestor of the target directory. In windows, by default, group > "Everyone" has no permissions on any directory in the install drive. It's > unreasonable to expect this test to pass and we should skip it on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1338) Recover localized resource cache state upon nodemanager restart
[
https://issues.apache.org/jira/browse/YARN-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964313#comment-13964313
]
Hadoop QA commented on YARN-1338:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639419/YARN-1338v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3539//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3539//console
This message is automatically generated.
> Recover localized resource cache state upon nodemanager restart
> ---
>
> Key: YARN-1338
> URL: https://issues.apache.org/jira/browse/YARN-1338
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.3.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-1338.patch, YARN-1338v2.patch
>
>
> Today when node manager restarts we clean up all the distributed cache files
> from disk. This is definitely not ideal from 2 aspects.
> * For work preserving restart we definitely want them as running containers
> are using them
> * For even non work preserving restart this will be useful in the sense that
> we don't have to download them again if needed by future tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1907) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails
[ https://issues.apache.org/jira/browse/YARN-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964294#comment-13964294 ] Hudson commented on YARN-1907: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5475 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5475/]) YARN-1907. TestRMApplicationHistoryWriter#testRMWritingMassiveHistory intermittently fails. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585992) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ahs/TestRMApplicationHistoryWriter.java > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails > - > > Key: YARN-1907 > URL: https://issues.apache.org/jira/browse/YARN-1907 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6195.patch > > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1907) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails
[ https://issues.apache.org/jira/browse/YARN-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964284#comment-13964284 ] Kihwal Lee commented on YARN-1907: -- +1 for the change. bq. However, is it a better code practice to loop until all the containers are cleaned up (removing the 200 round bounds), and set a suitable timeout for this test case? The unit tests do not always run in a controlled environment, so timing can vary widely. Tight timeout or timing-based performance checks are not recommended in unit tests. > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails > - > > Key: YARN-1907 > URL: https://issues.apache.org/jira/browse/YARN-1907 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-6195.patch > > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1338) Recover localized resource cache state upon nodemanager restart
[ https://issues.apache.org/jira/browse/YARN-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-1338: - Attachment: YARN-1338v2.patch Updating patch after YARN-1757 and other recent changes on trunk. > Recover localized resource cache state upon nodemanager restart > --- > > Key: YARN-1338 > URL: https://issues.apache.org/jira/browse/YARN-1338 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-1338.patch, YARN-1338v2.patch > > > Today when node manager restarts we clean up all the distributed cache files > from disk. This is definitely not ideal from 2 aspects. > * For work preserving restart we definitely want them as running containers > are using them > * For even non work preserving restart this will be useful in the sense that > we don't have to download them again if needed by future tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1912) ResourceLocalizer started without any jvm memory control
[ https://issues.apache.org/jira/browse/YARN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964198#comment-13964198 ] Nathan Roberts commented on YARN-1912: -- Ah. Thanks Stanley for the pointer and clarification. > ResourceLocalizer started without any jvm memory control > > > Key: YARN-1912 > URL: https://issues.apache.org/jira/browse/YARN-1912 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: stanley shi > > In the LinuxContainerExecutor.java#startLocalizer, it does not specify any > "-Xmx" configurations in the command, this caused the ResourceLocalizer to be > started with default memory setting. > In an server-level hardware, it will use 25% of the system memory as the max > heap size, this will cause memory issue in some cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1908) Distributed shell with custom script has permission error.
[
https://issues.apache.org/jira/browse/YARN-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964168#comment-13964168
]
Hudson commented on YARN-1908:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1726 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1726/])
YARN-1908. Fixed DistributedShell to not fail in secure clusters. Contributed
by Vinod Kumar Vavilapalli and Jian He. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585849)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Distributed shell with custom script has permission error.
> --
>
> Key: YARN-1908
> URL: https://issues.apache.org/jira/browse/YARN-1908
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.4.0
>Reporter: Tassapol Athiapinya
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 2.4.1
>
> Attachments: YARN-1908.1.patch, YARN-1908.2.patch, YARN-1908.3.patch,
> YARN-1908.4.patch
>
>
> Create test1.sh having "pwd".
> Run this command as user1:
> hadoop jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -shell_script test1.sh
> NM is run by yarn user. An exception is thrown because yarn user has no
> permissions on custom script in hdfs path. The custom script is created with
> distributed shell app.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=yarn, access=WRITE,
> inode="/user/user1/DistributedShell/70":user1:user1:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1757) NM Recovery. Auxiliary service support.
[ https://issues.apache.org/jira/browse/YARN-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964164#comment-13964164 ] Hudson commented on YARN-1757: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1726/]) YARN-1757. NM Recovery. Auxiliary service support. (Jason Lowe via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585783) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/AuxiliaryService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java > NM Recovery. Auxiliary service support. > --- > > Key: YARN-1757 > URL: https://issues.apache.org/jira/browse/YARN-1757 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.5.0 > > Attachments: YARN-1757-v2.patch, YARN-1757.patch, YARN-1757.patch > > > There needs to be a mechanism for communicating to auxiliary services whether > nodemanager recovery is enabled and where they should store their state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1784) TestContainerAllocation assumes CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964169#comment-13964169 ] Hudson commented on YARN-1784: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1726/]) YARN-1784. TestContainerAllocation assumes CapacityScheduler. (Robert Kanter via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585894) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java > TestContainerAllocation assumes CapacityScheduler > - > > Key: YARN-1784 > URL: https://issues.apache.org/jira/browse/YARN-1784 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Robert Kanter >Priority: Minor > Fix For: 2.5.0 > > Attachments: YARN-1784.patch, YARN-1784.patch > > > TestContainerAllocation assumes CapacityScheduler -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1784) TestContainerAllocation assumes CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964139#comment-13964139 ] Hudson commented on YARN-1784: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1752 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1752/]) YARN-1784. TestContainerAllocation assumes CapacityScheduler. (Robert Kanter via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585894) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java > TestContainerAllocation assumes CapacityScheduler > - > > Key: YARN-1784 > URL: https://issues.apache.org/jira/browse/YARN-1784 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Robert Kanter >Priority: Minor > Fix For: 2.5.0 > > Attachments: YARN-1784.patch, YARN-1784.patch > > > TestContainerAllocation assumes CapacityScheduler -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1757) NM Recovery. Auxiliary service support.
[ https://issues.apache.org/jira/browse/YARN-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964135#comment-13964135 ] Hudson commented on YARN-1757: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1752 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1752/]) YARN-1757. NM Recovery. Auxiliary service support. (Jason Lowe via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585783) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/AuxiliaryService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java > NM Recovery. Auxiliary service support. > --- > > Key: YARN-1757 > URL: https://issues.apache.org/jira/browse/YARN-1757 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.5.0 > > Attachments: YARN-1757-v2.patch, YARN-1757.patch, YARN-1757.patch > > > There needs to be a mechanism for communicating to auxiliary services whether > nodemanager recovery is enabled and where they should store their state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1908) Distributed shell with custom script has permission error.
[
https://issues.apache.org/jira/browse/YARN-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964138#comment-13964138
]
Hudson commented on YARN-1908:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1752 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1752/])
YARN-1908. Fixed DistributedShell to not fail in secure clusters. Contributed
by Vinod Kumar Vavilapalli and Jian He. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585849)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Distributed shell with custom script has permission error.
> --
>
> Key: YARN-1908
> URL: https://issues.apache.org/jira/browse/YARN-1908
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.4.0
>Reporter: Tassapol Athiapinya
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 2.4.1
>
> Attachments: YARN-1908.1.patch, YARN-1908.2.patch, YARN-1908.3.patch,
> YARN-1908.4.patch
>
>
> Create test1.sh having "pwd".
> Run this command as user1:
> hadoop jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -shell_script test1.sh
> NM is run by yarn user. An exception is thrown because yarn user has no
> permissions on custom script in hdfs path. The custom script is created with
> distributed shell app.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=yarn, access=WRITE,
> inode="/user/user1/DistributedShell/70":user1:user1:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1784) TestContainerAllocation assumes CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964034#comment-13964034 ] Hudson commented on YARN-1784: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #534 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/534/]) YARN-1784. TestContainerAllocation assumes CapacityScheduler. (Robert Kanter via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585894) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java > TestContainerAllocation assumes CapacityScheduler > - > > Key: YARN-1784 > URL: https://issues.apache.org/jira/browse/YARN-1784 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Robert Kanter >Priority: Minor > Fix For: 2.5.0 > > Attachments: YARN-1784.patch, YARN-1784.patch > > > TestContainerAllocation assumes CapacityScheduler -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1757) NM Recovery. Auxiliary service support.
[ https://issues.apache.org/jira/browse/YARN-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964031#comment-13964031 ] Hudson commented on YARN-1757: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #534 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/534/]) YARN-1757. NM Recovery. Auxiliary service support. (Jason Lowe via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585783) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/AuxiliaryService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java > NM Recovery. Auxiliary service support. > --- > > Key: YARN-1757 > URL: https://issues.apache.org/jira/browse/YARN-1757 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.5.0 > > Attachments: YARN-1757-v2.patch, YARN-1757.patch, YARN-1757.patch > > > There needs to be a mechanism for communicating to auxiliary services whether > nodemanager recovery is enabled and where they should store their state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1908) Distributed shell with custom script has permission error.
[
https://issues.apache.org/jira/browse/YARN-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964033#comment-13964033
]
Hudson commented on YARN-1908:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #534 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/534/])
YARN-1908. Fixed DistributedShell to not fail in secure clusters. Contributed
by Vinod Kumar Vavilapalli and Jian He. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585849)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Distributed shell with custom script has permission error.
> --
>
> Key: YARN-1908
> URL: https://issues.apache.org/jira/browse/YARN-1908
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.4.0
>Reporter: Tassapol Athiapinya
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 2.4.1
>
> Attachments: YARN-1908.1.patch, YARN-1908.2.patch, YARN-1908.3.patch,
> YARN-1908.4.patch
>
>
> Create test1.sh having "pwd".
> Run this command as user1:
> hadoop jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
> -shell_script test1.sh
> NM is run by yarn user. An exception is thrown because yarn user has no
> permissions on custom script in hdfs path. The custom script is created with
> distributed shell app.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=yarn, access=WRITE,
> inode="/user/user1/DistributedShell/70":user1:user1:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (YARN-1919) NPE in Resource Manager if we don't set 'yarn.resourcemanager.cluster-id' for HA
Devaraj K created YARN-1919:
---
Summary: NPE in Resource Manager if we don't set
'yarn.resourcemanager.cluster-id' for HA
Key: YARN-1919
URL: https://issues.apache.org/jira/browse/YARN-1919
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.3.0
Reporter: Devaraj K
Priority: Minor
{code:xml}
2014-04-09 16:14:16,392 WARN org.apache.hadoop.service.AbstractService: When
stopping the service
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService :
java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.serviceStop(EmbeddedElectorService.java:108)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceInit(AdminService.java:122)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:232)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1038)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (YARN-1918) Typo in description and error message for 'yarn.resourcemanager.cluster-id'
Devaraj K created YARN-1918:
---
Summary: Typo in description and error message for
'yarn.resourcemanager.cluster-id'
Key: YARN-1918
URL: https://issues.apache.org/jira/browse/YARN-1918
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Devaraj K
Priority: Trivial
1. In yarn-default.xml
{code:xml}
Name of the cluster. In a HA setting,
this is used to ensure the RM participates in leader
election fo this cluster and ensures it does not affect
other clusters
yarn.resourcemanager.cluster-id
{code}
Here the line 'election fo this cluster and ensures it does not affect' should
be replaced with 'election for this cluster and ensures it does not affect'.
2.
{code:xml}
org.apache.hadoop.HadoopIllegalArgumentException: Configuration doesn't
specifyyarn.resourcemanager.cluster-id
at
org.apache.hadoop.yarn.conf.YarnConfiguration.getClusterId(YarnConfiguration.java:1336)
{code}
In the above exception message, it is missing a space between message and
configuration name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1917) Add "waitForCompletion" interface to YarnClient
[ https://issues.apache.org/jira/browse/YARN-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963959#comment-13963959 ] Wangda Tan commented on YARN-1917: -- My basic thinking is we can leverage existing implementation in UnmanagedAMLauncher.monitorApplication, it should be enough for many use cases. The method could be, YarnClient.waitForCompletion(ApplicationId appId). Appreciate your comments :) > Add "waitForCompletion" interface to YarnClient > --- > > Key: YARN-1917 > URL: https://issues.apache.org/jira/browse/YARN-1917 > Project: Hadoop YARN > Issue Type: New Feature > Components: client >Affects Versions: 2.4.0 >Reporter: Wangda Tan > > Currently, YARN dosen't have this method. Users needs to write > implementations like UnmanagedAMLauncher.monitorApplication or > mapreduce.Job.monitorAndPrintJob on their own. This feature should be helpful > to end users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1917) Add "waitForCompletion" interface to YarnClient
[ https://issues.apache.org/jira/browse/YARN-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-1917: - Priority: Minor (was: Major) > Add "waitForCompletion" interface to YarnClient > --- > > Key: YARN-1917 > URL: https://issues.apache.org/jira/browse/YARN-1917 > Project: Hadoop YARN > Issue Type: New Feature > Components: client >Affects Versions: 2.4.0 >Reporter: Wangda Tan >Priority: Minor > > Currently, YARN dosen't have this method. Users needs to write > implementations like UnmanagedAMLauncher.monitorApplication or > mapreduce.Job.monitorAndPrintJob on their own. This feature should be helpful > to end users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1917) Add "waitForCompletion" interface to YarnClient
Wangda Tan created YARN-1917: Summary: Add "waitForCompletion" interface to YarnClient Key: YARN-1917 URL: https://issues.apache.org/jira/browse/YARN-1917 Project: Hadoop YARN Issue Type: New Feature Components: client Affects Versions: 2.4.0 Reporter: Wangda Tan Currently, YARN dosen't have this method. Users needs to write implementations like UnmanagedAMLauncher.monitorApplication or mapreduce.Job.monitorAndPrintJob on their own. This feature should be helpful to end users. -- This message was sent by Atlassian JIRA (v6.2#6252)
