[jira] [Created] (YARN-1938) Kerberos authentication for the timeline server
Zhijie Shen created YARN-1938: - Summary: Kerberos authentication for the timeline server Key: YARN-1938 URL: https://issues.apache.org/jira/browse/YARN-1938 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1937) Access control of per-framework data
Zhijie Shen created YARN-1937: - Summary: Access control of per-framework data Key: YARN-1937 URL: https://issues.apache.org/jira/browse/YARN-1937 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1936) Secured timeline client
Zhijie Shen created YARN-1936: - Summary: Secured timeline client Key: YARN-1936 URL: https://issues.apache.org/jira/browse/YARN-1936 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-996) REST API support for node resource configuration
[ https://issues.apache.org/jira/browse/YARN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968123#comment-13968123 ] Kenji Kikushima commented on YARN-996: -- [~tgraves], thanks for your comment. Certainly, we should add test for admin acls, but updateNodeResource isn't protected. I think we should check acls in updateNodeResource. So, I searched related JIRA, but I couldn't find. Can I create a JIRA about this issue? [~djp], please let us know what do you think. > REST API support for node resource configuration > > > Key: YARN-996 > URL: https://issues.apache.org/jira/browse/YARN-996 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Kenji Kikushima > Attachments: YARN-996-sample.patch > > > Besides admin protocol and CLI, REST API should also be supported for node > resource configuration -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1935) Security for timeline server
[ https://issues.apache.org/jira/browse/YARN-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1935: -- Summary: Security for timeline server (was: Security for timeline service) > Security for timeline server > > > Key: YARN-1935 > URL: https://issues.apache.org/jira/browse/YARN-1935 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Arun C Murthy >Assignee: Zhijie Shen > > Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1935) Security for timeline service
[ https://issues.apache.org/jira/browse/YARN-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1935: -- Issue Type: New Feature (was: Sub-task) Parent: (was: YARN-1530) > Security for timeline service > - > > Key: YARN-1935 > URL: https://issues.apache.org/jira/browse/YARN-1935 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Arun C Murthy >Assignee: Zhijie Shen > > Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1935) Security for timeline service
[ https://issues.apache.org/jira/browse/YARN-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968121#comment-13968121 ] Zhijie Shen commented on YARN-1935: --- I'm going to take care of the security issues of the timeline server > Security for timeline service > - > > Key: YARN-1935 > URL: https://issues.apache.org/jira/browse/YARN-1935 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Zhijie Shen > > Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (YARN-1935) Security for timeline service
[ https://issues.apache.org/jira/browse/YARN-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reassigned YARN-1935: - Assignee: Zhijie Shen > Security for timeline service > - > > Key: YARN-1935 > URL: https://issues.apache.org/jira/browse/YARN-1935 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Zhijie Shen > > Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1932) Javascript injection on the job status page
[ https://issues.apache.org/jira/browse/YARN-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-1932: Priority: Blocker (was: Critical) > Javascript injection on the job status page > --- > > Key: YARN-1932 > URL: https://issues.apache.org/jira/browse/YARN-1932 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 0.23.9, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai >Priority: Blocker > Attachments: YARN-1932.patch > > > Scripts can be injected into the job status page as the diagnostics field is > not sanitized. Whatever string you set there will show up to the jobs page as > it is ... ie. if you put any script commands, they will be executed in the > browser of the user who is opening the page. > We need escaping the diagnostic string in order to not run the scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968100#comment-13968100
]
Hadoop QA commented on YARN-1864:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640022/YARN-1864-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3565//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3565//console
This message is automatically generated.
> Fair Scheduler Dynamic Hierarchical User Queues
> ---
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
>Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
[ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated YARN-1864: - Attachment: YARN-1864-v2.txt Patch YARN-1864-v2.txt has the following features : 1. Hierarchical User queue implemented using nested rules. With this feature, we can create user queues with primary group,secondary group,default,specified rules etc by just nesting that rule. Please see TestQueuePlacementPolicy.java in the patch for examples/use cases. It is extensible,so any new rule in the future can get user queue support with hierarchicalUserQueue rule without much code changes. 2. A way to configure parent queues in the fair scheduler alloc xml without specifying any leaf queues through a new attribute called 'type' in the queue element. //other queue properties Please see TestFairScheduler#testHierarchicalUserQueue for an example. 3. 'default' queue placement rule can take a default queue as input. If no default queue specified, we fall back to root.default queue. I've added a bunch of unit tests and also manually tested hierarchicalUserQueue rule with combinations of other rules. Also verified that preemption involving hierarchical user queues works. > Fair Scheduler Dynamic Hierarchical User Queues > --- > > Key: YARN-1864 > URL: https://issues.apache.org/jira/browse/YARN-1864 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Reporter: Ashwin Shankar > Labels: scheduler > Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt > > > In Fair Scheduler, we want to be able to create user queues under any parent > queue in the hierarchy. For eg. Say user1 submits a job to a parent queue > called root.allUserQueues, we want be able to create a new queue called > root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted > by this user to root.allUserQueues will be run in this newly created > root.allUserQueues.user1. > This is very similar to the 'user-as-default' feature in Fair Scheduler which > creates user queues under root queue. But we want the ability to create user > queues under ANY parent queue. > Why do we want this ? > 1. Preemption : these dynamically created user queues can preempt each other > if its fair share is not met. So there is fairness among users. > User queues can also preempt other non-user leaf queue as well if below fair > share. > 2. Allocation to user queues : we want all the user queries(adhoc) to consume > only a fraction of resources in the shared cluster. By creating this > feature,we could do that by giving a fair share to the parent user queue > which is then redistributed to all the dynamically created user queues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1870) FileInputStream is not closed in ProcfsBasedProcessTree#constructProcessSMAPInfo()
[
https://issues.apache.org/jira/browse/YARN-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated YARN-1870:
--
Component/s: resourcemanager
> FileInputStream is not closed in
> ProcfsBasedProcessTree#constructProcessSMAPInfo()
> --
>
> Key: YARN-1870
> URL: https://issues.apache.org/jira/browse/YARN-1870
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Ted Yu
>Assignee: Fengdong Yu
>Priority: Minor
> Fix For: 2.4.1
>
> Attachments: YARN-1870.patch
>
>
> {code}
> List lines = IOUtils.readLines(new FileInputStream(file));
> {code}
> FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1870) FileInputStream is not closed in ProcfsBasedProcessTree#constructProcessSMAPInfo()
[
https://issues.apache.org/jira/browse/YARN-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated YARN-1870:
--
Affects Version/s: 2.4.0
> FileInputStream is not closed in
> ProcfsBasedProcessTree#constructProcessSMAPInfo()
> --
>
> Key: YARN-1870
> URL: https://issues.apache.org/jira/browse/YARN-1870
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Ted Yu
>Assignee: Fengdong Yu
>Priority: Minor
> Fix For: 2.4.1
>
> Attachments: YARN-1870.patch
>
>
> {code}
> List lines = IOUtils.readLines(new FileInputStream(file));
> {code}
> FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1870) FileInputStream is not closed in ProcfsBasedProcessTree#constructProcessSMAPInfo()
[
https://issues.apache.org/jira/browse/YARN-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated YARN-1870:
--
Fix Version/s: 2.4.1
> FileInputStream is not closed in
> ProcfsBasedProcessTree#constructProcessSMAPInfo()
> --
>
> Key: YARN-1870
> URL: https://issues.apache.org/jira/browse/YARN-1870
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Ted Yu
>Assignee: Fengdong Yu
>Priority: Minor
> Fix For: 2.4.1
>
> Attachments: YARN-1870.patch
>
>
> {code}
> List lines = IOUtils.readLines(new FileInputStream(file));
> {code}
> FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1870) FileInputStream is not closed in ProcfsBasedProcessTree#constructProcessSMAPInfo()
[
https://issues.apache.org/jira/browse/YARN-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968030#comment-13968030
]
Fengdong Yu commented on YARN-1870:
---
who can commit this? Thanks.
> FileInputStream is not closed in
> ProcfsBasedProcessTree#constructProcessSMAPInfo()
> --
>
> Key: YARN-1870
> URL: https://issues.apache.org/jira/browse/YARN-1870
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Fengdong Yu
>Priority: Minor
> Attachments: YARN-1870.patch
>
>
> {code}
> List lines = IOUtils.readLines(new FileInputStream(file));
> {code}
> FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1935) Security for timeline service
[ https://issues.apache.org/jira/browse/YARN-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1935: -- Summary: Security for timeline service (was: Security for ATS) > Security for timeline service > - > > Key: YARN-1935 > URL: https://issues.apache.org/jira/browse/YARN-1935 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun C Murthy > > Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1935) Security for ATS
Arun C Murthy created YARN-1935: --- Summary: Security for ATS Key: YARN-1935 URL: https://issues.apache.org/jira/browse/YARN-1935 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Jira to track work to secure the ATS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1928) TestAMRMRPCNodeUpdates fails ocassionally
[
https://issues.apache.org/jira/browse/YARN-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967975#comment-13967975
]
Hudson commented on YARN-1928:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5512 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5512/])
YARN-1928. Fixed a race condition in TestAMRMRPCNodeUpdates which caused it to
fail occassionally. Contributed by Zhijie Shen. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1587114)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java
> TestAMRMRPCNodeUpdates fails ocassionally
> -
>
> Key: YARN-1928
> URL: https://issues.apache.org/jira/browse/YARN-1928
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.4.1
>
> Attachments: YARN-1928.1.patch
>
>
> {code}
> junit.framework.AssertionFailedError: expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates.testAMRMUnusableNodes(TestAMRMRPCNodeUpdates.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1928) TestAMRMRPCNodeUpdates fails ocassionally
[
https://issues.apache.org/jira/browse/YARN-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967969#comment-13967969
]
Vinod Kumar Vavilapalli commented on YARN-1928:
---
The reasoning and the patch look good to me. +1. Will check this in.
One side note is that on register, AMs are not getting notified of all nodes in
the system. That will be taken care of at YARN-435, at which point this test
will need to change.
> TestAMRMRPCNodeUpdates fails ocassionally
> -
>
> Key: YARN-1928
> URL: https://issues.apache.org/jira/browse/YARN-1928
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-1928.1.patch
>
>
> {code}
> junit.framework.AssertionFailedError: expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates.testAMRMUnusableNodes(TestAMRMRPCNodeUpdates.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (YARN-1722) AMRMProtocol should have a way of getting all the nodes in the cluster
[ https://issues.apache.org/jira/browse/YARN-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-1722. --- Resolution: Duplicate Fix Version/s: (was: 2.5.0) Dup of YARN-435. YARN-435 didn't make progress not because of any technical reasons. We are going to work on it soon. Closing this as dup. > AMRMProtocol should have a way of getting all the nodes in the cluster > -- > > Key: YARN-1722 > URL: https://issues.apache.org/jira/browse/YARN-1722 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Bikas Saha > > There is no way for an AM to find out the names of all the nodes in the > cluster via the AMRMProtocol. An AM can only at best ask for containers at * > location. The only way to get that information is via the ClientRMProtocol > but that is secured by Kerberos or RMDelegationToken while the AM has an > AMRMToken. This is a pretty important piece of missing functionality. There > are other jiras opened about getting cluster topology etc. but they havent > been addressed due to a clear definition of cluster topology perhaps. Adding > a means to at least get the node information would be a good first step. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1933) TestAMRestart and TestNodeHealthService failing sometimes on Windows
[ https://issues.apache.org/jira/browse/YARN-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967956#comment-13967956 ] Hudson commented on YARN-1933: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5511 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5511/]) YARN-1933. Fixed test issues with TestAMRestart and TestNodeHealthService. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1587104) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeHealthService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > TestAMRestart and TestNodeHealthService failing sometimes on Windows > > > Key: YARN-1933 > URL: https://issues.apache.org/jira/browse/YARN-1933 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.4.1 > > Attachments: YARN-1933.1.patch, YARN-1933.2.patch > > > TestNodeHealthService failures: > testNodeHealthScript(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) > Time elapsed: 1.405 sec <<< ERROR! > java.io.FileNotFoundException: > C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd > (The process cannot access the file because it is being used by another > process) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(FileOutputStream.java:221) > at java.io.FileOutputStream.(FileOutputStream.java:171) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScript(TestNodeHealthService.java:154) > testNodeHealthScriptShouldRun(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) > Time elapsed: 0 sec <<< ERROR! > java.io.FileNotFoundException: > C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd > (Access is denied) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(FileOutputStream.java:221) > at java.io.FileOutputStream.(FileOutputStream.java:171) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScriptShouldRun(TestNodeHealthService.java:103) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1933) TestAMRestart and TestNodeHealthService failing sometimes on Windows
[
https://issues.apache.org/jira/browse/YARN-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967944#comment-13967944
]
Hadoop QA commented on YARN-1933:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639991/YARN-1933.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3564//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3564//console
This message is automatically generated.
> TestAMRestart and TestNodeHealthService failing sometimes on Windows
>
>
> Key: YARN-1933
> URL: https://issues.apache.org/jira/browse/YARN-1933
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-1933.1.patch, YARN-1933.2.patch
>
>
> TestNodeHealthService failures:
> testNodeHealthScript(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService)
> Time elapsed: 1.405 sec <<< ERROR!
> java.io.FileNotFoundException:
> C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd
> (The process cannot access the file because it is being used by another
> process)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScript(TestNodeHealthService.java:154)
> testNodeHealthScriptShouldRun(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService)
> Time elapsed: 0 sec <<< ERROR!
> java.io.FileNotFoundException:
> C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd
> (Access is denied)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScriptShouldRun(TestNodeHealthService.java:103)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1933) TestAMRestart and TestNodeHealthService failing sometimes on Windows
[ https://issues.apache.org/jira/browse/YARN-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1933: -- Attachment: YARN-1933.2.patch Same patch with the import issues fixed. Will check this in if Jenkins says okay.. > TestAMRestart and TestNodeHealthService failing sometimes on Windows > > > Key: YARN-1933 > URL: https://issues.apache.org/jira/browse/YARN-1933 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-1933.1.patch, YARN-1933.2.patch > > > TestNodeHealthService failures: > testNodeHealthScript(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) > Time elapsed: 1.405 sec <<< ERROR! > java.io.FileNotFoundException: > C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd > (The process cannot access the file because it is being used by another > process) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(FileOutputStream.java:221) > at java.io.FileOutputStream.(FileOutputStream.java:171) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScript(TestNodeHealthService.java:154) > testNodeHealthScriptShouldRun(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) > Time elapsed: 0 sec <<< ERROR! > java.io.FileNotFoundException: > C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd > (Access is denied) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(FileOutputStream.java:221) > at java.io.FileOutputStream.(FileOutputStream.java:171) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScriptShouldRun(TestNodeHealthService.java:103) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1701) Improve default paths of timeline store and generic history store
[
https://issues.apache.org/jira/browse/YARN-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated YARN-1701:
Target Version/s: 2.4.1
> Improve default paths of timeline store and generic history store
> -
>
> Key: YARN-1701
> URL: https://issues.apache.org/jira/browse/YARN-1701
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: YARN-1701.v01.patch, YARN-1701.v02.patch
>
>
> When I enable AHS via yarn.ahs.enabled, the app history is still not visible
> in AHS webUI. This is due to NullApplicationHistoryStore as
> yarn.resourcemanager.history-writer.class. It would be good to have just one
> key to enable basic functionality.
> yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is
> local file system location. However, FileSystemApplicationHistoryStore uses
> DFS by default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1701) Improve default paths of timeline store and generic history store
[
https://issues.apache.org/jira/browse/YARN-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated YARN-1701:
Priority: Blocker (was: Major)
> Improve default paths of timeline store and generic history store
> -
>
> Key: YARN-1701
> URL: https://issues.apache.org/jira/browse/YARN-1701
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: YARN-1701.v01.patch, YARN-1701.v02.patch
>
>
> When I enable AHS via yarn.ahs.enabled, the app history is still not visible
> in AHS webUI. This is due to NullApplicationHistoryStore as
> yarn.resourcemanager.history-writer.class. It would be good to have just one
> key to enable basic functionality.
> yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is
> local file system location. However, FileSystemApplicationHistoryStore uses
> DFS by default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1701) Improve default paths of timeline store and generic history store
[
https://issues.apache.org/jira/browse/YARN-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated YARN-1701:
Affects Version/s: (was: 2.4.1)
2.4.0
> Improve default paths of timeline store and generic history store
> -
>
> Key: YARN-1701
> URL: https://issues.apache.org/jira/browse/YARN-1701
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: YARN-1701.v01.patch, YARN-1701.v02.patch
>
>
> When I enable AHS via yarn.ahs.enabled, the app history is still not visible
> in AHS webUI. This is due to NullApplicationHistoryStore as
> yarn.resourcemanager.history-writer.class. It would be good to have just one
> key to enable basic functionality.
> yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is
> local file system location. However, FileSystemApplicationHistoryStore uses
> DFS by default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967780#comment-13967780 ] Tsuyoshi OZAWA commented on YARN-1474: -- [~kasha] , now the patch passed findbugs and is ready for review. Could you take a look? Thanks. > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.2.patch, YARN-1474.3.patch, > YARN-1474.4.patch, YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch, > YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1474) Make schedulers services
[
https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967770#comment-13967770
]
Hadoop QA commented on YARN-1474:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639978/YARN-1474.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3563//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3563//console
This message is automatically generated.
> Make schedulers services
>
>
> Key: YARN-1474
> URL: https://issues.apache.org/jira/browse/YARN-1474
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 2.3.0
>Reporter: Sandy Ryza
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1474.1.patch, YARN-1474.2.patch, YARN-1474.3.patch,
> YARN-1474.4.patch, YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch,
> YARN-1474.8.patch, YARN-1474.9.patch
>
>
> Schedulers currently have a reinitialize but no start and stop. Fitting them
> into the YARN service model would make things more coherent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1928) TestAMRMRPCNodeUpdates fails ocassionally
[
https://issues.apache.org/jira/browse/YARN-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967767#comment-13967767
]
Tsuyoshi OZAWA commented on YARN-1928:
--
LGTM.
> TestAMRMRPCNodeUpdates fails ocassionally
> -
>
> Key: YARN-1928
> URL: https://issues.apache.org/jira/browse/YARN-1928
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-1928.1.patch
>
>
> {code}
> junit.framework.AssertionFailedError: expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates.testAMRMUnusableNodes(TestAMRMRPCNodeUpdates.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: YARN-1474.9.patch Updated a patch to synchronize Scheduler related methods correctly. Removed @Lock(Lock.NoLock.class) from CapacityScheduler#getApplicationAttempt() because the method is made synchronized. > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.2.patch, YARN-1474.3.patch, > YARN-1474.4.patch, YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch, > YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
