[jira] [Commented] (YARN-1320) Custom log4j properties in Distributed shell does not work properly.
[
https://issues.apache.org/jira/browse/YARN-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828536#comment-13828536
]
Hadoop QA commented on YARN-1320:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615068/YARN-1320.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2506//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2506//console
This message is automatically generated.
> Custom log4j properties in Distributed shell does not work properly.
>
>
> Key: YARN-1320
> URL: https://issues.apache.org/jira/browse/YARN-1320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1320.1.patch, YARN-1320.2.patch, YARN-1320.3.patch,
> YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch,
> YARN-1320.4.patch, YARN-1320.5.patch, YARN-1320.6.patch, YARN-1320.6.patch,
> YARN-1320.7.patch, YARN-1320.8.patch, YARN-1320.9.patch
>
>
> Distributed shell cannot pick up custom log4j properties (specified with
> -log_properties). It always uses default log4j properties.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828531#comment-13828531 ] Hudson commented on YARN-1303: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4774 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4774/]) YARN-1303. Reverted the wrong patch committed earlier and committing the correct patch now. In one go. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544029) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.3.0 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch, > YARN-1303.8.2.patch, YARN-1303.8.patch, YARN-1303.9.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1426) YARN Components need to unregister their beans upon shutdown
[ https://issues.apache.org/jira/browse/YARN-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-1426: -- Attachment: YARN-1426.patch > YARN Components need to unregister their beans upon shutdown > > > Key: YARN-1426 > URL: https://issues.apache.org/jira/browse/YARN-1426 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: YARN-1426.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (YARN-1426) YARN Components need to unregister their beans upon shutdown
[ https://issues.apache.org/jira/browse/YARN-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned YARN-1426: - Assignee: Jonathan Eagles > YARN Components need to unregister their beans upon shutdown > > > Key: YARN-1426 > URL: https://issues.apache.org/jira/browse/YARN-1426 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1320) Custom log4j properties in Distributed shell does not work properly.
[
https://issues.apache.org/jira/browse/YARN-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828488#comment-13828488
]
Hadoop QA commented on YARN-1320:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615068/YARN-1320.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2504//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2504//console
This message is automatically generated.
> Custom log4j properties in Distributed shell does not work properly.
>
>
> Key: YARN-1320
> URL: https://issues.apache.org/jira/browse/YARN-1320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1320.1.patch, YARN-1320.2.patch, YARN-1320.3.patch,
> YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch,
> YARN-1320.4.patch, YARN-1320.5.patch, YARN-1320.6.patch, YARN-1320.6.patch,
> YARN-1320.7.patch, YARN-1320.8.patch, YARN-1320.9.patch
>
>
> Distributed shell cannot pick up custom log4j properties (specified with
> -log_properties). It always uses default log4j properties.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[
https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828485#comment-13828485
]
Hadoop QA commented on YARN-1239:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615046/YARN-1239.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2502//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2502//console
This message is automatically generated.
> Save version information in the state store
> ---
>
> Key: YARN-1239
> URL: https://issues.apache.org/jira/browse/YARN-1239
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch,
> YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch
>
>
> When creating root dir for the first time we should write version 1. If root
> dir exists then we should check that the version in the state store matches
> the version from config.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[
https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828483#comment-13828483
]
Hadoop QA commented on YARN-1417:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615062/YARN-1417.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2503//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2503//console
This message is automatically generated.
> RM may issue expired container tokens to AM while issuing new containers.
> -
>
> Key: YARN-1417
> URL: https://issues.apache.org/jira/browse/YARN-1417
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-1417.2.patch
>
>
> Today we create new container token when we create container in RM as a part
> of schedule cycle. However that container may get reserved or assigned. If
> the container gets reserved and remains like that (in reserved state) for
> more than container token expiry interval then RM will end up issuing
> container with expired token.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1320) Custom log4j properties in Distributed shell does not work properly.
[ https://issues.apache.org/jira/browse/YARN-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828478#comment-13828478 ] Xuan Gong commented on YARN-1320: - bq. Seems like we don't need to pass anything to the AM. AM can verify if a file by name log4j.properties exists and use it. So, the cmd line option log_properties to AM can be removed. REMOVED bq. In the client, you can use the "logj4Path" constant while setting up the localResource. Changed bq.Link this JIRA to YARN-1303 and reuse test-code? We can re-use the code > Custom log4j properties in Distributed shell does not work properly. > > > Key: YARN-1320 > URL: https://issues.apache.org/jira/browse/YARN-1320 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: YARN-1320.1.patch, YARN-1320.2.patch, YARN-1320.3.patch, > YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, > YARN-1320.4.patch, YARN-1320.5.patch, YARN-1320.6.patch, YARN-1320.6.patch, > YARN-1320.7.patch, YARN-1320.8.patch > > > Distributed shell cannot pick up custom log4j properties (specified with > -log_properties). It always uses default log4j properties. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1320) Custom log4j properties in Distributed shell does not work properly.
[ https://issues.apache.org/jira/browse/YARN-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1320: Attachment: YARN-1320.9.patch > Custom log4j properties in Distributed shell does not work properly. > > > Key: YARN-1320 > URL: https://issues.apache.org/jira/browse/YARN-1320 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: YARN-1320.1.patch, YARN-1320.2.patch, YARN-1320.3.patch, > YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, YARN-1320.4.patch, > YARN-1320.4.patch, YARN-1320.5.patch, YARN-1320.6.patch, YARN-1320.6.patch, > YARN-1320.7.patch, YARN-1320.8.patch, YARN-1320.9.patch > > > Distributed shell cannot pick up custom log4j properties (specified with > -log_properties). It always uses default log4j properties. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828475#comment-13828475 ] Hudson commented on YARN-1303: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4773 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4773/]) YARN-1303. Fixed DistributedShell to not fail with multiple commands separated by a semi-colon as shell-command. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544023) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.3.0 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch, > YARN-1303.8.2.patch, YARN-1303.8.patch, YARN-1303.9.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1422) RM CapacityScheduler can deadlock when getQueueUserAclInfo() is called and a container is completing
[
https://issues.apache.org/jira/browse/YARN-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1422:
--
Priority: Blocker (was: Critical)
> RM CapacityScheduler can deadlock when getQueueUserAclInfo() is called and a
> container is completing
>
>
> Key: YARN-1422
> URL: https://issues.apache.org/jira/browse/YARN-1422
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects Versions: 2.2.0
>Reporter: Adam Kawa
>Priority: Blocker
>
> If getQueueUserAclInfo() on a parent/root queue (e.g. via
> CapacityScheduler.getQueueUserAclInfo) is called, and a container is
> completing, then the ResourceManager can deadlock.
> It is similar to https://issues.apache.org/jira/browse/YARN-325.
> *More details:*
> * Thread A
> 1) In a synchronized block of code (a lockid
> 0xc18d8870=LeafQueue.class), LeafQueue.completedContainer wants to
> inform the parent queue that a container is being completed and invokes
> ParentQueue.completedContainer method.
> 3) The ParentQueue.completedContainer waits to aquire a lock on itself (a
> lockid 0xc1846350=ParentQueue.class) to go to synchronized block of
> code. It can not accuire this lock, because Thread B already has this lock.
> * Thread B
> 0) A moment earlier, CapacityScheduler.getQueueUserAclInfo is called. This
> method invokes a synchronized method on ParentQueue.class i.e.
> ParentQueue.getQueueUserAclInfo (a lockid
> 0xc1846350=ParentQueue.class) and aquires the lock that Thread A will
> be waiting for.
> 2) Unluckyly, ParentQueue.getQueueUserAclInfo iterates over children queue
> acls and it wants to run a synchonized method, LeafQueue.getQueueUserAclInfo,
> but it does not have a lock on LeafQueue.class (a lockid
> 0xc18d8870=LeafQueue.class). This lock is already held by
> LeafQueue.completedContainer in Thread A.
> The order that causes the deadlock: B0 -> A1 -> B2 -> A3.
> *Java Stacktrace*
> {code}
> Found one Java-level deadlock:
> =
> "1956747953@qtp-109760451-1959":
> waiting to lock monitor 0x434e10c8 (object 0xc1846350, a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue),
> which is held by "IPC Server handler 39 on 8032"
> "IPC Server handler 39 on 8032":
> waiting to lock monitor 0x422bbc58 (object 0xc18d8870, a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue),
> which is held by "ResourceManager Event Processor"
> "ResourceManager Event Processor":
> waiting to lock monitor 0x434e10c8 (object 0xc1846350, a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue),
> which is held by "IPC Server handler 39 on 8032"
> Java stack information for the threads listed above:
> ===
> "1956747953@qtp-109760451-1959":
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getUsedCapacity(ParentQueue.java:276)
> - waiting to lock <0xc1846350> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.CapacitySchedulerInfo.(CapacitySchedulerInfo.java:49)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:203)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.scheduler(RmController.java:76)
> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:82
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[
https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828462#comment-13828462
]
Xuan Gong commented on YARN-1303:
-
Did the test on a single node cluster :
Using the command :
{code}
hadoop-3.0.0-SNAPSHOT/bin/hadoop jar
hadoop-yarn-project-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
org.apache.hadoop.yarn.applications.distributedshell.Client --jar
hadoop-yarn-project-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
--shell_command "ls|ps" --num_containers 2
{code}
And From launch_container.sh of Container :
{code}
exec /bin/bash -c "ls|ps
1>/Users/xuan/dep/hadoop-3.0.0-SNAPSHOT/logs/application_1385002853820_0002/container_1385002853820_0002_01_03/stdout
2>/Users/xuan/dep/hadoop-3.0.0-SNAPSHOT/logs/application_1385002853820_0002/container_1385002853820_0002_01_03/stderr
"
{code}
> Allow multiple commands separating with ";" in distributed-shell
>
>
> Key: YARN-1303
> URL: https://issues.apache.org/jira/browse/YARN-1303
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications/distributed-shell
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch,
> YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch,
> YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch,
> YARN-1303.8.2.patch, YARN-1303.8.patch, YARN-1303.9.patch
>
>
> In shell, we can do "ls; ls" to run 2 commands at once.
> In distributed shell, this is not working. We should improve to allow this to
> occur. There are practical use cases that I know of to run multiple commands
> or to set environment variables before a command.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (YARN-1431) TestWebAppProxyServlet is failing on trunk
Omkar Vinit Joshi created YARN-1431: --- Summary: TestWebAppProxyServlet is failing on trunk Key: YARN-1431 URL: https://issues.apache.org/jira/browse/YARN-1431 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.1 Reporter: Omkar Vinit Joshi Priority: Blocker Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.609 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet testWebAppProxyServerMainMethod(org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet) Time elapsed: 5.006 sec <<< ERROR! java.lang.Exception: test timed out after 5000 milliseconds at java.net.Inet4AddressImpl.getHostByAddr(Native Method) at java.net.InetAddress$1.getHostByAddr(InetAddress.java:881) at java.net.InetAddress.getHostFromNameService(InetAddress.java:560) at java.net.InetAddress.getCanonicalHostName(InetAddress.java:531) at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:227) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:247) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.doSecureLogin(WebAppProxyServer.java:72) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.serviceInit(WebAppProxyServer.java:57) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.startServer(WebAppProxyServer.java:99) at org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet.testWebAppProxyServerMainMethod(TestWebAppProxyServlet.java:187) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1431) TestWebAppProxyServlet is failing on trunk
[
https://issues.apache.org/jira/browse/YARN-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828444#comment-13828444
]
Omkar Vinit Joshi commented on YARN-1431:
-
extract from surefire logs
{code}
2013-11-20 18:59:47,514 INFO [Thread-4] mortbay.log (Slf4jLog.java:info(67)) -
Extract
jar:file:/Users/ojoshi/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/proxy
to
/var/folders/h8/dlw3rlfx0_b5zjrw7kn1752mgn/T/Jetty_localhost_57922_proxyrtadom/webapp
2013-11-20 18:59:47,678 INFO [Thread-4] mortbay.log (Slf4jLog.java:info(67)) -
Started SelectChannelConnector@localhost:57922
Proxy server is started at port 57922
2013-11-20 18:59:47,797 ERROR [1023736867@qtp-568432173-0] mortbay.log
(Slf4jLog.java:warn(87)) - /proxy/app
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing
application ID: app
at org.apache.hadoop.yarn.util.Apps.throwParseException(Apps.java:69)
at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:54)
at org.apache.hadoop.yarn.util.Apps.toAppID(Apps.java:49)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:252)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1220)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2013-11-20 18:59:47,854 INFO [1023736867@qtp-568432173-0]
webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(322)) - dr.who is
accessing unchecked http://localhost:57919/foo/bar/ which is the app master GUI
of application_00_0 owned by dr.who
2013-11-20 18:59:47,925 WARN [1023736867@qtp-568432173-0]
webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(278)) - dr.who
Attempting to access application_0_ that was not found
{code}
> TestWebAppProxyServlet is failing on trunk
> --
>
> Key: YARN-1431
> URL: https://issues.apache.org/jira/browse/YARN-1431
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.2.1
>Reporter: Omkar Vinit Joshi
>Priority: Blocker
>
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.609 sec <<<
> FAILURE! - in org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet
> testWebAppProxyServerMainMethod(org.apache.hadoop.yarn.server.webproxy.TestWebAppProxyServlet)
> Time elapsed: 5.006 sec <<< ERROR!
> java.lang.Exception: test timed out after 5000 milliseconds
> at java.net.Inet4AddressImpl.getHostByAddr(Native Method)
> at java.net.InetAddress$1.getHostByAddr(InetAddress.java:881)
> at java.net.InetAddress.getHostFromNameService(InetAddress.j
[jira] [Updated] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[ https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-1417: Attachment: YARN-1417.2.patch > RM may issue expired container tokens to AM while issuing new containers. > - > > Key: YARN-1417 > URL: https://issues.apache.org/jira/browse/YARN-1417 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-1417.2.patch > > > Today we create new container token when we create container in RM as a part > of schedule cycle. However that container may get reserved or assigned. If > the container gets reserved and remains like that (in reserved state) for > more than container token expiry interval then RM will end up issuing > container with expired token. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[ https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828435#comment-13828435 ] Omkar Vinit Joshi commented on YARN-1417: - updating basic patch here.. without test cases .. if this approach looks ok then we can easily fix YARN-713.. > RM may issue expired container tokens to AM while issuing new containers. > - > > Key: YARN-1417 > URL: https://issues.apache.org/jira/browse/YARN-1417 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-1417.2.patch > > > Today we create new container token when we create container in RM as a part > of schedule cycle. However that container may get reserved or assigned. If > the container gets reserved and remains like that (in reserved state) for > more than container token expiry interval then RM will end up issuing > container with expired token. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828428#comment-13828428 ] Tsuyoshi OZAWA commented on YARN-1239: -- The scope of this JIRA looks reasonable for me. I'll review your patch later. > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1412) Allocating Containers on a particular Node in Yarn
[
https://issues.apache.org/jira/browse/YARN-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828412#comment-13828412
]
Bikas Saha commented on YARN-1412:
--
I mentioned allocating multiple containers at the same priority as an
experiment to check if the above theory is correct or not.
bq. But since it is not working we are trying to code it.
I am saying that your code (yes, null, true) is the same as the default. The
behavior that you will get (correct or buggy as it is now) will be the same in
both cases.
bq. yarn.scheduler.capacity.node-locality-delay is set to 50 and the size of
cluster is 36.
The max value of this should be the number of nodes in the cluster. Higher than
that has no effect.
Btw, are all nodes in the same rack?
> Allocating Containers on a particular Node in Yarn
> --
>
> Key: YARN-1412
> URL: https://issues.apache.org/jira/browse/YARN-1412
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: centos, Hadoop 2.2.0
>Reporter: gaurav gupta
>
> Summary of the problem:
> If I pass the node on which I want container and set relax locality default
> which is true, I don't get back the container on the node specified even if
> the resources are available on the node. It doesn't matter if I set rack or
> not.
> Here is the snippet of the code that I am using
> AMRMClient amRmClient = AMRMClient.createAMRMClient();;
> String host = "h1";
> Resource capability = Records.newRecord(Resource.class);
> capability.setMemory(memory);
> nodes = new String[] {host};
> // in order to request a host, we also have to request the rack
> racks = new String[] {"/default-rack"};
> List containerRequests = new
> ArrayList();
> List releasedContainers = new ArrayList();
> containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority)));
> if (containerRequests.size() > 0) {
> LOG.info("Asking RM for containers: " + containerRequests);
> for (ContainerRequest cr : containerRequests) {
> LOG.info("Requested container: {}", cr.toString());
> amRmClient.addContainerRequest(cr);
> }
> }
> for (ContainerId containerId : releasedContainers) {
> LOG.info("Released container, id={}", containerId.getId());
> amRmClient.releaseAssignedContainer(containerId);
> }
> return amRmClient.allocate(0);
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[
https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828405#comment-13828405
]
Hadoop QA commented on YARN-1303:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615047/YARN-1303.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2501//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2501//console
This message is automatically generated.
> Allow multiple commands separating with ";" in distributed-shell
>
>
> Key: YARN-1303
> URL: https://issues.apache.org/jira/browse/YARN-1303
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications/distributed-shell
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch,
> YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch,
> YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch,
> YARN-1303.8.2.patch, YARN-1303.8.patch, YARN-1303.9.patch
>
>
> In shell, we can do "ls; ls" to run 2 commands at once.
> In distributed shell, this is not working. We should improve to allow this to
> occur. There are practical use cases that I know of to run multiple commands
> or to set environment variables before a command.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (YARN-1430) InvalidStateTransition exceptions are ignored in state machines
Omkar Vinit Joshi created YARN-1430: --- Summary: InvalidStateTransition exceptions are ignored in state machines Key: YARN-1430 URL: https://issues.apache.org/jira/browse/YARN-1430 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi We have all state machines ignoring InvalidStateTransitions. These exceptions will get logged but will not crash the RM / NM. We definitely should crash it as they move the system into some invalid / unacceptable state. *Places where we hide this exception :- ** JobImpl ** TaskAttemptImpl ** TaskImpl ** NMClientAsyncImpl ** ApplicationImpl ** ContainerImpl ** LocalizedResource ** RMAppAttemptImpl ** RMAppImpl ** RMContainerImpl ** RMNodeImpl thoughts? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1430) InvalidStateTransition exceptions are ignored in state machines
[ https://issues.apache.org/jira/browse/YARN-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-1430: Description: We have all state machines ignoring InvalidStateTransitions. These exceptions will get logged but will not crash the RM / NM. We definitely should crash it as they move the system into some invalid / unacceptable state. * Places where we hide this exception :- ** JobImpl ** TaskAttemptImpl ** TaskImpl ** NMClientAsyncImpl ** ApplicationImpl ** ContainerImpl ** LocalizedResource ** RMAppAttemptImpl ** RMAppImpl ** RMContainerImpl ** RMNodeImpl thoughts? was: We have all state machines ignoring InvalidStateTransitions. These exceptions will get logged but will not crash the RM / NM. We definitely should crash it as they move the system into some invalid / unacceptable state. *Places where we hide this exception :- ** JobImpl ** TaskAttemptImpl ** TaskImpl ** NMClientAsyncImpl ** ApplicationImpl ** ContainerImpl ** LocalizedResource ** RMAppAttemptImpl ** RMAppImpl ** RMContainerImpl ** RMNodeImpl thoughts? > InvalidStateTransition exceptions are ignored in state machines > --- > > Key: YARN-1430 > URL: https://issues.apache.org/jira/browse/YARN-1430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > > We have all state machines ignoring InvalidStateTransitions. These exceptions > will get logged but will not crash the RM / NM. We definitely should crash it > as they move the system into some invalid / unacceptable state. > * Places where we hide this exception :- > ** JobImpl > ** TaskAttemptImpl > ** TaskImpl > ** NMClientAsyncImpl > ** ApplicationImpl > ** ContainerImpl > ** LocalizedResource > ** RMAppAttemptImpl > ** RMAppImpl > ** RMContainerImpl > ** RMNodeImpl > thoughts? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1266) inheriting Application client and History Protocol from base protocol and implement PB service and clients.
[ https://issues.apache.org/jira/browse/YARN-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828376#comment-13828376 ] Vinod Kumar Vavilapalli commented on YARN-1266: --- Sigh, I don't like this overall if can't reuse everywhere. Sorry to let you lose a wild goose chase. You are going to kill me when I say this, but it seems like we should just commit your first patch. Because this neither-here-nor-there code-reuse seems not much useful. The original reason why i asked this is so that clients can seamlessly cross over like in MR between RM and AHS. But it seems like clients have to know which protocol they are talking to. This half-reuse is more confusing for new devs who see this code for the first time. > inheriting Application client and History Protocol from base protocol and > implement PB service and clients. > --- > > Key: YARN-1266 > URL: https://issues.apache.org/jira/browse/YARN-1266 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-1266-1.patch, YARN-1266-2.patch, YARN-1266-3.patch, > YARN-1266-4.patch, YARN-1266-5.patch > > > Adding ApplicationHistoryProtocolPBService to make web apps to work and > changing yarn to run AHS as a seprate process -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828374#comment-13828374 ] Wangda Tan commented on YARN-1197: -- [~sandyr] These bunch of code will be hard for review, but I will make and upload a all-in-one draft patch for early preview before split it. I hope I can finish the alpha patch during this weekend :-] I agree you mentioned YARN enforce memory monitoring via ContainersMonitor, I ask this because I noticed the design doc of YARN-3, https://issues.apache.org/jira/secure/attachment/12538426/mapreduce-4334-design-doc-v2.txt, the "Design 2(b)" mentioned "ResoureEnforcer.init()", before create cgroups. I found this is not done yet, so we can safely update the resource enforce via update limit of ContainersMonitor now. > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: yarn-1197-v2.pdf, yarn-1197-v3.pdf, yarn-1197-v4.pdf, > yarn-1197.pdf > > > Currently, YARN cannot support merge several containers in one node to a big > container, which can make us incrementally ask resources, merge them to a > bigger one, and launch our processes. The user scenario is described in the > comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1412) Allocating Containers on a particular Node in Yarn
[
https://issues.apache.org/jira/browse/YARN-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828368#comment-13828368
]
gaurav gupta commented on YARN-1412:
yarn.scheduler.capacity.node-locality-delay is set to 50 and the size of
cluster is 36.
Since this property affects the cluster, not sure if it is right thing to
depend on this property for container allocation
For the nature of our application we are requesting containers with incremental
priorities so we have 1 container per priority, so we can't request for
multiple containers at pri 0 and there are some applications were total number
of containers are less than the cluster size.
We want what you mentioned "Btw, what you are trying to do (node=specific,
rack=null and relaxLocality=true) is the default behavior of existing
schedulers. They will always try to relax locality to rack and then off-switch
by default. So you dont need to explicitly code for it. "... But since it is
not working we are trying to code it.
> Allocating Containers on a particular Node in Yarn
> --
>
> Key: YARN-1412
> URL: https://issues.apache.org/jira/browse/YARN-1412
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: centos, Hadoop 2.2.0
>Reporter: gaurav gupta
>
> Summary of the problem:
> If I pass the node on which I want container and set relax locality default
> which is true, I don't get back the container on the node specified even if
> the resources are available on the node. It doesn't matter if I set rack or
> not.
> Here is the snippet of the code that I am using
> AMRMClient amRmClient = AMRMClient.createAMRMClient();;
> String host = "h1";
> Resource capability = Records.newRecord(Resource.class);
> capability.setMemory(memory);
> nodes = new String[] {host};
> // in order to request a host, we also have to request the rack
> racks = new String[] {"/default-rack"};
> List containerRequests = new
> ArrayList();
> List releasedContainers = new ArrayList();
> containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority)));
> if (containerRequests.size() > 0) {
> LOG.info("Asking RM for containers: " + containerRequests);
> for (ContainerRequest cr : containerRequests) {
> LOG.info("Requested container: {}", cr.toString());
> amRmClient.addContainerRequest(cr);
> }
> }
> for (ContainerId containerId : releasedContainers) {
> LOG.info("Released container, id={}", containerId.getId());
> amRmClient.releaseAssignedContainer(containerId);
> }
> return amRmClient.allocate(0);
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1303: Attachment: YARN-1303.9.patch > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch, > YARN-1303.8.2.patch, YARN-1303.8.patch, YARN-1303.9.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828363#comment-13828363 ] Xuan Gong commented on YARN-1303: - bq.Use LocalResource.newInstance methods. changed bq.Make the string "shellCommands" as a constant. fixed bq.Change the command line parameter to the AM as shell_command_path Think that we do not need this option shell_command in AM anymore. We can simply check whether the shellCommandPath exists or not. bq.Change the example to use a command with';' as per JIRA description? changed > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch, > YARN-1303.8.2.patch, YARN-1303.8.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1239: -- Attachment: YARN-1239.4.patch addendum fixes > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.4.patch, YARN-1239.4.patch, YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1239: -- Attachment: (was: YARN-1239.4.patch) > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.4.patch, YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1239: -- Attachment: YARN-1239.4.patch > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.4.patch, YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1239: -- Attachment: YARN-1239.4.patch upload a patch: - revert the earlier field tag number changes of proto, as per PB guide, we should not change the tag numbers of any existing fields for compatibility. - create a RMStateVersion object for wrapping version info - Workflow on recovery: -- if there's no version info, write the default version info, -- if the loaded version info is compatible, rewrite the version info. -- if the loaded version info is not compatible, throw exception. we can leave the upgrade state implementation in a separate upgrade tool. > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.4.patch, YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1412) Allocating Containers on a particular Node in Yarn
[
https://issues.apache.org/jira/browse/YARN-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828340#comment-13828340
]
Bikas Saha commented on YARN-1412:
--
What is the value of the following configuration?
yarn.scheduler.capacity.node-locality-delay
It looks like you are being hit by a bug that will happen with small number of
container requests.
LeafQueue.assignContainersOnNode()
Looks like if rackLocalityDelay is not met, even then the scheduler falls back
to off-switch assignment. The delay calculation for off-switch assignment is
basically (#different-locations/#nodes-in-cluster)*#containers <
#node-heartbeats-without-assignment. In your case, if you have 20 nodes in all,
(2/20)*1 == 0.1. So the moment we skip 1 node (waiting for locality delay) we
end up assigning an off-switch container to the request.
Try the following, set the node locality delay mentioned at the beginning to
the number of nodes in the cluster. Then instead of asking for 1 container at
pri 0, ask for 20 containers, each for a specific node, rack=false, relax=true.
The above off-switch locality delay will become 20/20*1 == 20 missed
assignments.
If you see correct assignments then the above theory is correct about the bug.
Btw, what you are trying to do (node=specific, rack=null and
relaxLocality=true) is the default behavior of existing schedulers. They will
always try to relax locality to rack and then off-switch by default. So you
dont need to explicitly code for it.
> Allocating Containers on a particular Node in Yarn
> --
>
> Key: YARN-1412
> URL: https://issues.apache.org/jira/browse/YARN-1412
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: centos, Hadoop 2.2.0
>Reporter: gaurav gupta
>
> Summary of the problem:
> If I pass the node on which I want container and set relax locality default
> which is true, I don't get back the container on the node specified even if
> the resources are available on the node. It doesn't matter if I set rack or
> not.
> Here is the snippet of the code that I am using
> AMRMClient amRmClient = AMRMClient.createAMRMClient();;
> String host = "h1";
> Resource capability = Records.newRecord(Resource.class);
> capability.setMemory(memory);
> nodes = new String[] {host};
> // in order to request a host, we also have to request the rack
> racks = new String[] {"/default-rack"};
> List containerRequests = new
> ArrayList();
> List releasedContainers = new ArrayList();
> containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority)));
> if (containerRequests.size() > 0) {
> LOG.info("Asking RM for containers: " + containerRequests);
> for (ContainerRequest cr : containerRequests) {
> LOG.info("Requested container: {}", cr.toString());
> amRmClient.addContainerRequest(cr);
> }
> }
> for (ContainerId containerId : releasedContainers) {
> LOG.info("Released container, id={}", containerId.getId());
> amRmClient.releaseAssignedContainer(containerId);
> }
> return amRmClient.allocate(0);
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1416) InvalidStateTransitions getting reported in multiple test cases even though they pass
[
https://issues.apache.org/jira/browse/YARN-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828287#comment-13828287
]
Omkar Vinit Joshi commented on YARN-1416:
-
Thanks [~jianhe]
I have basic question.. RM should have crashed right? we can't just ignore such
invalid state transitions? Should we? I see that someone has modified it to log
the exception but ignore it inside RMAppImpl.java.
{code}
try {
/* keep the master in sync with the state machine */
this.stateMachine.doTransition(event.getType(), event);
} catch (InvalidStateTransitonException e) {
LOG.error("Can't handle this event at current state", e);
/* TODO fail the application on the failed transition */
}
{code}
I see that other places too we are ignoring this after logging it. Not sure if
this is right because we may just move the system into corrupted state without
crashing/stopping it. At least we should add assert statements to all the state
machines to make sure that such transitions don't go unnoticed.
I applied the patch and tested locally.. one more test needs to be fixed..
{code}
2013-11-20 15:23:52,127 INFO [AsyncDispatcher event handler]
attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(645)) -
appattempt_1384989831257_0042_01 State change from NEW to SUBMITTED
2013-11-20 15:23:52,129 ERROR [AsyncDispatcher event handler] rmapp.RMAppImpl
(RMAppImpl.java:handle(593)) - Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
APP_ACCEPTED at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:591)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions$TestApplicationEventDispatcher.handle(TestRMAppTransitions.java:139)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions$TestApplicationEventDispatcher.handle(TestRMAppTransitions.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:159)
at
org.apache.hadoop.yarn.event.DrainDispatcher$1.run(DrainDispatcher.java:65)
at java.lang.Thread.run(Thread.java:680)
{code}
> InvalidStateTransitions getting reported in multiple test cases even though
> they pass
> -
>
> Key: YARN-1416
> URL: https://issues.apache.org/jira/browse/YARN-1416
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Jian He
> Attachments: YARN-1416.1.patch, YARN-1416.1.patch
>
>
> It might be worth checking why they are reporting this.
> Testcase : TestRMAppTransitions, TestRM
> there are large number of such errors.
> can't handle RMAppEventType.APP_UPDATE_SAVED at RMAppState.FAILED
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828270#comment-13828270 ] Jonathan Eagles commented on YARN-1343: --- This change introduced a test failure in TestRMContainerAllocator#testUpdatedNodes MAPREDUCE-5632 since it is counting the jobUpdatedNodeEvents. Can someone [~tucu00] or [~bikassaha] verify the patch and make sure that the test reflects the new proper behavior and that I'm not masking a real error in the code. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch, YARN-1343.patch, YARN-1343.patch, > YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1412) Allocating Containers on a particular Node in Yarn
[
https://issues.apache.org/jira/browse/YARN-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gaurav gupta updated YARN-1412:
---
Description:
Summary of the problem:
If I pass the node on which I want container and set relax locality default
which is true, I don't get back the container on the node specified even if the
resources are available on the node. It doesn't matter if I set rack or not.
Here is the snippet of the code that I am using
AMRMClient amRmClient = AMRMClient.createAMRMClient();;
String host = "h1";
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(memory);
nodes = new String[] {host};
// in order to request a host, we also have to request the rack
racks = new String[] {"/default-rack"};
List containerRequests = new
ArrayList();
List releasedContainers = new ArrayList();
containerRequests.add(new ContainerRequest(capability, nodes, racks,
Priority.newInstance(priority)));
if (containerRequests.size() > 0) {
LOG.info("Asking RM for containers: " + containerRequests);
for (ContainerRequest cr : containerRequests) {
LOG.info("Requested container: {}", cr.toString());
amRmClient.addContainerRequest(cr);
}
}
for (ContainerId containerId : releasedContainers) {
LOG.info("Released container, id={}", containerId.getId());
amRmClient.releaseAssignedContainer(containerId);
}
return amRmClient.allocate(0);
was:
I am trying to allocate containers on a particular node in Yarn but Yarn is
returning me containers on different node although the requested node has
resources available.
Here is the snippet of the code that I am using
AMRMClient amRmClient = AMRMClient.createAMRMClient();;
String host = "h1";
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(memory);
nodes = new String[] {host};
// in order to request a host, we also have to request the rack
racks = new String[] {"/default-rack"};
List containerRequests = new
ArrayList();
List releasedContainers = new ArrayList();
containerRequests.add(new ContainerRequest(capability, nodes, racks,
Priority.newInstance(priority)));
if (containerRequests.size() > 0) {
LOG.info("Asking RM for containers: " + containerRequests);
for (ContainerRequest cr : containerRequests) {
LOG.info("Requested container: {}", cr.toString());
amRmClient.addContainerRequest(cr);
}
}
for (ContainerId containerId : releasedContainers) {
LOG.info("Released container, id={}", containerId.getId());
amRmClient.releaseAssignedContainer(containerId);
}
return amRmClient.allocate(0);
> Allocating Containers on a particular Node in Yarn
> --
>
> Key: YARN-1412
> URL: https://issues.apache.org/jira/browse/YARN-1412
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: centos, Hadoop 2.2.0
>Reporter: gaurav gupta
>
> Summary of the problem:
> If I pass the node on which I want container and set relax locality default
> which is true, I don't get back the container on the node specified even if
> the resources are available on the node. It doesn't matter if I set rack or
> not.
> Here is the snippet of the code that I am using
> AMRMClient amRmClient = AMRMClient.createAMRMClient();;
> String host = "h1";
> Resource capability = Records.newRecord(Resource.class);
> capability.setMemory(memory);
> nodes = new String[] {host};
> // in order to request a host, we also have to request the rack
> racks = new String[] {"/default-rack"};
> List containerRequests = new
> ArrayList();
> List releasedContainers = new ArrayList();
> containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority)));
> if (containerRequests.size() > 0) {
> LOG.info("Asking RM for containers: " + containerRequests);
> for (ContainerRequest cr : containerRequests) {
> LOG.info("Requested container: {}", cr.toString());
> amRmClient.addContainerRequest(cr);
> }
> }
> for (ContainerId containerId : releasedContainers) {
> LOG.info("Released container, id={}", containerId.getId());
> amRmClient.releaseAssignedContainer(containerId);
> }
> return amRmClient.allocate(0);
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1239) Save version information in the state store
[ https://issues.apache.org/jira/browse/YARN-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828225#comment-13828225 ] Jian He commented on YARN-1239: --- Discussed with Vinod offline, in the scope of this jira, we can just check whether this is a compatible state. If it is compatible, RM can load it and proceed as normal. In case of incompatible, throw exception and indicate user to upgrade the state with a separate tool to do things like rearranging the directory structure or renaming the fields back, etc. For that, each incompatible implementation needs to write their own version parser for parsing and loading the data. > Save version information in the state store > --- > > Key: YARN-1239 > URL: https://issues.apache.org/jira/browse/YARN-1239 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-1239.1.patch, YARN-1239.2.patch, YARN-1239.3.patch, > YARN-1239.patch > > > When creating root dir for the first time we should write version 1. If root > dir exists then we should check that the version in the state store matches > the version from config. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1053) Diagnostic message from ContainerExitEvent is ignored in ContainerImpl
[ https://issues.apache.org/jira/browse/YARN-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828221#comment-13828221 ] Hudson commented on YARN-1053: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4772 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4772/]) YARN-1053. Diagnostic message from ContainerExitEvent is ignored in ContainerImpl (Omkar Vinit Joshi via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543973) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java > Diagnostic message from ContainerExitEvent is ignored in ContainerImpl > -- > > Key: YARN-1053 > URL: https://issues.apache.org/jira/browse/YARN-1053 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0, 2.2.1 >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Labels: newbie > Fix For: 2.3.0 > > Attachments: YARN-1053.1.patch, YARN-1053.20130809.patch > > > If the container launch fails then we send ContainerExitEvent. This event > contains exitCode and diagnostic message. Today we are ignoring diagnostic > message while handling this event inside ContainerImpl. Fixing it as it is > useful in diagnosing the failure. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1053) Diagnostic message from ContainerExitEvent is ignored in ContainerImpl
[
https://issues.apache.org/jira/browse/YARN-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828198#comment-13828198
]
Hadoop QA commented on YARN-1053:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615002/YARN-1053.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2500//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2500//console
This message is automatically generated.
> Diagnostic message from ContainerExitEvent is ignored in ContainerImpl
> --
>
> Key: YARN-1053
> URL: https://issues.apache.org/jira/browse/YARN-1053
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.2.1
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Labels: newbie
> Fix For: 2.3.0, 2.2.1
>
> Attachments: YARN-1053.1.patch, YARN-1053.20130809.patch
>
>
> If the container launch fails then we send ContainerExitEvent. This event
> contains exitCode and diagnostic message. Today we are ignoring diagnostic
> message while handling this event inside ContainerImpl. Fixing it as it is
> useful in diagnosing the failure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-955) [YARN-321] Implementation of ApplicationHistoryProtocol
[
https://issues.apache.org/jira/browse/YARN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828178#comment-13828178
]
Hadoop QA commented on YARN-955:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615004/YARN-955-6.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2499//console
This message is automatically generated.
> [YARN-321] Implementation of ApplicationHistoryProtocol
> ---
>
> Key: YARN-955
> URL: https://issues.apache.org/jira/browse/YARN-955
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Mayank Bansal
> Attachments: YARN-955-1.patch, YARN-955-2.patch, YARN-955-3.patch,
> YARN-955-4.patch, YARN-955-5.patch, YARN-955-6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1266) inheriting Application client and History Protocol from base protocol and implement PB service and clients.
[ https://issues.apache.org/jira/browse/YARN-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828179#comment-13828179 ] Zhijie Shen commented on YARN-1266: --- The patch looks good except the duplication of client impl. Anyway, it's good block work towards end-to-end system. Let's live with it, and revisit the problem later. +1 > inheriting Application client and History Protocol from base protocol and > implement PB service and clients. > --- > > Key: YARN-1266 > URL: https://issues.apache.org/jira/browse/YARN-1266 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-1266-1.patch, YARN-1266-2.patch, YARN-1266-3.patch, > YARN-1266-4.patch, YARN-1266-5.patch > > > Adding ApplicationHistoryProtocolPBService to make web apps to work and > changing yarn to run AHS as a seprate process -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1425) TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument
[ https://issues.apache.org/jira/browse/YARN-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828159#comment-13828159 ] Hudson commented on YARN-1425: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4770 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4770/]) YARN-1425. TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument (Omkar Vinit Joshi via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543952) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > TestRMRestart fails because MockRM.waitForState(AttemptId) uses current > attempt instead of the attempt passed as argument > - > > Key: YARN-1425 > URL: https://issues.apache.org/jira/browse/YARN-1425 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.3.0 > > Attachments: YARN-1425.1.patch, error.log > > > TestRMRestart is failing on trunk. Fixing it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828155#comment-13828155 ] Omkar Vinit Joshi commented on YARN-713: Also fixed YARN-1417 as a part of this. It is straight forward. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1242) Script changes to start AHS as an individual process
[ https://issues.apache.org/jira/browse/YARN-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828156#comment-13828156 ] Mayank Bansal commented on YARN-1242: - bq. Thanks for verifying the script. So starting/stopping the AHS (as daemon or not daemon) works properly, right? I have only tried as a seprate daemon. with RM on same box or on different box. Thanks, Mayank > Script changes to start AHS as an individual process > > > Key: YARN-1242 > URL: https://issues.apache.org/jira/browse/YARN-1242 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: YARN-1242-1.patch, YARN-1242-2.patch, YARN-1242-3.patch > > > Add the command in yarn and yarn.cmd to start and stop AHS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-955) [YARN-321] Implementation of ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-955: --- Attachment: YARN-955-6.patch Updating the latest patch Thanks, Mayank > [YARN-321] Implementation of ApplicationHistoryProtocol > --- > > Key: YARN-955 > URL: https://issues.apache.org/jira/browse/YARN-955 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Mayank Bansal > Attachments: YARN-955-1.patch, YARN-955-2.patch, YARN-955-3.patch, > YARN-955-4.patch, YARN-955-5.patch, YARN-955-6.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-955) [YARN-321] Implementation of ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828153#comment-13828153 ] Mayank Bansal commented on YARN-955: Thanks [~vinodkv] for the review. bq. One nit: Can you rename historyService and historyContext to historyManager? Done Thanks, Mayank > [YARN-321] Implementation of ApplicationHistoryProtocol > --- > > Key: YARN-955 > URL: https://issues.apache.org/jira/browse/YARN-955 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Mayank Bansal > Attachments: YARN-955-1.patch, YARN-955-2.patch, YARN-955-3.patch, > YARN-955-4.patch, YARN-955-5.patch, YARN-955-6.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1053) Diagnostic message from ContainerExitEvent is ignored in ContainerImpl
[ https://issues.apache.org/jira/browse/YARN-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-1053: Attachment: YARN-1053.1.patch Thanks [~bikassaha] .. Added a null check and also updated test case which verifies both diagnostic message and exitCode. > Diagnostic message from ContainerExitEvent is ignored in ContainerImpl > -- > > Key: YARN-1053 > URL: https://issues.apache.org/jira/browse/YARN-1053 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0, 2.2.1 >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Labels: newbie > Fix For: 2.3.0, 2.2.1 > > Attachments: YARN-1053.1.patch, YARN-1053.20130809.patch > > > If the container launch fails then we send ContainerExitEvent. This event > contains exitCode and diagnostic message. Today we are ignoring diagnostic > message while handling this event inside ContainerImpl. Fixing it as it is > useful in diagnosing the failure. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-1429) YARN_CLASSPATH is referenced in yarn command comments but doesn't do anything
Sandy Ryza created YARN-1429: Summary: YARN_CLASSPATH is referenced in yarn command comments but doesn't do anything Key: YARN-1429 URL: https://issues.apache.org/jira/browse/YARN-1429 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sandy Ryza Priority: Trivial YARN_CLASSPATH is referenced in the comments in ./hadoop-yarn-project/hadoop-yarn/bin/yarn and ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1425) TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument
[ https://issues.apache.org/jira/browse/YARN-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-1425: - Summary: TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument (was: TestRMRestart is failing on trunk) > TestRMRestart fails because MockRM.waitForState(AttemptId) uses current > attempt instead of the attempt passed as argument > - > > Key: YARN-1425 > URL: https://issues.apache.org/jira/browse/YARN-1425 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-1425.1.patch, error.log > > > TestRMRestart is failing on trunk. Fixing it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828106#comment-13828106 ] Sandy Ryza commented on YARN-1197: -- +1 on the current design. If possible it would be best to split this up between a couple JIRAs - one for the increase and one for the decrease. We could do a third for the common proto definitions if necessary. I believe that YARN does memory enforcement in the same way whether or not the LCE is used. I.e. it monitors the container process and kills it if its memory consumption goes over the configured limit. > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: yarn-1197-v2.pdf, yarn-1197-v3.pdf, yarn-1197-v4.pdf, > yarn-1197.pdf > > > Currently, YARN cannot support merge several containers in one node to a big > container, which can make us incrementally ask resources, merge them to a > bigger one, and launch our processes. The user scenario is described in the > comments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1266) inheriting Application client and History Protocol from base protocol and implement PB service and clients.
[
https://issues.apache.org/jira/browse/YARN-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828098#comment-13828098
]
Hadoop QA commented on YARN-1266:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614986/YARN-1266-5.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2498//console
This message is automatically generated.
> inheriting Application client and History Protocol from base protocol and
> implement PB service and clients.
> ---
>
> Key: YARN-1266
> URL: https://issues.apache.org/jira/browse/YARN-1266
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Mayank Bansal
> Attachments: YARN-1266-1.patch, YARN-1266-2.patch, YARN-1266-3.patch,
> YARN-1266-4.patch, YARN-1266-5.patch
>
>
> Adding ApplicationHistoryProtocolPBService to make web apps to work and
> changing yarn to run AHS as a seprate process
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1266) inheriting Application client and History Protocol from base protocol and implement PB service and clients.
[ https://issues.apache.org/jira/browse/YARN-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828094#comment-13828094 ] Mayank Bansal commented on YARN-1266: - Thanks [~vinodkv] and [~zjshen] for the review. bq. 1. Let's mark getApplicationReport/getApplications stable, though they are moved to base protocol. How do you think? Done bq. 2. In ApplicationBaseProtocol, please do not mention history sever. It contains both RM and AHS which is correct bq. 3. Where's ApplicationBaseProtocolPBClientImpl? bq. 4. Should you modify ApplicationClientProtocolPBClientImpl as well? bq. 5. ApplicationHistoryProtocolPBClientImpl should extend ApplicationBaseProtocolPBClientImpl. We discussed offline and we are planning not to add that. bq. I hope this is a compatible change - moving methods to a super-interface. Can you please check? Yes these are compatible changes. bq.Mark application_base_protocol.proto and ApplicationBaseProtocol as some kind of internal interfaces not to be used directly? May be by making them only package private and protected or something. Along with that we'll need to put correct audience and visibility annotations. bq. Similarly for the client and service wrappers. Done > inheriting Application client and History Protocol from base protocol and > implement PB service and clients. > --- > > Key: YARN-1266 > URL: https://issues.apache.org/jira/browse/YARN-1266 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-1266-1.patch, YARN-1266-2.patch, YARN-1266-3.patch, > YARN-1266-4.patch, YARN-1266-5.patch > > > Adding ApplicationHistoryProtocolPBService to make web apps to work and > changing yarn to run AHS as a seprate process -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1266) inheriting Application client and History Protocol from base protocol and implement PB service and clients.
[ https://issues.apache.org/jira/browse/YARN-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-1266: Attachment: YARN-1266-5.patch Attaching Latest Patch. Thanks, Mayank > inheriting Application client and History Protocol from base protocol and > implement PB service and clients. > --- > > Key: YARN-1266 > URL: https://issues.apache.org/jira/browse/YARN-1266 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-1266-1.patch, YARN-1266-2.patch, YARN-1266-3.patch, > YARN-1266-4.patch, YARN-1266-5.patch > > > Adding ApplicationHistoryProtocolPBService to make web apps to work and > changing yarn to run AHS as a seprate process -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1318) Promote AdminService to an Always-On service and merge in RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828070#comment-13828070 ] Karthik Kambatla commented on YARN-1318: Thanks [~vinodkv]. Sorry for the delay - I am out traveling Mon through Thu. Agree with your comments. Will post an updated patch as soon as I get back (tentatively Thursday night). > Promote AdminService to an Always-On service and merge in RMHAProtocolService > - > > Key: YARN-1318 > URL: https://issues.apache.org/jira/browse/YARN-1318 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > Labels: ha > Attachments: yarn-1318-0.patch, yarn-1318-1.patch, yarn-1318-2.patch, > yarn-1318-2.patch, yarn-1318-3.patch > > > Per discussion in YARN-1068, we want AdminService to handle HA-admin > operations in addition to the regular non-HA admin operations. To facilitate > this, we need to move AdminService an Always-On service. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1181) Augment MiniYARNCluster to support HA mode
[ https://issues.apache.org/jira/browse/YARN-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828071#comment-13828071 ] Karthik Kambatla commented on YARN-1181: Will take a look at the failed tests in a couple of days. > Augment MiniYARNCluster to support HA mode > -- > > Key: YARN-1181 > URL: https://issues.apache.org/jira/browse/YARN-1181 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-1181-1.patch, yarn-1181-2.patch > > > MiniYARNHACluster, along the lines of MiniYARNCluster, is needed for > end-to-end HA tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1425) TestRMRestart is failing on trunk
[ https://issues.apache.org/jira/browse/YARN-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828041#comment-13828041 ] Omkar Vinit Joshi commented on YARN-1425: - yes yarn-tests are passing locally. > TestRMRestart is failing on trunk > - > > Key: YARN-1425 > URL: https://issues.apache.org/jira/browse/YARN-1425 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-1425.1.patch, error.log > > > TestRMRestart is failing on trunk. Fixing it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[
https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828016#comment-13828016
]
Hadoop QA commented on YARN-1303:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614953/YARN-1303.8.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2497//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2497//console
This message is automatically generated.
> Allow multiple commands separating with ";" in distributed-shell
>
>
> Key: YARN-1303
> URL: https://issues.apache.org/jira/browse/YARN-1303
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications/distributed-shell
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Fix For: 2.2.1
>
> Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch,
> YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch,
> YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch,
> YARN-1303.8.2.patch, YARN-1303.8.patch
>
>
> In shell, we can do "ls; ls" to run 2 commands at once.
> In distributed shell, this is not working. We should improve to allow this to
> occur. There are practical use cases that I know of to run multiple commands
> or to set environment variables before a command.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1303: Attachment: YARN-1303.8.2.patch > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.2.1 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch, YARN-1303.8.1.patch, > YARN-1303.8.2.patch, YARN-1303.8.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-967) [YARN-321] Command Line Interface(CLI) for Reading Application History Storage Data
[
https://issues.apache.org/jira/browse/YARN-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827975#comment-13827975
]
Zhijie Shen commented on YARN-967:
--
Reviewed the latest patch. Here're some comments
1. Change yarn.cmd accordingly.
2. Not necessary, no log is written in AHSClientImpl.
{code}
+ private static final Log LOG = LogFactory.getLog(AHSClientImpl.class);
{code}
3. Where're the following configurations? Defined in other patch?
{code}
+return conf.getSocketAddr(YarnConfiguration.AHS_ADDRESS,
+YarnConfiguration.DEFAULT_AHS_ADDRESS,
+YarnConfiguration.DEFAULT_AHS_PORT);
{code}
4. Should AHSClientImpl use YarnClient configurations?
{code}
+statePollIntervalMillis = conf.getLong(
+YarnConfiguration.YARN_CLIENT_APP_SUBMISSION_POLL_INTERVAL_MS,
+YarnConfiguration.DEFAULT_YARN_CLIENT_APP_SUBMISSION_POLL_INTERVAL_MS);
{code}
5. Is the following condition correct?
{code}
+} else if (args[0].compareToIgnoreCase(APPLICATION_ATTEMPT) == 0
+|| args[0].compareToIgnoreCase(CONTAINER) == 0) {
{code}
6. One important issue here is that the command change is incompatible. The
users' old shell scripts will be break given the change here. It's good to make
the command compatible. For example, by default, it's going to the info of the
application(s). Or at least, we need to document the new behavior of the
command. [~vinodkv], how do you say?
7. Rename it to appAttemptReportStr? Also the javadoc.
{code}
+PrintWriter appReportStr = new PrintWriter(baos);
{code}
{code}
+ * Prints the application report for an application id.
{code}
8. Fix the above issue for printContainerReport as well.
9. Does AHS RPC protocol throw not found exception as well? If not, I think
it's good to do that to keep consistent. Maybe do the same for
getApplicationAttemptReport and getContainerReport
{code}
+if (appReport == null) {
+ appReport = historyClient.getApplicationReport(ConverterUtils
+ .toApplicationId(applicationId));
+}
{code}
10. Check getApplications as well. Make getApplicationAttempts and
getContainers behave similarly. This and the one above are the server-side
changes. Probably you'd like to coordinate your other patches.
11. For listApplications, if the users want the applications in
FINISHED/FAILED/KILLED states, why not going to historyClient as well?
12. AHSProxy is using a bunch of RM configurations instead of AHS ones. By the
way, it seems AHSProxy is almost the same as RMProxy. Is it possible to reuse
the code instead of duplicating it?
13. In YarnCLI, should we make getter for historyClient as well, like client?
14. The mock doesn't need to be defined in get and invoked every time
get is called. Define it once, it will behave the same in the following.
{code}
when(mockAppResponse.getApplicationList()).thenReturn(reports);
{code}
15. It's better to mock multiple attempts/containers to test gets.
16. The modified part of ApplicationCLI needs to be tested as well.
> [YARN-321] Command Line Interface(CLI) for Reading Application History
> Storage Data
> ---
>
> Key: YARN-967
> URL: https://issues.apache.org/jira/browse/YARN-967
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Devaraj K
>Assignee: Mayank Bansal
> Attachments: YARN-967-1.patch, YARN-967-2.patch, YARN-967-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[
https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827830#comment-13827830
]
Hadoop QA commented on YARN-691:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614910/Yarn-691.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2495//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2495//console
This message is automatically generated.
> Invalid NaN values in Hadoop REST API JSON response
> ---
>
> Key: YARN-691
> URL: https://issues.apache.org/jira/browse/YARN-691
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 0.23.6, 2.0.4-alpha
>Reporter: Kendall Thrapp
>Assignee: Chen He
> Fix For: 2.3.0
>
> Attachments: Yarn-691.patch
>
>
> I've been occasionally coming across instances where Hadoop's Cluster
> Applications REST API
> (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API)
> has returned JSON that PHP's json_decode function failed to parse. I've
> tracked the syntax error down to the presence of the unquoted word NaN
> appearing as a value in the JSON. For example:
> "progress":NaN,
> NaN is not part of the JSON spec, so its presence renders the whole JSON
> string invalid. Hadoop needs to return something other than NaN in this case
> -- perhaps an empty string or the quoted string "NaN".
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1420) TestRMContainerAllocator#testUpdatedNodes fails
[
https://issues.apache.org/jira/browse/YARN-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-1420:
--
Attachment: YARN-1420.patch
> TestRMContainerAllocator#testUpdatedNodes fails
> ---
>
> Key: YARN-1420
> URL: https://issues.apache.org/jira/browse/YARN-1420
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Jonathan Eagles
> Attachments: YARN-1420.patch
>
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/console :
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.78 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
> Time elapsed: 3.125 sec <<< FAILURE!
> junit.framework.AssertionFailedError: null
> at junit.framework.Assert.fail(Assert.java:48)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertTrue(Assert.java:27)
> at
> org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:779)
> {code}
> This assertion fails:
> {code}
> Assert.assertTrue(allocator.getJobUpdatedNodeEvents().isEmpty());
> {code}
> The List returned by allocator.getJobUpdatedNodeEvents() is:
> [EventType: JOB_UPDATED_NODES]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated YARN-691: - Attachment: Yarn-691.patch > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp >Assignee: Chen He > Fix For: 2.3.0 > > Attachments: Yarn-691.patch > > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated YARN-691: - Attachment: (was: YARN-691.patch) > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp >Assignee: Chen He > Fix For: 2.3.0 > > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user
[ https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827776#comment-13827776 ] Larry McCay commented on YARN-1063: --- Hi Kyle, I am curious about the status of this work. > Winutils needs ability to create task as domain user > > > Key: YARN-1063 > URL: https://issues.apache.org/jira/browse/YARN-1063 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: trunk-win > Environment: Windows >Reporter: Kyle Leckie > Labels: security > Fix For: trunk-win > > Attachments: YARN-1063.patch > > > h1. Summary: > Securing a Hadoop cluster requires constructing some form of security > boundary around the processes executed in YARN containers. Isolation based on > Windows user isolation seems most feasible. This approach is similar to the > approach taken by the existing LinuxContainerExecutor. The current patch to > winutils.exe adds the ability to create a process as a domain user. > h1. Alternative Methods considered: > h2. Process rights limited by security token restriction: > On Windows access decisions are made by examining the security token of a > process. It is possible to spawn a process with a restricted security token. > Any of the rights granted by SIDs of the default token may be restricted. It > is possible to see this in action by examining the security tone of a > sandboxed process launch be a web browser. Typically the launched process > will have a fully restricted token and need to access machine resources > through a dedicated broker process that enforces a custom security policy. > This broker process mechanism would break compatibility with the typical > Hadoop container process. The Container process must be able to utilize > standard function calls for disk and network IO. I performed some work > looking at ways to ACL the local files to the specific launched without > granting rights to other processes launched on the same machine but found > this to be an overly complex solution. > h2. Relying on APP containers: > Recent versions of windows have the ability to launch processes within an > isolated container. Application containers are supported for execution of > WinRT based executables. This method was ruled out due to the lack of > official support for standard windows APIs. At some point in the future > windows may support functionality similar to BSD jails or Linux containers, > at that point support for containers should be added. > h1. Create As User Feature Description: > h2. Usage: > A new sub command was added to the set of task commands. Here is the syntax: > winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE] > Some notes: > * The username specified is in the format of "user@domain" > * The machine executing this command must be joined to the domain of the user > specified > * The domain controller must allow the account executing the command access > to the user information. For this join the account to the predefined group > labeled "Pre-Windows 2000 Compatible Access" > * The account running the command must have several rights on the local > machine. These can be managed manually using secpol.msc: > ** "Act as part of the operating system" - SE_TCB_NAME > ** "Replace a process-level token" - SE_ASSIGNPRIMARYTOKEN_NAME > ** "Adjust memory quotas for a process" - SE_INCREASE_QUOTA_NAME > * The launched process will not have rights to the desktop so will not be > able to display any information or create UI. > * The launched process will have no network credentials. Any access of > network resources that requires domain authentication will fail. > h2. Implementation: > Winutils performs the following steps: > # Enable the required privileges for the current process. > # Register as a trusted process with the Local Security Authority (LSA). > # Create a new logon for the user passed on the command line. > # Load/Create a profile on the local machine for the new logon. > # Create a new environment for the new logon. > # Launch the new process in a job with the task name specified and using the > created logon. > # Wait for the JOB to exit. > h2. Future work: > The following work was scoped out of this check in: > * Support for non-domain users or machine that are not domain joined. > * Support for privilege isolation by running the task launcher in a high > privilege service with access over an ACLed named pipe. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (YARN-1420) TestRMContainerAllocator#testUpdatedNodes fails
[
https://issues.apache.org/jira/browse/YARN-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles reassigned YARN-1420:
-
Assignee: Jonathan Eagles
> TestRMContainerAllocator#testUpdatedNodes fails
> ---
>
> Key: YARN-1420
> URL: https://issues.apache.org/jira/browse/YARN-1420
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Jonathan Eagles
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/console :
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.78 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
> Time elapsed: 3.125 sec <<< FAILURE!
> junit.framework.AssertionFailedError: null
> at junit.framework.Assert.fail(Assert.java:48)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertTrue(Assert.java:27)
> at
> org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:779)
> {code}
> This assertion fails:
> {code}
> Assert.assertTrue(allocator.getJobUpdatedNodeEvents().isEmpty());
> {code}
> The List returned by allocator.getJobUpdatedNodeEvents() is:
> [EventType: JOB_UPDATED_NODES]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1307) Rethink znode structure for RM HA
[
https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827754#comment-13827754
]
Tsuyoshi OZAWA commented on YARN-1307:
--
[~jianhe], OK, I'll update a patch to remove the version stuff.
> Rethink znode structure for RM HA
> -
>
> Key: YARN-1307
> URL: https://issues.apache.org/jira/browse/YARN-1307
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1307.1.patch, YARN-1307.2.patch, YARN-1307.3.patch,
> YARN-1307.4-2.patch, YARN-1307.4-3.patch, YARN-1307.4.patch,
> YARN-1307.5.patch, YARN-1307.6.patch
>
>
> Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659,
> YARN-1222). The motivation of this JIRA is quoted from Bikas' comment in
> YARN-1222:
> {quote}
> We should move to creating a node hierarchy for apps such that all znodes for
> an app are stored under an app znode instead of the app root znode. This will
> help in removeApplication and also in scaling better on ZK. The earlier code
> was written this way to ensure create/delete happens under a root znode for
> fencing. But given that we have moved to multi-operations globally, this isnt
> required anymore.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1423) Support queue placement by secondary group in the Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827740#comment-13827740
]
Hadoop QA commented on YARN-1423:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614894/YARN-1423.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2494//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2494//console
This message is automatically generated.
> Support queue placement by secondary group in the Fair Scheduler
>
>
> Key: YARN-1423
> URL: https://issues.apache.org/jira/browse/YARN-1423
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Sandy Ryza
> Attachments: YARN-1423.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-786) Expose application resource usage in RM REST API
[ https://issues.apache.org/jira/browse/YARN-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827713#comment-13827713 ] Jason Lowe commented on YARN-786: - Thanks for quickly addressing this, Sandy! > Expose application resource usage in RM REST API > > > Key: YARN-786 > URL: https://issues.apache.org/jira/browse/YARN-786 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.3.0 > > Attachments: YARN-786-1.patch, YARN-786-2.patch, > YARN-786-addendum.patch, YARN-786.patch > > > It might be good to require users to explicitly ask for this information, as > it's a little more expensive to collect than the other fields in AppInfo. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1423) Support queue placement by secondary group in the Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated YARN-1423: -- Attachment: YARN-1423.patch Added support for selecting a queue based on secondary groups. Modified the following files 1. FairScheduler.apt.vm 2. TestFairScheduler.java 3. SimpleGroupsMapping.java 4. QueuePlacementRule.java 5. QueuePlacementPolicy.java There are documentation, junits, and code changes in this patch > Support queue placement by secondary group in the Fair Scheduler > > > Key: YARN-1423 > URL: https://issues.apache.org/jira/browse/YARN-1423 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Sandy Ryza > Attachments: YARN-1423.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-786) Expose application resource usage in RM REST API
[ https://issues.apache.org/jira/browse/YARN-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827642#comment-13827642 ] Hudson commented on YARN-786: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1614 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1614/]) YARN-786: Addendum so that RMAppAttemptImpl#getApplicationResourceUsageReport won't return null (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543597) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java > Expose application resource usage in RM REST API > > > Key: YARN-786 > URL: https://issues.apache.org/jira/browse/YARN-786 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.3.0 > > Attachments: YARN-786-1.patch, YARN-786-2.patch, > YARN-786-addendum.patch, YARN-786.patch > > > It might be good to require users to explicitly ask for this information, as > it's a little more expensive to collect than the other fields in AppInfo. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1407) RM Web UI and REST APIs should uniformly use YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827638#comment-13827638 ] Hudson commented on YARN-1407: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1614 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1614/]) Move YARN-1407 under 2.2.1 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543681) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1407. RM Web UI and REST APIs should uniformly use YarnApplicationState (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543675) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm > RM Web UI and REST APIs should uniformly use YarnApplicationState > - > > Key: YARN-1407 > URL: https://issues.apache.org/jira/browse/YARN-1407 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1407-1.patch, YARN-1407-2.patch, YARN-1407.patch > > > RMAppState isn't a public facing enum like YarnApplicationState, so we > shouldn't return values or list filters that come from it. However, some > Blocks and AppInfo are still using RMAppState. > It is not 100% clear to me whether or not fixing this would be a > backwards-incompatible change. The change would only reduce the set of > possible strings that the API returns, so I think not. We have also been > changing the contents of RMAppState since 2.2.0, e.g. in YARN-891. It would > still be good to fix this ASAP (i.e. for 2.2.1). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827639#comment-13827639 ] Hudson commented on YARN-744: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1614 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1614/]) YARN-744. Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. (Omkar Vinit Joshi via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543707) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Race condition in ApplicationMasterService.allocate .. It might process same > allocate request twice resulting in additional containers getting allocated. > - > > Key: YARN-744 > URL: https://issues.apache.org/jira/browse/YARN-744 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Omkar Vinit Joshi >Priority: Minor > Attachments: MAPREDUCE-3899-branch-0.23.patch, > YARN-744-20130711.1.patch, YARN-744-20130715.1.patch, > YARN-744-20130726.1.patch, YARN-744.1.patch, YARN-744.2.patch, YARN-744.patch > > > Looks like the lock taken in this is broken. It takes a lock on lastResponse > object and then puts a new lastResponse object into the map. At this point a > new thread entering this function will get a new lastResponse object and will > be able to take its lock and enter the critical section. Presumably we want > to limit one response per app attempt. So the lock could be taken on the > ApplicationAttemptId key of the response map object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1407) RM Web UI and REST APIs should uniformly use YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827626#comment-13827626 ] Hudson commented on YARN-1407: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1588 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1588/]) Move YARN-1407 under 2.2.1 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543681) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1407. RM Web UI and REST APIs should uniformly use YarnApplicationState (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543675) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm > RM Web UI and REST APIs should uniformly use YarnApplicationState > - > > Key: YARN-1407 > URL: https://issues.apache.org/jira/browse/YARN-1407 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1407-1.patch, YARN-1407-2.patch, YARN-1407.patch > > > RMAppState isn't a public facing enum like YarnApplicationState, so we > shouldn't return values or list filters that come from it. However, some > Blocks and AppInfo are still using RMAppState. > It is not 100% clear to me whether or not fixing this would be a > backwards-incompatible change. The change would only reduce the set of > possible strings that the API returns, so I think not. We have also been > changing the contents of RMAppState since 2.2.0, e.g. in YARN-891. It would > still be good to fix this ASAP (i.e. for 2.2.1). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827627#comment-13827627 ] Hudson commented on YARN-744: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1588 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1588/]) YARN-744. Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. (Omkar Vinit Joshi via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543707) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Race condition in ApplicationMasterService.allocate .. It might process same > allocate request twice resulting in additional containers getting allocated. > - > > Key: YARN-744 > URL: https://issues.apache.org/jira/browse/YARN-744 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Omkar Vinit Joshi >Priority: Minor > Attachments: MAPREDUCE-3899-branch-0.23.patch, > YARN-744-20130711.1.patch, YARN-744-20130715.1.patch, > YARN-744-20130726.1.patch, YARN-744.1.patch, YARN-744.2.patch, YARN-744.patch > > > Looks like the lock taken in this is broken. It takes a lock on lastResponse > object and then puts a new lastResponse object into the map. At this point a > new thread entering this function will get a new lastResponse object and will > be able to take its lock and enter the critical section. Presumably we want > to limit one response per app attempt. So the lock could be taken on the > ApplicationAttemptId key of the response map object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-786) Expose application resource usage in RM REST API
[ https://issues.apache.org/jira/browse/YARN-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827630#comment-13827630 ] Hudson commented on YARN-786: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1588 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1588/]) YARN-786: Addendum so that RMAppAttemptImpl#getApplicationResourceUsageReport won't return null (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543597) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java > Expose application resource usage in RM REST API > > > Key: YARN-786 > URL: https://issues.apache.org/jira/browse/YARN-786 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.3.0 > > Attachments: YARN-786-1.patch, YARN-786-2.patch, > YARN-786-addendum.patch, YARN-786.patch > > > It might be good to require users to explicitly ask for this information, as > it's a little more expensive to collect than the other fields in AppInfo. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-786) Expose application resource usage in RM REST API
[ https://issues.apache.org/jira/browse/YARN-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827541#comment-13827541 ] Hudson commented on YARN-786: - SUCCESS: Integrated in Hadoop-Yarn-trunk #397 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/397/]) YARN-786: Addendum so that RMAppAttemptImpl#getApplicationResourceUsageReport won't return null (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543597) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java > Expose application resource usage in RM REST API > > > Key: YARN-786 > URL: https://issues.apache.org/jira/browse/YARN-786 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.3.0 > > Attachments: YARN-786-1.patch, YARN-786-2.patch, > YARN-786-addendum.patch, YARN-786.patch > > > It might be good to require users to explicitly ask for this information, as > it's a little more expensive to collect than the other fields in AppInfo. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-744) Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated.
[ https://issues.apache.org/jira/browse/YARN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827538#comment-13827538 ] Hudson commented on YARN-744: - SUCCESS: Integrated in Hadoop-Yarn-trunk #397 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/397/]) YARN-744. Race condition in ApplicationMasterService.allocate .. It might process same allocate request twice resulting in additional containers getting allocated. (Omkar Vinit Joshi via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543707) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Race condition in ApplicationMasterService.allocate .. It might process same > allocate request twice resulting in additional containers getting allocated. > - > > Key: YARN-744 > URL: https://issues.apache.org/jira/browse/YARN-744 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Omkar Vinit Joshi >Priority: Minor > Attachments: MAPREDUCE-3899-branch-0.23.patch, > YARN-744-20130711.1.patch, YARN-744-20130715.1.patch, > YARN-744-20130726.1.patch, YARN-744.1.patch, YARN-744.2.patch, YARN-744.patch > > > Looks like the lock taken in this is broken. It takes a lock on lastResponse > object and then puts a new lastResponse object into the map. At this point a > new thread entering this function will get a new lastResponse object and will > be able to take its lock and enter the critical section. Presumably we want > to limit one response per app attempt. So the lock could be taken on the > ApplicationAttemptId key of the response map object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1407) RM Web UI and REST APIs should uniformly use YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827537#comment-13827537 ] Hudson commented on YARN-1407: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #397 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/397/]) Move YARN-1407 under 2.2.1 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543681) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1407. RM Web UI and REST APIs should uniformly use YarnApplicationState (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543675) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm > RM Web UI and REST APIs should uniformly use YarnApplicationState > - > > Key: YARN-1407 > URL: https://issues.apache.org/jira/browse/YARN-1407 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1407-1.patch, YARN-1407-2.patch, YARN-1407.patch > > > RMAppState isn't a public facing enum like YarnApplicationState, so we > shouldn't return values or list filters that come from it. However, some > Blocks and AppInfo are still using RMAppState. > It is not 100% clear to me whether or not fixing this would be a > backwards-incompatible change. The change would only reduce the set of > possible strings that the API returns, so I think not. We have also been > changing the contents of RMAppState since 2.2.0, e.g. in YARN-891. It would > still be good to fix this ASAP (i.e. for 2.2.1). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (YARN-936) RMWebServices filtering apps by states uses RMAppState instead of YarnAppilcationState.
[ https://issues.apache.org/jira/browse/YARN-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved YARN-936. - Resolution: Duplicate > RMWebServices filtering apps by states uses RMAppState instead of > YarnAppilcationState. > --- > > Key: YARN-936 > URL: https://issues.apache.org/jira/browse/YARN-936 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli > > Realized this while reviewing YARN-696. YarnApplicationState is the end user > API and one that users expect to pass as argument to the REST API. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1407) RM Web UI and REST APIs should uniformly use YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827443#comment-13827443 ] Sandy Ryza commented on YARN-1407: -- Didn't notice that - yeah, looks like it is. Just closed it as duplicate. > RM Web UI and REST APIs should uniformly use YarnApplicationState > - > > Key: YARN-1407 > URL: https://issues.apache.org/jira/browse/YARN-1407 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1407-1.patch, YARN-1407-2.patch, YARN-1407.patch > > > RMAppState isn't a public facing enum like YarnApplicationState, so we > shouldn't return values or list filters that come from it. However, some > Blocks and AppInfo are still using RMAppState. > It is not 100% clear to me whether or not fixing this would be a > backwards-incompatible change. The change would only reduce the set of > possible strings that the API returns, so I think not. We have also been > changing the contents of RMAppState since 2.2.0, e.g. in YARN-891. It would > still be good to fix this ASAP (i.e. for 2.2.1). -- This message was sent by Atlassian JIRA (v6.1#6144)
