[jira] [Updated] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-417: Issue Type: Sub-task (was: New Feature) Parent: YARN-418 Add a poller that allows the AM to receive notifications when it is assigned containers --- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-417.patch Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-424) Allow resource requests to provide hints to the scheduler
Sandy Ryza created YARN-424: --- Summary: Allow resource requests to provide hints to the scheduler Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585744#comment-13585744 ] Sandy Ryza commented on YARN-392: - Filed YARN-424 for Alejandro's hints proposal. Make it possible to schedule to specific nodes without dropping locality Key: YARN-392 URL: https://issues.apache.org/jira/browse/YARN-392 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Sandy Ryza Attachments: YARN-392.patch Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-111) Application level priority in Resource Manager Schedulers
[ https://issues.apache.org/jira/browse/YARN-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou resolved YARN-111. Resolution: Won't Fix Application level priority in Resource Manager Schedulers - Key: YARN-111 URL: https://issues.apache.org/jira/browse/YARN-111 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: nemon lou We need application level priority for Hadoop 2.0,both in FIFO scheduler and Capacity Scheduler. In Hadoop 1.0.x,job priority is supported. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-425) coverage fix for yarn api
[
https://issues.apache.org/jira/browse/YARN-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585821#comment-13585821
]
Hadoop QA commented on YARN-425:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570775/YARN-425-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/426//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/426//console
This message is automatically generated.
coverage fix for yarn api
-
Key: YARN-425
URL: https://issues.apache.org/jira/browse/YARN-425
Project: Hadoop YARN
Issue Type: Test
Affects Versions: 3.0.0, 0.23.7, 2.0.4-beta
Reporter: Aleksey Gorshkov
Attachments: YARN-425-branch-0.23.patch, YARN-425-branch-2.patch,
YARN-425-trunk.patch
coverage fix for yarn api
patch YARN-425-trunk.patch for trunk
patch YARN-425-branch-2.patch for branch-2
patch YARN-425-branch-0.23.patch for branch-0.23
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-379) yarn [node,application] command print logger info messages
[ https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-379: --- Assignee: Abhishek Kapoor yarn [node,application] command print logger info messages -- Key: YARN-379 URL: https://issues.apache.org/jira/browse/YARN-379 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability Running the yarn node and yarn applications command results in annoying log info messages being printed: $ yarn node -list 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Total Nodes:1 Node-IdNode-State Node-Http-Address Health-Status(isNodeHealthy)Running-Containers foo:8041RUNNING foo:8042 true 0 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. $ yarn application 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Invalid Command Usage : usage: application -kill arg Kills the application. -list Lists all the Applications from RM. -status arg Prints the status of the application. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages
[ https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585949#comment-13585949 ] Thomas Graves commented on YARN-379: I added you as a contributor to the YARN project and assigned it to you. Thanks for taking this up. yarn [node,application] command print logger info messages -- Key: YARN-379 URL: https://issues.apache.org/jira/browse/YARN-379 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability Running the yarn node and yarn applications command results in annoying log info messages being printed: $ yarn node -list 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Total Nodes:1 Node-IdNode-State Node-Http-Address Health-Status(isNodeHealthy)Running-Containers foo:8041RUNNING foo:8042 true 0 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. $ yarn application 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Invalid Command Usage : usage: application -kill arg Kills the application. -list Lists all the Applications from RM. -status arg Prints the status of the application. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585965#comment-13585965 ] Alejandro Abdelnur commented on YARN-424: - Regarding the format/type for hints, while using 16 bytes would do, I'd prefer to use something that does not require having to deal with byte masking. While chatting about this, Bikas suggested a byte[], that woud be certainly much better. Still, I think having a capped String (50 or 100 chars) would something much easier to deal with the plus that printing a hint in the logs would be straight forward. My take is that hints should be a comma separated list of key/values or values * HINTS - ΓΈ | HINT | HINTS,HINT * HINT - VALUE | KEY=VALUE * KEY - word * VALUE - word Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-345) Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager
[
https://issues.apache.org/jira/browse/YARN-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Parker updated YARN-345:
---
Attachment: YARN-345.patch
Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager
--
Key: YARN-345
URL: https://issues.apache.org/jira/browse/YARN-345
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha, 0.23.5
Reporter: Devaraj K
Assignee: Robert Parker
Priority: Critical
Attachments: YARN-345.patch
{code:xml}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 04:03:46,726 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at APPLICATION_RESOURCES_CLEANINGUP
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 00:01:11,006 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHING_CONTAINERS_WAIT
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 10:56:36,975 INFO
[jira] [Commented] (YARN-345) Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager
[
https://issues.apache.org/jira/browse/YARN-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585981#comment-13585981
]
Hadoop QA commented on YARN-345:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570807/YARN-345.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/427//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/427//console
This message is automatically generated.
Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager
--
Key: YARN-345
URL: https://issues.apache.org/jira/browse/YARN-345
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha, 0.23.5
Reporter: Devaraj K
Assignee: Robert Parker
Priority: Critical
Attachments: YARN-345.patch
{code:xml}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 04:03:46,726 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at APPLICATION_RESOURCES_CLEANINGUP
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 00:01:11,006 WARN
[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages
[ https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586002#comment-13586002 ] Abhishek Kapoor commented on YARN-379: -- It order to fix this request, two files will be tweaked File name: 1) yarn 2) yarn-env.sh I have added a new OPTS called COMMON_LOGGING_OPTS in yarn-env.sh to turn off the unnecessary log info messages. COMMON_LOGGING_OPTS is used into the yarn shell script to turn the log message. Suggestion are welcome yarn [node,application] command print logger info messages -- Key: YARN-379 URL: https://issues.apache.org/jira/browse/YARN-379 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability Running the yarn node and yarn applications command results in annoying log info messages being printed: $ yarn node -list 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Total Nodes:1 Node-IdNode-State Node-Http-Address Health-Status(isNodeHealthy)Running-Containers foo:8041RUNNING foo:8042 true 0 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. $ yarn application 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Invalid Command Usage : usage: application -kill arg Kills the application. -list Lists all the Applications from RM. -status arg Prints the status of the application. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-420) Enable the RM to work with AM's that are not managed by it
[ https://issues.apache.org/jira/browse/YARN-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-420: --- Labels: incom mrv2 (was: mrv2) Enable the RM to work with AM's that are not managed by it -- Key: YARN-420 URL: https://issues.apache.org/jira/browse/YARN-420 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Labels: incom, mrv2 Fix For: 2.0.2-alpha Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work. It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier. 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-421) Create asynchronous AM RM Client
[ https://issues.apache.org/jira/browse/YARN-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586033#comment-13586033 ] Chris Riccomini commented on YARN-421: -- Seems like it might be. I'm going to comment on 417 with my feedback. Create asynchronous AM RM Client Key: YARN-421 URL: https://issues.apache.org/jira/browse/YARN-421 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Bikas Saha The basic AM RM client hides the protocol and does request translation for the RM protocol. Creating a version of the client that can hide the heartbeat and provide notifications about interesting events like receiving containers etc will help unburden app developers from implementing this common scenario and focus on their app logic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586051#comment-13586051 ] Bikas Saha commented on YARN-417: - Sandy, it will be beneficial if you can mention if the patch is work in progress (like this one seems) or not. And also describe it a little bit more clearly. Add a poller that allows the AM to receive notifications when it is assigned containers --- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-417.patch Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages
[ https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586086#comment-13586086 ] Abhishek Kapoor commented on YARN-379: -- Do I need any special permission to attach a patch file for this jira ? Sorry this is my first time. Appreciated yarn [node,application] command print logger info messages -- Key: YARN-379 URL: https://issues.apache.org/jira/browse/YARN-379 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability Fix For: trunk-win Running the yarn node and yarn applications command results in annoying log info messages being printed: $ yarn node -list 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Total Nodes:1 Node-IdNode-State Node-Http-Address Health-Status(isNodeHealthy)Running-Containers foo:8041RUNNING foo:8042 true 0 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. $ yarn application 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Invalid Command Usage : usage: application -kill arg Kills the application. -list Lists all the Applications from RM. -status arg Prints the status of the application. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-379) yarn [node,application] command print logger info messages
[ https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kapoor updated YARN-379: - Attachment: YARN-379.patch Please review the code. Many Thanks yarn [node,application] command print logger info messages -- Key: YARN-379 URL: https://issues.apache.org/jira/browse/YARN-379 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability Fix For: trunk-win Attachments: YARN-379.patch Running the yarn node and yarn applications command results in annoying log info messages being printed: $ yarn node -list 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Total Nodes:1 Node-IdNode-State Node-Http-Address Health-Status(isNodeHealthy)Running-Containers foo:8041RUNNING foo:8042 true 0 13/02/06 02:36:50 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. $ yarn application 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. Invalid Command Usage : usage: application -kill arg Kills the application. -list Lists all the Applications from RM. -status arg Prints the status of the application. 13/02/06 02:38:47 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586092#comment-13586092 ] Bikas Saha commented on YARN-417: - My first comment would be to see if we can provide this functionality via a single object instead of a combination of AMRMClient, Heartbeater and AllocationListener. Every new object and its combination with others increases mental complexity. Also, while callbacks are a reasonable approach, it would be good to spend a little time thinking about other notification mechanisms like events or futures. Something else to think about. completedContainers and allocatedContainers are essentially coming at the same time. Breaking them into 2 calls may open up ordering issues. What is the use of allocationResponse? If we call it on every heartbeat then we are implicitly exposing it anyways. I can see the point about flexibility of going one way or the other, but in this case, I think we should take a stand about what exactly we want to expose to the user to make their life easy. Lets figure out what the user interaction surface looks like and carefully create new APIs. Add a poller that allows the AM to receive notifications when it is assigned containers --- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-417.patch Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated YARN-417: - Attachment: YarnAppMasterListener.java YarnAppMaster.java Hey Guys, I'm attaching two files, to provide some perspective. This is how I handle dealing with the YARN AM, right now. Sandy, I think your patch is following a similar idea, but models a little bit less of the lifecycle. One open question, regarding how I've set things up, is whether it makes sense to just have onContainerComplete, as it is in the patch, or have onContainerLost/onContainerFail/onContainerSuccess. I think the latter makes thins easier, since YARN has some pre-defined return codes (e.g. -100) that are a bit confusing to deal with if you don't read the code/Javadocs. Lastly, my patch isn't using the AMRMClient patch that was recently added - it's using the raw API plus my own internal application master helper. Should probably be converted to use the new AMRMClient. Cheers, Chris Add a poller that allows the AM to receive notifications when it is assigned containers --- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-426) Failure to download a public resource on a node prevents further downloads of the resource from that node
Jason Lowe created YARN-426:
---
Summary: Failure to download a public resource on a node prevents
further downloads of the resource from that node
Key: YARN-426
URL: https://issues.apache.org/jira/browse/YARN-426
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.6, 2.0.3-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
If the NM encounters an error while downloading a public resource, it fails to
empty the list of request events corresponding to the resource request in
{{attempts}}. If the same public resource is subsequently requested on that
node, {{PublicLocalizer.addResource}} will skip the download since it will
mistakenly believe a download of that resource is already in progress. At that
point any container that requests the public resource will just hang in the
{{LOCALIZING}} state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586278#comment-13586278 ] Siddharth Seth commented on YARN-365: - Almost there. Couple more fixes needed though. - nextHeartbeat is being accessed by multiple threads (scheduler and main dispatcher). Should be volatile. - There's a race in handling nextHeartBeat. In the StatusUpdateWhenHealthyTransition, nextHeartBeat should be set to false before generating the event for the scheduler. Otherwise, there's a race between the scheduler event being processed (and setting the value to true) and the value being set to false. - Are the updates required to MockNodes.java. They don't seem to be used anywhere. - testStatusChanged can be modified to check the value of the queue size before the EXPIRE event. i.e. validate that events are being queued correctly. Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-365: --- Attachment: YARN-365.9.patch Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-390) ApplicationCLI and NodeCLI use hard-coded platform-specific line separator, which causes test failures on Windows
[
https://issues.apache.org/jira/browse/YARN-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586513#comment-13586513
]
Suresh Srinivas commented on YARN-390:
--
+1 for the patch. I will commit this shortly.
ApplicationCLI and NodeCLI use hard-coded platform-specific line separator,
which causes test failures on Windows
-
Key: YARN-390
URL: https://issues.apache.org/jira/browse/YARN-390
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: YARN-390.1.patch, YARN-390.2.patch
{{ApplicationCLI}}, {{NodeCLI}}, and the corresponding test {{TestYarnCLI}}
all use a hard-coded '\n' as the line separator. This causes test failures
on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-390) ApplicationCLI and NodeCLI use hard-coded platform-specific line separator, which causes test failures on Windows
[
https://issues.apache.org/jira/browse/YARN-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586524#comment-13586524
]
Hudson commented on YARN-390:
-
Integrated in Hadoop-trunk-Commit #3381 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3381/])
YARN-390. ApplicationCLI and NodeCLI hard-coded platform-specific line
separator causes test failures on Windows. Contributed by Chris Nauroth.
(Revision 1449980)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1449980
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
ApplicationCLI and NodeCLI use hard-coded platform-specific line separator,
which causes test failures on Windows
-
Key: YARN-390
URL: https://issues.apache.org/jira/browse/YARN-390
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: YARN-390.1.patch, YARN-390.2.patch
{{ApplicationCLI}}, {{NodeCLI}}, and the corresponding test {{TestYarnCLI}}
all use a hard-coded '\n' as the line separator. This causes test failures
on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586526#comment-13586526 ] Arun C Murthy commented on YARN-424: I'm not a fan of stringly-typed 'features' in the Scheduler. We should focus on actual use-cases and come up with well thought out solutions to those - MapReduce configuration is a very good example of how using key/value pairs leads us down to crazy nooks and corners. Gang-scheduling, deadline-scheduling etc. are core-features in the scheduler, and should be treated as such i.e. we should explicitly support them in the protocol - not as string hints. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586529#comment-13586529 ] Arun C Murthy commented on YARN-392: Gang-scheduling, deadline-scheduling etc. are not *hints*. They are core features in the scheduler and should be treated as such i.e. we should explicitly support them in the scheduler. I've commented on YARN-424 too. Make it possible to schedule to specific nodes without dropping locality Key: YARN-392 URL: https://issues.apache.org/jira/browse/YARN-392 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Sandy Ryza Attachments: YARN-392.patch Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586535#comment-13586535 ] Arun C Murthy commented on YARN-424: Also, the reason we have invested in using protobufs (or any sane serialization system) is that we can *add* features to the Scheduler protocol in a compatible manner when we need to - key/value pairs are evil, and completely unnecessary. I've had enough with key/value pairs, and then attendant demands to maintain compatibility for those key/value pairs forever. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586544#comment-13586544 ] Arun C Murthy commented on YARN-392: Also, we don't have to support *all* features in the YARN scheduler on day one - let's be judicious and careful. We need to ship first stable release of YARN (which we are close), learn from our experiences and then add things. Make it possible to schedule to specific nodes without dropping locality Key: YARN-392 URL: https://issues.apache.org/jira/browse/YARN-392 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Sandy Ryza Attachments: YARN-392.patch Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586560#comment-13586560 ] Bikas Saha commented on YARN-424: - I agree that being able to extend the API without breaking back-compat is important. I am +1 for Arun's argument about protobuf's being the right approach to that. There is no need to add a string or any other hint. Simple add an optional member to the AllocateRequest which the new scheduler will understand. Other schedulers simply dont know about the new field. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[
https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586571#comment-13586571
]
Hadoop QA commented on YARN-365:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570885/YARN-365.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/429//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/429//console
This message is automatically generated.
Each NM heartbeat should not generate an event for the Scheduler
Key: YARN-365
URL: https://issues.apache.org/jira/browse/YARN-365
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Affects Versions: 0.23.5
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch,
YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch,
YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch
Follow up from YARN-275
https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-398) Allow white-list and black-list of resources
[ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586576#comment-13586576 ] Arun C Murthy commented on YARN-398: Here is an approach I have a prelim patch on which I'll put up shortly... Essentially, we need an extra (boolean?) field per ResourceRequest (RR) called 'blacklist' or, essentially, 'do not fall through'. What this provides is a way to tell the Scheduler to stop it's current fall-through scheduling mechanism of node-to-rack-ANY ... When the flag is set, the scheduler should *not* fall-through and look for rack-locality or ANY. Example scenarios: # if you want to blacklist a node, set the flag on RR of the node # if you want to blacklist a rack, set the flag on the RR of the rack (obviously do not send any RRs for any nodes in the rack) # if you want to whitelist a node (or a set of them): blacklist ANY and either don't pass other rack RRs or blacklist all rack RRs (for those nodes you care about). Obviously, this works across priorities since each RR is per-priority. Essentially, this flag provides the ability for the application to ask the scheduler to disable it's 'locality relaxation' or 'fall-through'. This way, we don't radically re-design our protocol in YARN v1 (i.e. hadoop-2.x) and yet provide blacklist/whitelist for individual and a set of nodes/racks. Clearly, we can make this easier for end-users by adding a simple client api which takes care by providing apis like: blacklistNodes/whitelistNodes and does it automatically for each priority simplifying the application writer's life. Thoughts? Do folks see any long-term (i.e. hadoop-2.x), negative repercussions due to this? Allow white-list and black-list of resources Key: YARN-398 URL: https://issues.apache.org/jira/browse/YARN-398 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Allow white-list and black-list of resources in scheduler api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586579#comment-13586579 ] Arun C Murthy commented on YARN-392: Take a look at YARN-398 for a simpler alternate proposal without needing to resort to hints etc. Make it possible to schedule to specific nodes without dropping locality Key: YARN-392 URL: https://issues.apache.org/jira/browse/YARN-392 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Sandy Ryza Attachments: YARN-392.patch Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586601#comment-13586601 ] Siddharth Seth commented on YARN-365: - +1. Committing this, and ignoring the jenkins -1 since the patch adds a timeout to each of the tests it has modified. Thanks Xuan! Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586603#comment-13586603 ] Siddharth Seth commented on YARN-365: - One minor refactor, for which I'm uploading a patch. Renaming getContainerUpdates to pullContainerUpdates to be consistent with similar interfaces elsewhere. Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated YARN-365: Attachment: YARN-365.10.patch Updated with the renamed method, and some javadoc. Will commit post jenkins. Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-398) Allow white-list and black-list of resources
[ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586615#comment-13586615 ] Alejandro Abdelnur commented on YARN-398: - Arun, how is this different from option #1 in YARN-392? Allow white-list and black-list of resources Key: YARN-398 URL: https://issues.apache.org/jira/browse/YARN-398 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Allow white-list and black-list of resources in scheduler api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[
https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586635#comment-13586635
]
Hadoop QA commented on YARN-365:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570908/YARN-365.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/430//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/430//console
This message is automatically generated.
Each NM heartbeat should not generate an event for the Scheduler
Key: YARN-365
URL: https://issues.apache.org/jira/browse/YARN-365
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Affects Versions: 0.23.5
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch,
YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch,
YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch,
YARN-365.9.patch
Follow up from YARN-275
https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[
https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586637#comment-13586637
]
Hadoop QA commented on YARN-365:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570908/YARN-365.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/431//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/431//console
This message is automatically generated.
Each NM heartbeat should not generate an event for the Scheduler
Key: YARN-365
URL: https://issues.apache.org/jira/browse/YARN-365
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Affects Versions: 0.23.5
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch,
YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch,
YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch,
YARN-365.9.patch
Follow up from YARN-275
https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[
https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586650#comment-13586650
]
Hadoop QA commented on YARN-365:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570908/YARN-365.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/432//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/432//console
This message is automatically generated.
Each NM heartbeat should not generate an event for the Scheduler
Key: YARN-365
URL: https://issues.apache.org/jira/browse/YARN-365
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager, scheduler
Affects Versions: 0.23.5
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch,
YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch,
YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch,
YARN-365.9.patch
Follow up from YARN-275
https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586700#comment-13586700 ] Hudson commented on YARN-365: - Integrated in Hadoop-trunk-Commit #3384 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3384/]) YARN-365. Change NM heartbeat handling to not generate a scheduler event on each heartbeat. (Contributed by Xuan Gong) (Revision 1450007) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1450007 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/UpdatedContainerInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/NodeUpdateSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Fix For: 2.0.4-beta Attachments: Prototype2.txt, Prototype3.txt, YARN-365.10.patch, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch, YARN-365.7.patch, YARN-365.8.patch, YARN-365.9.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[
https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586702#comment-13586702
]
Sandy Ryza commented on YARN-417:
-
Thanks for the comments Bikas, and for your perspective, Chris.
Chris,
Your code brings up that we should probably have a reboot callback as well. I
also like the idea of onContainerLost/onContainerFail/onContainerSuccess. Do
you have an opinion on whether more of the lifecycle should be handled here?
My initial inclination would be to do that separately/on top of the async
client.
Bikas,
To reduce the number of classes a user needs to deal with, we could have
Heartbeater wrap the AMRMClient methods and rename it something like
AsyncAMRMClient. To also eliminate the AllocationListener interface, we could
have the user subclass Heartbeater? For the latter, though, in this case to me
registering callbacks seems like an easier mental model than subclassing, as it
better separates the client from the user code in the user's mind. But I don't
feel super strongly about this.
The futures idea is interesting - if submitContainerRequest returned a future
it would be cool that someone could scheduler containers just using:
{code}
ListContainer myAllocatedContainers =
heartbeater.submitContainerRequests().getContainers();
{code}
However, I don't think this captures enough of a majority of use cases, as many
YARN apps will want to start containers asynchronously as they trickle back in.
Regarding the callbacks, I didn't notice that the info returned in AMResponse
is stored in AMRMClient. As the user can get access to it that way, I agree,
Bikas, it's probably clearer to take out the allocateResponse method.
Regarding the completedContainers and allocatedContainers as two methods or
one, I don't feel super strongly about either approach, but if all else is
equal, to me it seems easier for the developer to think about them separately,
especially with reboot and if we want to do the
onContainerLost/onContainerFail/onContainerSuccess that Chris mentioned. In
java awt/swing, mouseReleased and mouseClicked events come at the same time,
rather than mouseReleased containing an isClick param, and this always seemed
intuitive to me. I can see that it might be useful to know what containers
were finished before knowing what containers were allocated, but I can't come
up with a reason for the other way around.
Add a poller that allows the AM to receive notifications when it is assigned
containers
---
Key: YARN-417
URL: https://issues.apache.org/jira/browse/YARN-417
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, applications
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-417.patch, YarnAppMaster.java,
YarnAppMasterListener.java
Writing AMs would be easier for some if they did not have to handle
heartbeating to the RM on their own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-310) RM changes to handle NM heartbeat during overload
[ https://issues.apache.org/jira/browse/YARN-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-310. Resolution: Duplicate RM changes to handle NM heartbeat during overload - Key: YARN-310 URL: https://issues.apache.org/jira/browse/YARN-310 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages
[
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586742#comment-13586742
]
Hadoop QA commented on YARN-379:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570829/YARN-379.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/433//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/433//console
This message is automatically generated.
yarn [node,application] command print logger info messages
--
Key: YARN-379
URL: https://issues.apache.org/jira/browse/YARN-379
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Abhishek Kapoor
Labels: usability
Fix For: trunk-win
Attachments: YARN-379.patch
Running the yarn node and yarn applications command results in annoying log
info messages being printed:
$ yarn node -list
13/02/06 02:36:50 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/02/06 02:36:50 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Nodes:1
Node-IdNode-State Node-Http-Address
Health-Status(isNodeHealthy)Running-Containers
foo:8041RUNNING foo:8042 true
0
13/02/06 02:36:50 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
$ yarn application
13/02/06 02:38:47 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/02/06 02:38:47 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Invalid Command Usage :
usage: application
-kill arg Kills the application.
-list Lists all the Applications from RM.
-status arg Prints the status of the application.
13/02/06 02:38:47 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-380) yarn node -status prints Last-Last-Node-Status
[ https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kapoor reassigned YARN-380: Assignee: Abhishek Kapoor yarn node -status prints Last-Last-Node-Status -- Key: YARN-380 URL: https://issues.apache.org/jira/browse/YARN-380 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability I assume the Last-Last-NodeStatus is a typo and it should just be Last-Node-Status. $ yarn node -status foo.com:8041 Node Report : Node-Id : foo.com:8041 Rack : /10.10.10.0 Node-State : RUNNING Node-Http-Address : foo.com:8042 Health-Status(isNodeHealthy) : true Last-Last-Health-Update : 1360118400219 Health-Report : Containers : 0 Memory-Used : 0M Memory-Capacity : 24576 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-380) yarn node -status prints Last-Last-Node-Status
[ https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586843#comment-13586843 ] Abhishek Kapoor commented on YARN-380: -- I guess Thomas you mean instead of Last-Last-Health-Update it should be Last-Node-Status with a human readable date format ? Please clarify. I have looked into the issue and will be patching if after the update from you. Thanks Regards Abhishek yarn node -status prints Last-Last-Node-Status -- Key: YARN-380 URL: https://issues.apache.org/jira/browse/YARN-380 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability I assume the Last-Last-NodeStatus is a typo and it should just be Last-Node-Status. $ yarn node -status foo.com:8041 Node Report : Node-Id : foo.com:8041 Rack : /10.10.10.0 Node-State : RUNNING Node-Http-Address : foo.com:8042 Health-Status(isNodeHealthy) : true Last-Last-Health-Update : 1360118400219 Health-Report : Containers : 0 Memory-Used : 0M Memory-Capacity : 24576 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-380) yarn node -status prints Last-Last-Node-Status
[ https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586845#comment-13586845 ] Abhishek Kapoor commented on YARN-380: -- Also I am proposing to use DateFormat.LONG which will be for example (26 February, 2013 11:51:35 AM IST) yarn node -status prints Last-Last-Node-Status -- Key: YARN-380 URL: https://issues.apache.org/jira/browse/YARN-380 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Thomas Graves Assignee: Abhishek Kapoor Labels: usability I assume the Last-Last-NodeStatus is a typo and it should just be Last-Node-Status. $ yarn node -status foo.com:8041 Node Report : Node-Id : foo.com:8041 Rack : /10.10.10.0 Node-State : RUNNING Node-Http-Address : foo.com:8042 Health-Status(isNodeHealthy) : true Last-Last-Health-Update : 1360118400219 Health-Report : Containers : 0 Memory-Used : 0M Memory-Capacity : 24576 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
