[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649862#comment-13649862 ] Luke Lu commented on YARN-18: - The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory? This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650054#comment-13650054 ] Luke Lu commented on YARN-18: - Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf" > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler
[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696160#comment-13696160 ] Luke Lu commented on YARN-877: -- The change is straightforward and the patch looks good. +1. Will commit shortly. > Allow for black-listing resources in FifoScheduler > -- > > Key: YARN-877 > URL: https://issues.apache.org/jira/browse/YARN-877 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-877-2.patch, YARN-877.patch > > > YARN-750 already addressed black-list staff in YARN API and CS scheduler, > this jira add implementation for FifoScheduler. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler
[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696191#comment-13696191 ] Luke Lu commented on YARN-877: -- Committed to trunk, branch-2 and 2.1-beta. Thanks Junping! > Allow for black-listing resources in FifoScheduler > -- > > Key: YARN-877 > URL: https://issues.apache.org/jira/browse/YARN-877 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-877-2.patch, YARN-877.patch > > > YARN-750 already addressed black-list staff in YARN API and CS scheduler, > this jira add implementation for FifoScheduler. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-914) Support graceful decommission of nodemanager
Luke Lu created YARN-914: Summary: Support graceful decommission of nodemanager Key: YARN-914 URL: https://issues.apache.org/jira/browse/YARN-914 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Luke Lu Assignee: Junping Du When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NN is decommissioned, all running containers on the NN need to be rescheduled on other NNs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager
[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled on other NMs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. was: When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NN is decommissioned, all running containers on the NN need to be rescheduled on other NNs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. > Support graceful decommission of nodemanager > > > Key: YARN-914 > URL: https://issues.apache.org/jira/browse/YARN-914 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Luke Lu >Assignee: Junping Du > > When NNs are decommissioned for non-fault reasons (capacity change etc.), > it's desirable to minimize the impact to running applications. > Currently if a NM is decommissioned, all running containers on the NM need to > be rescheduled on other NMs. Further more, for finished map tasks, if their > map output are not fetched by the reducers of the job, these map tasks will > need to be rerun as well. > We propose to introduce a mechanism to optionally gracefully decommission a > node manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager
[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled on other NMs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. was: When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled on other NMs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. > Support graceful decommission of nodemanager > > > Key: YARN-914 > URL: https://issues.apache.org/jira/browse/YARN-914 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Luke Lu >Assignee: Junping Du > > When NMs are decommissioned for non-fault reasons (capacity change etc.), > it's desirable to minimize the impact to running applications. > Currently if a NM is decommissioned, all running containers on the NM need to > be rescheduled on other NMs. Further more, for finished map tasks, if their > map output are not fetched by the reducers of the job, these map tasks will > need to be rerun as well. > We propose to introduce a mechanism to optionally gracefully decommission a > node manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-914) Support graceful decommission of nodemanager
[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709144#comment-13709144 ] Luke Lu commented on YARN-914: -- [~atm]: Nice catch! Of course :) > Support graceful decommission of nodemanager > > > Key: YARN-914 > URL: https://issues.apache.org/jira/browse/YARN-914 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Luke Lu >Assignee: Junping Du > > When NMs are decommissioned for non-fault reasons (capacity change etc.), > it's desirable to minimize the impact to running applications. > Currently if a NM is decommissioned, all running containers on the NM need to > be rescheduled on other NMs. Further more, for finished map tasks, if their > map output are not fetched by the reducers of the job, these map tasks will > need to be rerun as well. > We propose to introduce a mechanism to optionally gracefully decommission a > node manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-347) YARN node CLI should also show CPU info as memory info in node status
[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721439#comment-13721439 ] Luke Lu commented on YARN-347: -- The patch lgtm. +1. Will commmit shortly. > YARN node CLI should also show CPU info as memory info in node status > - > > Key: YARN-347 > URL: https://issues.apache.org/jira/browse/YARN-347 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-347.patch, YARN-347-v2.patch, YARN-347-v3.patch > > > With YARN-2 checked in, CPU info are taken into consideration in resource > scheduling. yarn node -status should show CPU used and capacity info > as memory info. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-347) YARN node CLI should also show CPU info besides memory info in node status
[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN node CLI should also show CPU info besides memory info in node status (was: YARN node CLI should also show CPU info as memory info in node status) > YARN node CLI should also show CPU info besides memory info in node status > -- > > Key: YARN-347 > URL: https://issues.apache.org/jira/browse/YARN-347 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-347.patch, YARN-347-v2.patch, YARN-347-v3.patch, > YARN-347-v4.patch > > > With YARN-2 checked in, CPU info are taken into consideration in resource > scheduling. yarn node -status should show CPU used and capacity info > as memory info. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-347) YARN CLI should show CPU info besides memory info in node status
[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN CLI should show CPU info besides memory info in node status (was: YARN node CLI should also show CPU info besides memory info in node status) > YARN CLI should show CPU info besides memory info in node status > > > Key: YARN-347 > URL: https://issues.apache.org/jira/browse/YARN-347 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-347.patch, YARN-347-v2.patch, YARN-347-v3.patch, > YARN-347-v4.patch > > > With YARN-2 checked in, CPU info are taken into consideration in resource > scheduling. yarn node -status should show CPU used and capacity info > as memory info. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735321#comment-13735321 ] Luke Lu commented on YARN-1024: --- As many have already pointed out, using vcore to express "unit of computation" causes more confusion than the "predictability" it tries to address. (whatever)core as a unit of thread-affinity/locality is an important concept to preserve. OTOH, I can see the need to define a YCU (YARN Computation Unit) for the properties that Arun tried to address. YCU can be implemented as deployment specific mapping to fractional cores, which preserves both thread-locality and cpu/core-slicing. > Define a virtual core unambigiously > --- > > Key: YARN-1024 > URL: https://issues.apache.org/jira/browse/YARN-1024 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > We need to clearly define the meaning of a virtual core unambiguously so that > it's easy to migrate applications between clusters. > For e.g. here is Amazon EC2 definition of ECU: > http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it > Essentially we need to clearly define a YARN Virtual Core (YVC). > Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the > equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741713#comment-13741713 ] Luke Lu commented on YARN-291: -- It'll better if the node resource config is grouped into a repeated field, so that you can config multiple nodes in one RPC, which is the common case for cluster management systems. > Dynamic resource configuration > -- > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-CoreAndAdmin.patch, > YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > More background and design details can be found in attached proposal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long
[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475811#comment-13475811 ] Luke Lu commented on YARN-151: -- The most straight forward fix would be upgrading to a more recent version of datatables and turn on defered rendering (not creating dom nodes for all rows, even they are hidden): http://datatables.net/release-datatables/examples/ajax/defer_render.html Sorting, searching through in js itself can easily handle up to 100k rows on modest hardware. > Browser thinks RM main page JS is taking too long > - > > Key: YARN-151 > URL: https://issues.apache.org/jira/browse/YARN-151 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans > > The main RM page with the default settings of 10,000 applications can cause > browsers to think that the JS on the page is stuck and ask you if you want to > kill it. This is a big usability problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long
[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487337#comment-13487337 ] Luke Lu commented on YARN-151: -- 10k apps is a small amount of data (a few hundred KB to a few MB) and that you don't need implement the logic (api that supports searching/sorting/paging) at server side and that you have better user response time (instant search experience) as the data is at the client side. The server side stuff would be better if we need to support more than 100K apps. If you come up with a patch, I'm happy to review it :) > Browser thinks RM main page JS is taking too long > - > > Key: YARN-151 > URL: https://issues.apache.org/jira/browse/YARN-151 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Ravi Prakash > > The main RM page with the default settings of 10,000 applications can cause > browsers to think that the JS on the page is stuck and ask you if you want to > kill it. This is a big usability problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-265) Allow locality preference for launching AM
Luke Lu created YARN-265: Summary: Allow locality preference for launching AM Key: YARN-265 URL: https://issues.apache.org/jira/browse/YARN-265 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Luke Lu There are many usage scenarios would like to convey the preference of the original placement of ApplicationMaster. e.g., small jobs that are executed in AM; Topology preference from external cluster management system etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-265) Allow locality preference for launching AM
[ https://issues.apache.org/jira/browse/YARN-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-265: - Description: There are many usage scenarios that would like to convey the preference of the original placement of ApplicationMaster. e.g., small jobs that are executed in AM; Topology preference from external cluster management system etc. (was: There are many usage scenarios would like to convey the preference of the original placement of ApplicationMaster. e.g., small jobs that are executed in AM; Topology preference from external cluster management system etc.) > Allow locality preference for launching AM > -- > > Key: YARN-265 > URL: https://issues.apache.org/jira/browse/YARN-265 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Luke Lu > > There are many usage scenarios that would like to convey the preference of > the original placement of ApplicationMaster. e.g., small jobs that are > executed in AM; Topology preference from external cluster management system > etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-265) Allow locality preference for launching AM
[ https://issues.apache.org/jira/browse/YARN-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526946#comment-13526946 ] Luke Lu commented on YARN-265: -- Looks like this is a superset of YARN-238, as I'm trying to allow resource request to be fully specified at client side. > Allow locality preference for launching AM > -- > > Key: YARN-265 > URL: https://issues.apache.org/jira/browse/YARN-265 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Luke Lu > > There are many usage scenarios that would like to convey the preference of > the original placement of ApplicationMaster. e.g., small jobs that are > executed in AM; Topology preference from external cluster management system > etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536228#comment-13536228 ] Luke Lu commented on YARN-223: -- # The changes to eliminate usage of ResourceCalculatorPlugin is incorrect. Process tree is an impl detail of ResourceCalculatorPlugin. Many systems don't have process tree or equivalent at all. # You only need to change the interface of ResourceCalculatorProcessTree and impl of ProcfsBasedProcessTree. > Change processTree interface to work better with native code > > > Key: YARN-223 > URL: https://issues.apache.org/jira/browse/YARN-223 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Critical > Attachments: pstree-update4.txt, pstree-update6.txt, > pstree-update6.txt > > > Problem is that on every update of processTree new object is required. This > is undesired when working with processTree implementation in native code. > replace ProcessTree.getProcessTree() with updateProcessTree(). No new object > allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536392#comment-13536392 ] Luke Lu commented on YARN-223: -- Upon second look, though the current ResourceCalculatorProcessTree interface is not ideal (checkPidPgrpidMatches threw me off, which should simply be checkOwnership), it's general enough I think. +1 for the patch. Will commit shortly. > Change processTree interface to work better with native code > > > Key: YARN-223 > URL: https://issues.apache.org/jira/browse/YARN-223 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Critical > Attachments: pstree-update4.txt, pstree-update6.txt, > pstree-update6.txt > > > Problem is that on every update of processTree new object is required. This > is undesired when working with processTree implementation in native code. > replace ProcessTree.getProcessTree() with updateProcessTree(). No new object > allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536636#comment-13536636 ] Luke Lu commented on YARN-223: -- Committed to trunk and branch-2. Thanks Radim for the patch and Chris and Bikas for the review. > Change processTree interface to work better with native code > > > Key: YARN-223 > URL: https://issues.apache.org/jira/browse/YARN-223 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Critical > Attachments: pstree-update4.txt, pstree-update6.txt, > pstree-update6.txt > > > Problem is that on every update of processTree new object is required. This > is undesired when working with processTree implementation in native code. > replace ProcessTree.getProcessTree() with updateProcessTree(). No new object > allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537165#comment-13537165 ] Luke Lu commented on YARN-223: -- Sorry folks! I got too used to seeing mapreduce code in YARN :) > Change processTree interface to work better with native code > > > Key: YARN-223 > URL: https://issues.apache.org/jira/browse/YARN-223 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Critical > Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 > > Attachments: pstree-update4.txt, pstree-update6.txt, > pstree-update6.txt > > > Problem is that on every update of processTree new object is required. This > is undesired when working with processTree implementation in native code. > replace ProcessTree.getProcessTree() with updateProcessTree(). No new object > allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537166#comment-13537166 ] Luke Lu commented on YARN-223: -- I think the PreCommit test patch system should either handle cross project patches or outright reject such patches. > Change processTree interface to work better with native code > > > Key: YARN-223 > URL: https://issues.apache.org/jira/browse/YARN-223 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Critical > Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 > > Attachments: pstree-update4.txt, pstree-update6.txt, > pstree-update6.txt > > > Problem is that on every update of processTree new object is required. This > is undesired when working with processTree implementation in native code. > replace ProcessTree.getProcessTree() with updateProcessTree(). No new object > allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration on NM
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542297#comment-13542297 ] Luke Lu commented on YARN-291: -- A simpler and more straightforward alternative would be changing the node resource on RM directly via JMX. No protocol changes would be necessary. Changing resource on NM and propagating the change to RM would lead to nasty race conditions where RM still thinks that an NM has enough resource and schedule new containers on the already downsized NM, which should fail and RM would need to reschedule the containers elsewhere, which is not pretty, especially when large number of NMs are affected, even if all the corner cases are handled. bq. we should do RPC and optionally the web-services to the NM directly. And we should add command line util too. Since the only near term user of the feature is *external* admin processes that prefer not having Hadoop jar dependencies, JMX change alone (a much smaller patch that doesn't impact major production code paths) would suffice. YARN RPC changes can be in a separate JIRA, when YARN itself needs to change per node resources. I'm not sure how useful a YARN CLI interface for this is, given it's impractical for people to manually change per node resources (when number of nodes is greater than 10). OTOH, I'm fine with having everything in, as long as it doesn't delay the inclusion of the basic functionality. > Dynamic resource configuration on NM > > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration on NM
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542825#comment-13542825 ] Luke Lu commented on YARN-291: -- Regarding the race condition, I meant that you could change the resource on NM in the middle of a heartbeat, where RM already assigned some containers to NM from the last heartbeat but you don't actually have the resource to launch these containers. It's relatively harmless now as NM is not enforcing its resource limit when launching containers, but the race condition can rear its ugly head if someone later decides to add code to enforce the limit when adding more resource features for things like GPU etc. The race condition would not be obvious to later maintainers. With the RM only approach, I think that it's OK/harmless not to change totalResource at all at NM side and leave it as the original resource limit, as long as we set the resource limit <= the original limit, even if limit enforcement is added later. This makes the change an order of magnitude smaller and less invasive, as it doesn't have to change the node update code paths. It would also be wasteful to heartbeat the node resource (especially when later enhanced to including features besides memory) that doesn't change most of the time. > Dynamic resource configuration on NM > > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599383#comment-13599383 ] Luke Lu commented on YARN-18: - Preliminary comments: # We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code. # We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects. # The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory. # Pluggable interface/class should be public with the InterfaceAudience.Private annotation. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607021#comment-13607021 ] Luke Lu commented on YARN-18: - Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps? I think we're getting close. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614383#comment-13614383 ] Luke Lu commented on YARN-18: - The original patch were already broken up into YARN-18 and YARN-19. YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration on NM
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618522#comment-13618522 ] Luke Lu commented on YARN-291: -- Resource scheduling is fundamentally centralized at RM. The global resource view is currently bootstrapped via the node registration process, which is more of a historical artifact based on convenience, since the resource view can also be constructed directly on RM via an inventory database. It's a round about, inconvenient and inefficient way to (re)construct the resource view by modifying per node config explicitly and propagate partial views to RM, if you already have an inventory database. For a practical example: if you have OLTP workload (say HBase) sharing the same hardware with YARN and there is a load surge on HBase, we need to stop scheduling tasks/containers immediately on relevant (potentially all) nodes. The current patch (JMX is just used as a portable protocol for external management client to communicate with RM) can take effect immediately most efficiently. If we explicitly modify each nodemanager config and let the NM-RM protocol to propagate the change to RM, it would waste resource (CPU and network bandwidth) to contact (potentially) all the nodemanagers and cause unnecessary scheduling delays if the propagation is via regular heartbeat and/or DDoS the RM if (potentially) all the NMs need to re-register out-of-band. This is not about JMX based tricks. This is about changing global resource view directly where the scheduler is vs the Rube Goldbergish way of changing NM config individually and propagate changes to RM to reconstruct the resource view. IMO, the direct way is better because NM doesn't really care about what resource it really has. > Dynamic resource configuration on NM > > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618525#comment-13618525 ] Luke Lu commented on YARN-18: - bq. I see a lot of factories and then no changes to actually support node-group.. YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19. bq. we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology. This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration on NM
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618697#comment-13618697 ] Luke Lu commented on YARN-291: -- bq. The NM is the node controller and responsible for managing the node resources. There is actually no/zero code in NM itself (other than picking up resource config and report to RM) so far that cares about how much resource the node has. NM is responsible for managing the containers that scheduler assigns to it. It does not need to know how much resource it really has. Only scheduler needs to care about how much resource a node has. Making NM oblivious to the total resource it has obviate the unnecessary split brain situation. bq. if it had more work to schedule and there were free resources then it would have already done so The point was about scheduling latency. There could be a window of OLTP surge where the nodes have free resource and large job is submitted. OLTP workload is real time. Also contacting NM to reconfig resource node by node is inefficient and wasteful. bq. the currently running tasks will hamper the HBase case even if future work is not scheduled. This is orthogonal to scheduling, which this JIRA is about. Management agent can suspend/move/kill the containers if necessary. bq. What we need is some way to allow HBase higher priority for system resources so that tasks do not impede HBase perf. Something like YARN-443. OS scheduling priority works with CPU and to a degree IO (depending on the IO scheduler) but not memory, which is critical for many OLTP workload. bq. This JIRA is part of HADOOP-9165 but that one has been resolved as invalid. I don't know what you are trying to say. HADOOP-9165 is resolved due to wrong JIRA component. It could be moved to YARN or MAPREDUCE, but the creator chose to file new ones instead. > Dynamic resource configuration on NM > > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-291) Dynamic resource configuration
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-291: - Summary: Dynamic resource configuration (was: Dynamic resource configuration on NM) bq. I have concerns on changing the available NMs capacity directly in the NMs. You probably haven't looked at the patch (YARN-311), which actually changes the global resource view of RM directly. NMs themselves are untouched. Changed title to reduce further confusion. bq. This would complicate significantly allocation changes/re-computation of resources for running applications. How so? In fact there is no need to change anything in NM, AFAICT. The existing containers can either be left alone until completion or suspended/moved/killed by a management agent. bq. An (unmanaged) AM requests unmanaged resources from the RM (a new flag in a resource request)... First this requires modifying existing applications to support the new request API, which has significant issues (best-effort, timeout) without preemption. Second, RM scheduling is fundamentally high latency and best-effort by design at container granularity. It's not suitable for low latency workload with high memory utilization via fine grained (page-level) memory sharing. Using the HBase example again, you'll need to request an "unmanaged resource" with a container size of at least the peak memory of the region server JVM, which significantly under utilize memory resource, as peak memory size is significantly higher than average memory usage. Note, the existing simple patch doesn't change existing API/protocols. It works with any existing unmodified OLTP applications and can achieve maximum resource utilization with mainstream hypervisors. > Dynamic resource configuration > -- > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-291) Dynamic resource configuration
[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619374#comment-13619374 ] Luke Lu commented on YARN-291: -- bq. Changes in NM capacity triggered from outside of the regular scheduling would unbalance existing distribution of allocations potentially triggering preemption. You'd need to handle this specially in the RM/scheduler to handle such scenarios. The existing mechanism would/should work by simply killing off containers when necessary. The container fault tolerant mechanism would/should take care of the rest (including preemption). We can do a better job to differentiate the faults induced by preemption, which would be straight-forward if we expose a preemption API, when we get around to implement the preemption feature. If container suspend/resume API is implemented, we can do that as well. bq. It depends how you design you AM that handles unmanaged containers. You could request several small resources on peak and then release them as you don't need them. This requires many missing features in RM in order to work properly: finer grain OS/application resource metrics, application priority, conflict arbitration, preemption and related security features (mostly related authorization stuff). This approach is also problematic to support coexistence of different instances/versions of YARN on the same physical cluster. bq. It is adding a new one, that is a change. The change doesn't affect existing/future YARN applications. The management protocol allows existing/future cluster schedulers to expose appropriate resource views to (multiple instances/versions of) YARN in a straight forward manner. IMO, the solution is orthogonal and to what you have proposed. It allows any existing non-YARN applications to efficiently coexist with YARN applications without having to write a special AM using "unmanaged resource" API, with no new features "required" in YARN now. In other words, it is a simple solution to allow YARN to coexist with other schedulers (including other instances/versions of YARN) that already have the features people use/want. I'd be interested in hearing cases, where our approach "breaks" YARN applications in any way. > Dynamic resource configuration > -- > > Key: YARN-291 > URL: https://issues.apache.org/jira/browse/YARN-291 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: Elastic Resources for YARN-v0.2.pdf, > YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, > YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, > YARN-291-JMXInterfaceOnNM-02.patch, > YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, > YARN-291-YARNClientCommandline-04.patch > > > The current Hadoop YARN resource management logic assumes per node resource > is static during the lifetime of the NM process. Allowing run-time > configuration on per node resource will give us finer granularity of resource > elasticity. This allows Hadoop workloads to coexist with other workloads on > the same hardware efficiently, whether or not the environment is virtualized. > About more background and design details, please refer: HADOOP-9165. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1042) add ability to specify affinity/anti-affinity in container requests
[ https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13763273#comment-13763273 ] Luke Lu commented on YARN-1042: --- bq. BTW, it seems the effort is more on application side. Do we think it is better to move to MAPREDUCE project? Although you can do a lot at the app side with container filtering, protocol and scheduler support will make it more efficient. I guess the intention of the jira is more for the latter, that affinity support should be app independent. > add ability to specify affinity/anti-affinity in container requests > --- > > Key: YARN-1042 > URL: https://issues.apache.org/jira/browse/YARN-1042 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Junping Du > Attachments: YARN-1042-demo.patch > > > container requests to the AM should be able to request anti-affinity to > ensure that things like Region Servers don't come up on the same failure > zones. > Similarly, you may be able to want to specify affinity to same host or rack > without specifying which specific host/rack. Example: bringing up a small > giraph cluster in a large YARN cluster would benefit from having the > processes in the same rack purely for bandwidth reasons. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765760#comment-13765760 ] Luke Lu commented on YARN-311: -- I think the v6.2 patch looks reasonable. If there is no further objections, I plan to commit it over the weekend. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-938) Hadoop 2 benchmarking
[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783446#comment-13783446 ] Luke Lu commented on YARN-938: -- Is jvm reuse (set to -1) turned on for Hadoop 1 runs? Unfortunately, container reuse is not in MRv2 yet (MAPREDUCE-3902 appear to be stalled). It'd be interesting to see numbers from Tez, which does have container reuse, as well for comparison. > Hadoop 2 benchmarking > -- > > Key: YARN-938 > URL: https://issues.apache.org/jira/browse/YARN-938 > Project: Hadoop YARN > Issue Type: Task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: Hadoop-benchmarking-2.x-vs-1.x.xls > > > I am running the benchmarks on Hadoop 2 and will update the results soon. > Thanks, > Mayank -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-7) Add support for DistributedShell to ask for CPUs along with memory
[ https://issues.apache.org/jira/browse/YARN-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790853#comment-13790853 ] Luke Lu commented on YARN-7: The latest patch lgtm. +1. > Add support for DistributedShell to ask for CPUs along with memory > -- > > Key: YARN-7 > URL: https://issues.apache.org/jira/browse/YARN-7 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.1-beta >Reporter: Arun C Murthy >Assignee: Junping Du > Labels: patch > Attachments: YARN-7.patch, YARN-7-v2.patch, YARN-7-v3.patch, > YARN-7-v4.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804038#comment-13804038 ] Luke Lu commented on YARN-311: -- v9 patch lgtm. +1. For the record, we've discussed the changes with Vinod et al offline. The consensus is that change is reasonable, low risk and fulfill a useful use case (resource arbitration between different resource pools managed by YARN or other (versions of) systems including but not limited to Hadoop 1.x on the same physical hardware). > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804320#comment-13804320 ] Luke Lu commented on YARN-311: -- [~djp]: Upon second thought, I think we should improve the setCapacity API to include scheduler hints/options (to deal with running containers when node capacity is set lower than the used capacity on the node) in this JIRA, in case it gets lost through the crack later. I recall the options as FINISH_RUNNING, TIMEOUT_RUNNING, TERMINATE_RUNNING. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804335#comment-13804335
]
Luke Lu commented on YARN-311:
--
We can probably infer the option by an overcommit timeout argument < 0: keep
running; == 0: terminate; > 0 timeout:
{code}
void setTotalCapacity(Resource resource, int overCommitTimeoutMillis);
{code}
> Dynamic node resource configuration: core scheduler changes
> ---
>
> Key: YARN-311
> URL: https://issues.apache.org/jira/browse/YARN-311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager, scheduler
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
> YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
> YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
> YARN-311-v9.patch
>
>
> As the first step, we go for resource change on RM side and expose admin APIs
> (admin protocol, CLI, REST and JMX API) later. In this jira, we will only
> contain changes in scheduler.
> The flow to update node's resource and awareness in resource scheduling is:
> 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
> 2. When next NM heartbeat for updating status comes, the RMNode's resource
> change will be aware and the delta resource is added to schedulerNode's
> availableResource before actual scheduling happens.
> 3. Scheduler do resource allocation according to new availableResource in
> SchedulerNode.
> For more design details, please refer proposal and discussions in parent
> JIRA: YARN-291.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804532#comment-13804532 ] Luke Lu commented on YARN-311: -- bq. Other policies/options will be addressed in YARN-999 Agreed. That was the consensus (default policy is to keep containers running until they finish) as well. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807484#comment-13807484 ] Luke Lu commented on YARN-311: -- I share the same concern with Bikas with the RMNode lock inside the static util method. It doesn't appear to be necessary, as long as both RMNodeImpl#totalCapacity and overCommitTimeout is volatile (the latter is not in the patch), which are simply accessed via get/set (no dependent op like increment). The lock could lead to deadlocks latter (after new code/refactor) even if it appears to be fine now. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v10.patch, YARN-311-v1.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807773#comment-13807773 ] Luke Lu commented on YARN-311: -- Junping: your [previous comment|https://issues.apache.org/jira/browse/YARN-311?focusedCommentId=13804690&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13804690] actually convinced me that being able independently set overCommitTimeout is a reasonable use case and that the condition (resource1, overCommitTimeout2) is a reasonable behavior (the later timeout takes effect), given the benefits. I was concerned that # Requiring RMNode lock at usage site is brittle and error-prone. # Extra RMNode lock per heartbeat could be expensive. We can even preserve the consistency without locks by using a ResourceConfig object which holds the resource and timeout together. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v10.patch, YARN-311-v1.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808130#comment-13808130 ] Luke Lu commented on YARN-311: -- IMO, using an immutable ResourceOption object is cleaner than mess with read/write locks, which has high constant overhead even when the lock is not contended. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v10.patch, YARN-311-v1.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808557#comment-13808557
]
Luke Lu commented on YARN-311:
--
With ResourceOption, you don't need a separate get/setOverCommitTimeout method.
You only need one pair of methods for all future resource options (prioritize
certain containers (last to kill) in over commit situations etc.)
{code}
private volatile ResourceOption totalCapacity;
...
void setTotalCapacity(ResourceOption ro);
void ResourceOption getTotalCapacity();
{code}
{{setTotalCapacity}} is now idempotent. Seems simpler to me.
> Dynamic node resource configuration: core scheduler changes
> ---
>
> Key: YARN-311
> URL: https://issues.apache.org/jira/browse/YARN-311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager, scheduler
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-311-v10.patch, YARN-311-v1.patch,
> YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch,
> YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch,
> YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch
>
>
> As the first step, we go for resource change on RM side and expose admin APIs
> (admin protocol, CLI, REST and JMX API) later. In this jira, we will only
> contain changes in scheduler.
> The flow to update node's resource and awareness in resource scheduling is:
> 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
> 2. When next NM heartbeat for updating status comes, the RMNode's resource
> change will be aware and the delta resource is added to schedulerNode's
> availableResource before actual scheduling happens.
> 3. Scheduler do resource allocation according to new availableResource in
> SchedulerNode.
> For more design details, please refer proposal and discussions in parent
> JIRA: YARN-291.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808600#comment-13808600 ] Luke Lu commented on YARN-311: -- bq. How it sounds to you? Sounds good to me :), assuming you'll add getResourceOption as well. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v10.patch, YARN-311-v1.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811714#comment-13811714 ] Luke Lu commented on YARN-311: -- v12 patch lgtm. +1. Will commit soon. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v10.patch, > YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813139#comment-13813139 ] Luke Lu commented on YARN-311: -- [~djp]: Unfortunately YARN-1343 got in before I tried to merge the patch. Now the patch won't compile due to the old RMNodeImpl ctor usage in TestRMNodeTransition. Can you rebase the patch? > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v10.patch, > YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, > YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, YARN-311-v4.patch, > YARN-311-v5.patch, YARN-311-v6.1.patch, YARN-311-v6.2.patch, > YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-311: - Fix Version/s: 2.3.0 Hadoop Flags: Reviewed Committed to trunk and branch-2. Thanks Junping for the patch! > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.3.0 > > Attachments: YARN-311-v1.patch, YARN-311-v10.patch, > YARN-311-v11.patch, YARN-311-v12.patch, YARN-311-v12b.patch, > YARN-311-v13.patch, YARN-311-v2.patch, YARN-311-v3.patch, YARN-311-v4.patch, > YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-938) Hadoop 2 benchmarking
[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823155#comment-13823155 ] Luke Lu commented on YARN-938: -- Yes, it'd be great if [~mayank_bansal] can share the configs and command lines to run the benchmarks, so others can reproduce the results. > Hadoop 2 benchmarking > -- > > Key: YARN-938 > URL: https://issues.apache.org/jira/browse/YARN-938 > Project: Hadoop YARN > Issue Type: Task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: Hadoop-benchmarking-2.x-vs-1.x-1.xls, > Hadoop-benchmarking-2.x-vs-1.x.xls > > > I am running the benchmarks on Hadoop 2 and will update the results soon. > Thanks, > Mayank -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-312) Add updateNodeResource in ResourceManagerAdministrationProtocol
[ https://issues.apache.org/jira/browse/YARN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823513#comment-13823513 ] Luke Lu commented on YARN-312: -- v5.1 looks reasonable (the verbosity of pb impl stuff still boggles my mind). +1. > Add updateNodeResource in ResourceManagerAdministrationProtocol > --- > > Key: YARN-312 > URL: https://issues.apache.org/jira/browse/YARN-312 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.2.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-312-v1.patch, YARN-312-v2.patch, YARN-312-v3.patch, > YARN-312-v4.1.patch, YARN-312-v4.patch, YARN-312-v5.1.patch, YARN-312-v5.patch > > > Add fundamental RPC (ResourceManagerAdministrationProtocol) to support node's > resource change. For design detail, please refer parent JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-938) Hadoop 2 benchmarking
[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13851507#comment-13851507 ] Luke Lu commented on YARN-938: -- Thanks for the results Jeff!. It's interesting to note that the best terasort throughput in your configuration is ~140MB/s (mrv1, 96MB/s for mrv2) per physical host for a 8TB data set, compared with ~23MB/s (1.x, 21MB/s for 2.2) per physical host in Mayank's results for a 1TB (?) data set. Obviously 10Gb networking and 12 15K RPM SAS disks per host helped. OTOH, I'd expect Mayank's results to be a lot faster as the data set fits into the 260 slave host cluster memory (buffer cache). It'll be interesting to show the Apache 1.2.1 results for Jeff's configuration as well, so it's more comparable to Mayank's results, as I suspect that CDH mrv1 have more optimizations than Apache. > Hadoop 2 benchmarking > -- > > Key: YARN-938 > URL: https://issues.apache.org/jira/browse/YARN-938 > Project: Hadoop YARN > Issue Type: Task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: Hadoop-benchmarking-2.x-vs-1.x-1.xls, > Hadoop-benchmarking-2.x-vs-1.x.xls, cdh500beta1_cpu_util.jpg, > cdh500beta1_mr1_mr2.xlsx > > > I am running the benchmarks on Hadoop 2 and will update the results soon. > Thanks, > Mayank -- This message was sent by Atlassian JIRA (v6.1.4#6159)
