[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649862#comment-13649862 ] Luke Lu commented on YARN-18: - The latest patch is getting close. I think the naming of Abstract

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650054#comment-13650054 ] Luke Lu commented on YARN-18: - Please find the summary/writeup in the attached "Pluggable topolog

[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler

[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696160#comment-13696160 ] Luke Lu commented on YARN-877: -- The change is straightforward and the patch looks good. +1. Wil

[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler

[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696191#comment-13696191 ] Luke Lu commented on YARN-877: -- Committed to trunk, branch-2 and 2.1-beta. Thanks Junping!

[jira] [Created] (YARN-914) Support graceful decommission of nodemanager

Luke Lu created YARN-914: Summary: Support graceful decommission of nodemanager Key: YARN-914 URL: https://issues.apache.org/jira/browse/YARN-914 Project: Hadoop YARN Issue Type: Improvement Affe

[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize

[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize

[jira] [Commented] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709144#comment-13709144 ] Luke Lu commented on YARN-914: -- [~atm]: Nice catch! Of course :) > Support gra

[jira] [Commented] (YARN-347) YARN node CLI should also show CPU info as memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721439#comment-13721439 ] Luke Lu commented on YARN-347: -- The patch lgtm. +1. Will commmit shortly. > YA

[jira] [Updated] (YARN-347) YARN node CLI should also show CPU info besides memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN node CLI should also show CPU info besides memory info in node status (was: YARN node CLI should als

[jira] [Updated] (YARN-347) YARN CLI should show CPU info besides memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN CLI should show CPU info besides memory info in node status (was: YARN node CLI should also show CPU

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735321#comment-13735321 ] Luke Lu commented on YARN-1024: --- As many have already pointed out, using vcore to express "un

[jira] [Commented] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741713#comment-13741713 ] Luke Lu commented on YARN-291: -- It'll better if the node resource config is grouped into a repe

[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long

[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475811#comment-13475811 ] Luke Lu commented on YARN-151: -- The most straight forward fix would be upgrading to a more rece

[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long

[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487337#comment-13487337 ] Luke Lu commented on YARN-151: -- 10k apps is a small amount of data (a few hundred KB to a few M

[jira] [Created] (YARN-265) Allow locality preference for launching AM

Luke Lu created YARN-265: Summary: Allow locality preference for launching AM Key: YARN-265 URL: https://issues.apache.org/jira/browse/YARN-265 Project: Hadoop YARN Issue Type: Improvement

[jira] [Updated] (YARN-265) Allow locality preference for launching AM

[ https://issues.apache.org/jira/browse/YARN-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-265: - Description: There are many usage scenarios that would like to convey the preference of the original placement of A

[jira] [Commented] (YARN-265) Allow locality preference for launching AM

[ https://issues.apache.org/jira/browse/YARN-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526946#comment-13526946 ] Luke Lu commented on YARN-265: -- Looks like this is a superset of YARN-238, as I'm trying to all

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536228#comment-13536228 ] Luke Lu commented on YARN-223: -- # The changes to eliminate usage of ResourceCalculatorPlugin is

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536392#comment-13536392 ] Luke Lu commented on YARN-223: -- Upon second look, though the current ResourceCalculatorProcessT

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536636#comment-13536636 ] Luke Lu commented on YARN-223: -- Committed to trunk and branch-2. Thanks Radim for the patch and

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537165#comment-13537165 ] Luke Lu commented on YARN-223: -- Sorry folks! I got too used to seeing mapreduce code in YARN :)

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537166#comment-13537166 ] Luke Lu commented on YARN-223: -- I think the PreCommit test patch system should either handle cr

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542297#comment-13542297 ] Luke Lu commented on YARN-291: -- A simpler and more straightforward alternative would be changin

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542825#comment-13542825 ] Luke Lu commented on YARN-291: -- Regarding the race condition, I meant that you could change the

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599383#comment-13599383 ] Luke Lu commented on YARN-18: - Preliminary comments: # We should use net.topology.nodegroup.aware

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607021#comment-13607021 ] Luke Lu commented on YARN-18: - Juping, you're right that ScheduledRequests is an app side constru

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614383#comment-13614383 ] Luke Lu commented on YARN-18: - The original patch were already broken up into YARN-18 and YARN-19

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618522#comment-13618522 ] Luke Lu commented on YARN-291: -- Resource scheduling is fundamentally centralized at RM. The glo

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618525#comment-13618525 ] Luke Lu commented on YARN-18: - bq. I see a lot of factories and then no changes to actually suppo

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618697#comment-13618697 ] Luke Lu commented on YARN-291: -- bq. The NM is the node controller and responsible for managing

[jira] [Updated] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-291: - Summary: Dynamic resource configuration (was: Dynamic resource configuration on NM) bq. I have concerns on changi

[jira] [Commented] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619374#comment-13619374 ] Luke Lu commented on YARN-291: -- bq. Changes in NM capacity triggered from outside of the regula

[jira] [Commented] (YARN-1042) add ability to specify affinity/anti-affinity in container requests

[ https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13763273#comment-13763273 ] Luke Lu commented on YARN-1042: --- bq. BTW, it seems the effort is more on application side. Do

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765760#comment-13765760 ] Luke Lu commented on YARN-311: -- I think the v6.2 patch looks reasonable. If there is no further

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783446#comment-13783446 ] Luke Lu commented on YARN-938: -- Is jvm reuse (set to -1) turned on for Hadoop 1 runs? Unfortuna

[jira] [Commented] (YARN-7) Add support for DistributedShell to ask for CPUs along with memory

[ https://issues.apache.org/jira/browse/YARN-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790853#comment-13790853 ] Luke Lu commented on YARN-7: The latest patch lgtm. +1. > Add support for DistributedShell to as

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804038#comment-13804038 ] Luke Lu commented on YARN-311: -- v9 patch lgtm. +1. For the record, we've discussed the changes

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804320#comment-13804320 ] Luke Lu commented on YARN-311: -- [~djp]: Upon second thought, I think we should improve the setC

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804335#comment-13804335 ] Luke Lu commented on YARN-311: -- We can probably infer the option by an overcommit timeout argum

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804532#comment-13804532 ] Luke Lu commented on YARN-311: -- bq. Other policies/options will be addressed in YARN-999 Agree

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807484#comment-13807484 ] Luke Lu commented on YARN-311: -- I share the same concern with Bikas with the RMNode lock inside

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807773#comment-13807773 ] Luke Lu commented on YARN-311: -- Junping: your [previous comment|https://issues.apache.org/jira

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808130#comment-13808130 ] Luke Lu commented on YARN-311: -- IMO, using an immutable ResourceOption object is cleaner than m

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808557#comment-13808557 ] Luke Lu commented on YARN-311: -- With ResourceOption, you don't need a separate get/setOverCommi

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808600#comment-13808600 ] Luke Lu commented on YARN-311: -- bq. How it sounds to you? Sounds good to me :), assuming you'l

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811714#comment-13811714 ] Luke Lu commented on YARN-311: -- v12 patch lgtm. +1. Will commit soon. > Dynamic node resource

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813139#comment-13813139 ] Luke Lu commented on YARN-311: -- [~djp]: Unfortunately YARN-1343 got in before I tried to merge

[jira] [Updated] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-311: - Fix Version/s: 2.3.0 Hadoop Flags: Reviewed Committed to trunk and branch-2. Thanks Junping for the patch! >

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823155#comment-13823155 ] Luke Lu commented on YARN-938: -- Yes, it'd be great if [~mayank_bansal] can share the configs an

[jira] [Commented] (YARN-312) Add updateNodeResource in ResourceManagerAdministrationProtocol

[ https://issues.apache.org/jira/browse/YARN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823513#comment-13823513 ] Luke Lu commented on YARN-312: -- v5.1 looks reasonable (the verbosity of pb impl stuff still bog

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13851507#comment-13851507 ] Luke Lu commented on YARN-938: -- Thanks for the results Jeff!. It's interesting to note that the