[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler

[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13696160#comment-13696160 ] Luke Lu commented on YARN-877: -- The change is straightforward and the patch looks good. +1.

[jira] [Commented] (YARN-877) Allow for black-listing resources in FifoScheduler

[ https://issues.apache.org/jira/browse/YARN-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13696191#comment-13696191 ] Luke Lu commented on YARN-877: -- Committed to trunk, branch-2 and 2.1-beta. Thanks Junping!

[jira] [Created] (YARN-914) Support graceful decommission of nodemanager

Luke Lu created YARN-914: Summary: Support graceful decommission of nodemanager Key: YARN-914 URL: https://issues.apache.org/jira/browse/YARN-914 Project: Hadoop YARN Issue Type: Improvement

[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize

[jira] [Updated] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-914: - Description: When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize

[jira] [Commented] (YARN-914) Support graceful decommission of nodemanager

[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13709144#comment-13709144 ] Luke Lu commented on YARN-914: -- [~atm]: Nice catch! Of course :) Support

[jira] [Commented] (YARN-347) YARN node CLI should also show CPU info as memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721439#comment-13721439 ] Luke Lu commented on YARN-347: -- The patch lgtm. +1. Will commmit shortly.

[jira] [Updated] (YARN-347) YARN node CLI should also show CPU info besides memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN node CLI should also show CPU info besides memory info in node status (was: YARN node CLI should

[jira] [Updated] (YARN-347) YARN CLI should show CPU info besides memory info in node status

[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-347: - Summary: YARN CLI should show CPU info besides memory info in node status (was: YARN node CLI should also show

[jira] [Commented] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741713#comment-13741713 ] Luke Lu commented on YARN-291: -- It'll better if the node resource config is grouped into a

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765760#comment-13765760 ] Luke Lu commented on YARN-311: -- I think the v6.2 patch looks reasonable. If there is no further

[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long

[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475811#comment-13475811 ] Luke Lu commented on YARN-151: -- The most straight forward fix would be upgrading to a more

[jira] [Commented] (YARN-151) Browser thinks RM main page JS is taking too long

[ https://issues.apache.org/jira/browse/YARN-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487337#comment-13487337 ] Luke Lu commented on YARN-151: -- 10k apps is a small amount of data (a few hundred KB to a few

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536228#comment-13536228 ] Luke Lu commented on YARN-223: -- # The changes to eliminate usage of ResourceCalculatorPlugin is

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536392#comment-13536392 ] Luke Lu commented on YARN-223: -- Upon second look, though the current

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536636#comment-13536636 ] Luke Lu commented on YARN-223: -- Committed to trunk and branch-2. Thanks Radim for the patch and

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537165#comment-13537165 ] Luke Lu commented on YARN-223: -- Sorry folks! I got too used to seeing mapreduce code in YARN :)

[jira] [Commented] (YARN-223) Change processTree interface to work better with native code

[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537166#comment-13537166 ] Luke Lu commented on YARN-223: -- I think the PreCommit test patch system should either handle

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542297#comment-13542297 ] Luke Lu commented on YARN-291: -- A simpler and more straightforward alternative would be

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542825#comment-13542825 ] Luke Lu commented on YARN-291: -- Regarding the race condition, I meant that you could change the

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599383#comment-13599383 ] Luke Lu commented on YARN-18: - Preliminary comments: # We should use net.topology.nodegroup.aware

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13607021#comment-13607021 ] Luke Lu commented on YARN-18: - Juping, you're right that ScheduledRequests is an app side

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618522#comment-13618522 ] Luke Lu commented on YARN-291: -- Resource scheduling is fundamentally centralized at RM. The

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618525#comment-13618525 ] Luke Lu commented on YARN-18: - bq. I see a lot of factories and then no changes to actually

[jira] [Commented] (YARN-291) Dynamic resource configuration on NM

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618697#comment-13618697 ] Luke Lu commented on YARN-291: -- bq. The NM is the node controller and responsible for managing

[jira] [Updated] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-291: - Summary: Dynamic resource configuration (was: Dynamic resource configuration on NM) bq. I have concerns on

[jira] [Commented] (YARN-291) Dynamic resource configuration

[ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619374#comment-13619374 ] Luke Lu commented on YARN-291: -- bq. Changes in NM capacity triggered from outside of the

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649862#comment-13649862 ] Luke Lu commented on YARN-18: - The latest patch is getting close. I think the naming of

[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650054#comment-13650054 ] Luke Lu commented on YARN-18: - Please find the summary/writeup in the attached Pluggable

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783446#comment-13783446 ] Luke Lu commented on YARN-938: -- Is jvm reuse (set to -1) turned on for Hadoop 1 runs?

[jira] [Commented] (YARN-7) Add support for DistributedShell to ask for CPUs along with memory

[ https://issues.apache.org/jira/browse/YARN-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790853#comment-13790853 ] Luke Lu commented on YARN-7: The latest patch lgtm. +1. Add support for DistributedShell to ask

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804038#comment-13804038 ] Luke Lu commented on YARN-311: -- v9 patch lgtm. +1. For the record, we've discussed the changes

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804320#comment-13804320 ] Luke Lu commented on YARN-311: -- [~djp]: Upon second thought, I think we should improve the

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804335#comment-13804335 ] Luke Lu commented on YARN-311: -- We can probably infer the option by an overcommit timeout

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804532#comment-13804532 ] Luke Lu commented on YARN-311: -- bq. Other policies/options will be addressed in YARN-999

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807484#comment-13807484 ] Luke Lu commented on YARN-311: -- I share the same concern with Bikas with the RMNode lock inside

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807773#comment-13807773 ] Luke Lu commented on YARN-311: -- Junping: your [previous

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808130#comment-13808130 ] Luke Lu commented on YARN-311: -- IMO, using an immutable ResourceOption object is cleaner than

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808557#comment-13808557 ] Luke Lu commented on YARN-311: -- With ResourceOption, you don't need a separate

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808600#comment-13808600 ] Luke Lu commented on YARN-311: -- bq. How it sounds to you? Sounds good to me :), assuming

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811714#comment-13811714 ] Luke Lu commented on YARN-311: -- v12 patch lgtm. +1. Will commit soon. Dynamic node resource

[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813139#comment-13813139 ] Luke Lu commented on YARN-311: -- [~djp]: Unfortunately YARN-1343 got in before I tried to merge

[jira] [Updated] (YARN-311) Dynamic node resource configuration: core scheduler changes

[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated YARN-311: - Fix Version/s: 2.3.0 Hadoop Flags: Reviewed Committed to trunk and branch-2. Thanks Junping for the patch!

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823155#comment-13823155 ] Luke Lu commented on YARN-938: -- Yes, it'd be great if [~mayank_bansal] can share the configs

[jira] [Commented] (YARN-312) Add updateNodeResource in ResourceManagerAdministrationProtocol

[ https://issues.apache.org/jira/browse/YARN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823513#comment-13823513 ] Luke Lu commented on YARN-312: -- v5.1 looks reasonable (the verbosity of pb impl stuff still

[jira] [Commented] (YARN-938) Hadoop 2 benchmarking

[ https://issues.apache.org/jira/browse/YARN-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851507#comment-13851507 ] Luke Lu commented on YARN-938: -- Thanks for the results Jeff!. It's interesting to note that the