[jira] [Assigned] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler

[ https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned YARN-3635: Assignee: Sandy Ryza (was: Wangda Tan) Get-queue-mapping should be a common interface of

[jira] [Updated] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler

[ https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-3635: - Assignee: Tan, Wangda (was: Sandy Ryza) Get-queue-mapping should be a common interface of YarnScheduler

[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler

[ https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629141#comment-14629141 ] Sandy Ryza commented on YARN-3635: -- BTW I got all this from QueuePlacementPolicy and

[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler

[ https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629139#comment-14629139 ] Sandy Ryza commented on YARN-3635: -- [~leftnoteasy], apologies for this quick drive-by

[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing

[ https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623047#comment-14623047 ] Sandy Ryza commented on YARN-3866: -- Hi [~jianhe]. Most application writers should be

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591055#comment-14591055 ] Sandy Ryza commented on YARN-1197: -- The latest proposal makes sense to me as well. Thanks

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588478#comment-14588478 ] Sandy Ryza commented on YARN-1197: -- bq. I think this assumes cluster is quite idle, I

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14586249#comment-14586249 ] Sandy Ryza commented on YARN-1197: -- Sorry, I've been quiet here for a while, but I'd be

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14586687#comment-14586687 ] Sandy Ryza commented on YARN-1197: -- bq. Going through RM directly is better as the RM will

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587127#comment-14587127 ] Sandy Ryza commented on YARN-1197: -- Option (a) can occur in the low hundreds of

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587072#comment-14587072 ] Sandy Ryza commented on YARN-1197: -- bq. RM still needs to wait for an acknowledgement from

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587067#comment-14587067 ] Sandy Ryza commented on YARN-1197: -- Is my understanding correct that the broader plan is

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587168#comment-14587168 ] Sandy Ryza commented on YARN-1197: -- bq. If you consider all now/future optimizations, such

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587174#comment-14587174 ] Sandy Ryza commented on YARN-1197: -- Regarding complexity in the AM, the NMClient utility

[jira] [Commented] (YARN-314) Schedulers should allow resource requests of different sizes at the same priority and location

[ https://issues.apache.org/jira/browse/YARN-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557912#comment-14557912 ] Sandy Ryza commented on YARN-314: - Do we have applications that need this capability?

[jira] [Commented] (YARN-3633) With Fair Scheduler, cluster can logjam when there are too many queues

[ https://issues.apache.org/jira/browse/YARN-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542897#comment-14542897 ] Sandy Ryza commented on YARN-3633: -- Another thought is that we could say the max AM share

[jira] [Commented] (YARN-5) Add support for FifoScheduler to schedule CPU along with memory.

[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14535008#comment-14535008 ] Sandy Ryza commented on YARN-5: --- +1 to Vinod's point Add support for FifoScheduler to schedule

[jira] [Updated] (YARN-810) Support CGroup ceiling enforcement on CPU

[ https://issues.apache.org/jira/browse/YARN-810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-810: Assignee: (was: Sandy Ryza) Support CGroup ceiling enforcement on CPU

[jira] [Commented] (YARN-3485) FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies

[ https://issues.apache.org/jira/browse/YARN-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518099#comment-14518099 ] Sandy Ryza commented on YARN-3485: -- It looks like the patch computes the headroom as

[jira] [Commented] (YARN-3485) FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies

[ https://issues.apache.org/jira/browse/YARN-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518222#comment-14518222 ] Sandy Ryza commented on YARN-3485: -- One nit: {code} +return Math.min( Math.min(value1,

[jira] [Updated] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a Fair Scheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-3415: - Summary: Non-AM containers can be counted towards amResourceUsage of a Fair Scheduler queue (was: Non-AM

[jira] [Commented] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391218#comment-14391218 ] Sandy Ryza commented on YARN-3415: -- +1 Non-AM containers can be counted towards

[jira] [Commented] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391624#comment-14391624 ] Sandy Ryza commented on YARN-3415: -- [~ragarwal] did you have any more comments before I

[jira] [Commented] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387425#comment-14387425 ] Sandy Ryza commented on YARN-3415: -- This looks mostly reasonable. A few comments: * In

[jira] [Commented] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385347#comment-14385347 ] Sandy Ryza commented on YARN-3415: -- Thanks for filing this [~ragarwal] and for taking this

[jira] [Updated] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-3415: - Target Version/s: 2.7.0, 2.6.1 Non-AM containers can be counted towards amResourceUsage of a

[jira] [Updated] (YARN-3415) Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue

[ https://issues.apache.org/jira/browse/YARN-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-3415: - Priority: Critical (was: Major) Non-AM containers can be counted towards amResourceUsage of a

[jira] [Commented] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests

[ https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310034#comment-14310034 ] Sandy Ryza commented on YARN-2990: -- +1. Sorry for the delay in getting to this.

[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it

[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307602#comment-14307602 ] Sandy Ryza commented on YARN-3101: -- +1 FairScheduler#fitInMaxShare was added to validate

[jira] [Updated] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share

[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-3101: - Summary: In Fair Scheduler, fix canceling of reservations for exceeding max share (was: Fix canceling of

[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it

[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298887#comment-14298887 ] Sandy Ryza commented on YARN-3101: -- [~adhoot] is this the same condition that's evaluated

[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it

[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299428#comment-14299428 ] Sandy Ryza commented on YARN-3101: -- In that case it sounds like the behavior is that we

[jira] [Commented] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests

[ https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287871#comment-14287871 ] Sandy Ryza commented on YARN-2990: -- Other than the addition of the anyLocalRequests check

[jira] [Commented] (YARN-2910) FSLeafQueue can throw ConcurrentModificationException

[ https://issues.apache.org/jira/browse/YARN-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234035#comment-14234035 ] Sandy Ryza commented on YARN-2910: -- Using a CopyOnWriteArrayList would make adding an

[jira] [Updated] (YARN-2910) FSLeafQueue can throw ConcurrentModificationException

[ https://issues.apache.org/jira/browse/YARN-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2910: - Assignee: Wilfred Spiegelenburg FSLeafQueue can throw ConcurrentModificationException

[jira] [Commented] (YARN-2669) FairScheduler: queueName shouldn't allow periods the allocation.xml

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221513#comment-14221513 ] Sandy Ryza commented on YARN-2669: -- This is looking good. A few comments. Can we add

[jira] [Commented] (YARN-2669) FairScheduler: queueName shouldn't allow periods the allocation.xml

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221604#comment-14221604 ] Sandy Ryza commented on YARN-2669: -- +1 FairScheduler: queueName shouldn't allow periods

[jira] [Updated] (YARN-2669) FairScheduler: queue names shouldn't allow periods

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2669: - Summary: FairScheduler: queue names shouldn't allow periods (was: FairScheduler: queueName shouldn't

[jira] [Updated] (YARN-2669) FairScheduler: queue names shouldn't allow periods

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2669: - Priority: Major (was: Minor) FairScheduler: queue names shouldn't allow periods

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212015#comment-14212015 ] Sandy Ryza commented on YARN-2811: -- This looks almost good to go - the last thing is that

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213045#comment-14213045 ] Sandy Ryza commented on YARN-2811: -- +1 Fair Scheduler is violating max memory settings

[jira] [Updated] (YARN-2811) In Fair Scheduler, reservation fulfillments shouldn't ignore max share

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2811: - Summary: In Fair Scheduler, reservation fulfillments shouldn't ignore max share (was: Fair Scheduler is

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208403#comment-14208403 ] Sandy Ryza commented on YARN-2811: -- IIUC, this looks like it will check the immediate

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203685#comment-14203685 ] Sandy Ryza commented on YARN-2811: -- I just realized an issue with this. maxResources can

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201884#comment-14201884 ] Sandy Ryza commented on YARN-2811: -- Cool, thanks for the updated patch. Are you able to

[jira] [Commented] (YARN-2811) Fair Scheduler is violating max memory settings in 2.4

[ https://issues.apache.org/jira/browse/YARN-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199980#comment-14199980 ] Sandy Ryza commented on YARN-2811: -- Thanks for uncovering this [~l201514]. I think that

[jira] [Commented] (YARN-2669) FairScheduler: print out a warning log when users provider a queueName starting with root. in the allocation.xml

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165496#comment-14165496 ] Sandy Ryza commented on YARN-2669: -- Might it make more sense to just throw a validation

[jira] [Commented] (YARN-2669) FairScheduler: print out a warning log when users provider a queueName starting with root. in the allocation.xml

[ https://issues.apache.org/jira/browse/YARN-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165642#comment-14165642 ] Sandy Ryza commented on YARN-2669: -- We shouldn't allow configured queue names to have

[jira] [Commented] (YARN-2635) TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS

[ https://issues.apache.org/jira/browse/YARN-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158328#comment-14158328 ] Sandy Ryza commented on YARN-2635: -- Parametrized should be spelled Paramet *e* rized. Can

[jira] [Created] (YARN-2643) Don't create a new DominantResourceCalculator on every FairScheduler.allocate call

Sandy Ryza created YARN-2643: Summary: Don't create a new DominantResourceCalculator on every FairScheduler.allocate call Key: YARN-2643 URL: https://issues.apache.org/jira/browse/YARN-2643 Project:

[jira] [Commented] (YARN-1414) with Fair Scheduler reserved MB in WebUI is leaking when killing waiting jobs

[ https://issues.apache.org/jira/browse/YARN-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157336#comment-14157336 ] Sandy Ryza commented on YARN-1414: -- Awesome with Fair Scheduler reserved MB in WebUI is

[jira] [Commented] (YARN-2635) TestRMRestart should run with all schedulers

[ https://issues.apache.org/jira/browse/YARN-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157624#comment-14157624 ] Sandy Ryza commented on YARN-2635: -- This seems like a good idea. A few stylistic

[jira] [Commented] (YARN-1414) with Fair Scheduler reserved MB in WebUI is leaking when killing waiting jobs

[ https://issues.apache.org/jira/browse/YARN-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156020#comment-14156020 ] Sandy Ryza commented on YARN-1414: -- [~jrottinghuis] I will take a look. [~l201514] mind

[jira] [Commented] (YARN-2596) TestWorkPreservingRMRestart for FairScheduler failed on trunk

[ https://issues.apache.org/jira/browse/YARN-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146921#comment-14146921 ] Sandy Ryza commented on YARN-2596: -- +1 pending jenkins TestWorkPreservingRMRestart for

[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144424#comment-14144424 ] Sandy Ryza commented on YARN-2252: -- +1 Intermittent failure for testcase

[jira] [Commented] (YARN-2555) Effective max-allocation-* should consider biggest node

[ https://issues.apache.org/jira/browse/YARN-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135013#comment-14135013 ] Sandy Ryza commented on YARN-2555: -- [~gp.leftnoteasy], this isn't the same as having an NM

[jira] [Commented] (YARN-415) Capture aggregate memory allocation at the app-level for chargeback

[ https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131908#comment-14131908 ] Sandy Ryza commented on YARN-415: - Awesome to see this go in! Capture aggregate memory

[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request

[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126005#comment-14126005 ] Sandy Ryza commented on YARN-2154: -- I'd like to add another constraint that I've been

[jira] [Commented] (YARN-2486) FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps

[ https://issues.apache.org/jira/browse/YARN-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117800#comment-14117800 ] Sandy Ryza commented on YARN-2486: -- Unfortunately these methods were made public in 2.5,

[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register

[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116828#comment-14116828 ] Sandy Ryza commented on YARN-2448: -- As Karthik mentioned, the ResourceCalculator is an

[jira] [Commented] (YARN-2422) yarn.scheduler.maximum-allocation-mb should not be hard-coded in yarn-default.xml

[ https://issues.apache.org/jira/browse/YARN-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101274#comment-14101274 ] Sandy Ryza commented on YARN-2422: -- I think it's weird to have a nodemanager property

[jira] [Commented] (YARN-2430) FairShareComparator: cache the results of getResourceUsage()

[ https://issues.apache.org/jira/browse/YARN-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101732#comment-14101732 ] Sandy Ryza commented on YARN-2430: -- I believe #3 is the best approach as it's more

[jira] [Commented] (YARN-2420) Fair Scheduler: change yarn.scheduler.fair.assignmultiple from boolean to integer

[ https://issues.apache.org/jira/browse/YARN-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14097937#comment-14097937 ] Sandy Ryza commented on YARN-2420: -- Does yarn.scheduler.fair.max.assign satisfy what

[jira] [Commented] (YARN-2420) Fair Scheduler: dynamically update yarn.scheduler.fair.max.assign based on cluster load

[ https://issues.apache.org/jira/browse/YARN-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14097961#comment-14097961 ] Sandy Ryza commented on YARN-2420: -- Cool. Regarding adjusting maxassign dynamically, my

[jira] [Commented] (YARN-2399) FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt

[ https://issues.apache.org/jira/browse/YARN-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094598#comment-14094598 ] Sandy Ryza commented on YARN-2399: -- +1 FairScheduler: Merge AppSchedulable and

[jira] [Commented] (YARN-2413) capacity scheduler will overallocate vcores

[ https://issues.apache.org/jira/browse/YARN-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094885#comment-14094885 ] Sandy Ryza commented on YARN-2413: -- The capacity scheduler truncates all vcore requests to

[jira] [Commented] (YARN-2413) capacity scheduler will overallocate vcores

[ https://issues.apache.org/jira/browse/YARN-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094899#comment-14094899 ] Sandy Ryza commented on YARN-2413: -- I believe this is the expected behavior (i.e. Capacity

[jira] [Commented] (YARN-2413) capacity scheduler will overallocate vcores

[ https://issues.apache.org/jira/browse/YARN-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094903#comment-14094903 ] Sandy Ryza commented on YARN-2413: -- I don't have an opinion on whether we should keep this

[jira] [Commented] (YARN-2399) FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt

[ https://issues.apache.org/jira/browse/YARN-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093592#comment-14093592 ] Sandy Ryza commented on YARN-2399: -- I noticed in FSAppAttempt there are some instance

[jira] [Commented] (YARN-2399) FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt

[ https://issues.apache.org/jira/browse/YARN-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093597#comment-14093597 ] Sandy Ryza commented on YARN-2399: -- Also, can we move all the methods that implement

[jira] [Commented] (YARN-807) When querying apps by queue, iterating over all apps is inefficient and limiting

[ https://issues.apache.org/jira/browse/YARN-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090420#comment-14090420 ] Sandy Ryza commented on YARN-807: - bq. If you think it's a bug, we can resolve it in

[jira] [Commented] (YARN-807) When querying apps by queue, iterating over all apps is inefficient and limiting

[ https://issues.apache.org/jira/browse/YARN-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090447#comment-14090447 ] Sandy Ryza commented on YARN-807: - I just remembered a couple reasons why it's important

[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance

[ https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089874#comment-14089874 ] Sandy Ryza commented on YARN-2352: -- My only comment is that I think it would make more

[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance

[ https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089880#comment-14089880 ] Sandy Ryza commented on YARN-2352: -- And also - is there a reason we need to change all the

[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance

[ https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089997#comment-14089997 ] Sandy Ryza commented on YARN-2352: -- +1 FairScheduler: Collect metrics on duration of

[jira] [Commented] (YARN-807) When querying apps by queue, iterating over all apps is inefficient and limiting

[ https://issues.apache.org/jira/browse/YARN-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090172#comment-14090172 ] Sandy Ryza commented on YARN-807: - Hi [~leftnoteasy], I think the expected behavior should

[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance

[ https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087887#comment-14087887 ] Sandy Ryza commented on YARN-2352: -- IIUC, this patch will only record the duration. If we

[jira] [Resolved] (YARN-2367) Make ResourceCalculator configurable for FairScheduler and FifoScheduler like CapacityScheduler

[ https://issues.apache.org/jira/browse/YARN-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved YARN-2367. -- Resolution: Not a Problem Hi Swapnil, The Fair Scheduler supports this through a different interface.

[jira] [Commented] (YARN-2328) FairScheduler: Verify update and continuous scheduling threads are stopped when the scheduler is stopped

[ https://issues.apache.org/jira/browse/YARN-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071361#comment-14071361 ] Sandy Ryza commented on YARN-2328: -- {code} -if (node != null

[jira] [Updated] (YARN-2313) Livelock can occur in FairScheduler when there are lots of running apps

[ https://issues.apache.org/jira/browse/YARN-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2313: - Summary: Livelock can occur in FairScheduler when there are lots of running apps (was: Livelock can

[jira] [Commented] (YARN-2313) Livelock can occur on FairScheduler when there are lots entry in queue

[ https://issues.apache.org/jira/browse/YARN-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067998#comment-14067998 ] Sandy Ryza commented on YARN-2313: -- Thanks for reporting this [~ozawa]. A couple nits: *

[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068017#comment-14068017 ] Sandy Ryza commented on YARN-796: - I'm worried that the proposal is becoming too complex.

[jira] [Commented] (YARN-2323) FairShareComparator creates too much Resource object

[ https://issues.apache.org/jira/browse/YARN-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068143#comment-14068143 ] Sandy Ryza commented on YARN-2323: -- As it's a static final variable, ONE should be all

[jira] [Updated] (YARN-2323) FairShareComparator creates too many Resource objects

[ https://issues.apache.org/jira/browse/YARN-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2323: - Summary: FairShareComparator creates too many Resource objects (was: FairShareComparator creates too

[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues

[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063749#comment-14063749 ] Sandy Ryza commented on YARN-2257: -- [~wangda] I agree with you that expecting admins to

[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues

[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060982#comment-14060982 ] Sandy Ryza commented on YARN-2257: -- Wangda, That policy would work well for a some

[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059074#comment-14059074 ] Sandy Ryza commented on YARN-796: - +1 on reducing the complexity of the label predicates.

[jira] [Commented] (YARN-2274) FairScheduler: Add debug information about cluster capacity, availability and reservations

[ https://issues.apache.org/jira/browse/YARN-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059292#comment-14059292 ] Sandy Ryza commented on YARN-2274: -- {code} +if (--updatesToSkipForDebug 0) { +

[jira] [Commented] (YARN-2026) Fair scheduler : Fair share for inactive queues causes unfair allocation in some scenarios

[ https://issues.apache.org/jira/browse/YARN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058009#comment-14058009 ] Sandy Ryza commented on YARN-2026: -- I think Ashwin makes a good point. I think displaying

[jira] [Commented] (YARN-2274) FairScheduler: Add debug information about cluster capacity, availability and reservations

[ https://issues.apache.org/jira/browse/YARN-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058085#comment-14058085 ] Sandy Ryza commented on YARN-2274: -- Demanded resources could also be a useful statistic to

[jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues

[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054617#comment-14054617 ] Sandy Ryza commented on YARN-2257: -- To add some background: since MR1, the Fair Scheduler

[jira] [Commented] (YARN-2026) Fair scheduler : Fair share for inactive queues causes unfair allocation in some scenarios

[ https://issues.apache.org/jira/browse/YARN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054629#comment-14054629 ] Sandy Ryza commented on YARN-2026: -- I had a conversation with [~kkambatl] about this, and

[jira] [Commented] (YARN-2257) Add user to queue mapping in Fair-Scheduler

[ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053846#comment-14053846 ] Sandy Ryza commented on YARN-2257: -- Definitely needed. This should be implemented as a

[jira] [Commented] (YARN-2250) FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical

[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052497#comment-14052497 ] Sandy Ryza commented on YARN-2250: -- +1. A couple lines go over 80 characters and the

[jira] [Updated] (YARN-2250) Moving apps between queues - FairScheduler

[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2250: - Target Version/s: 2.6.0 Fix Version/s: (was: 3.0.0) Moving apps between queues -

[jira] [Commented] (YARN-2250) Moving apps between queues - FairScheduler

[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051169#comment-14051169 ] Sandy Ryza commented on YARN-2250: -- Hi Krisztian, Would you mind including an example of a

[jira] [Commented] (YARN-2250) Moving apps between queues - FairScheduler

[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051692#comment-14051692 ] Sandy Ryza commented on YARN-2250: -- I think the bug can be fixed by replacing

[jira] [Updated] (YARN-2250) FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical

[ https://issues.apache.org/jira/browse/YARN-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2250: - Summary: FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical (was: Moving

[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness

[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044066#comment-14044066 ] Sandy Ryza commented on YARN-2214: -- Makes sense preemptContainerPreCheck() in

[jira] [Commented] (YARN-941) RM Should have a way to update the tokens it has for a running application

[ https://issues.apache.org/jira/browse/YARN-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039446#comment-14039446 ] Sandy Ryza commented on YARN-941: - Just to make sure we're not getting tripped up on

  1   2   3   4   5   6   7   8   9   10   >