[jira] [Commented] (YARN-1198) Capacity Scheduler headroom calculation does not work as expected

[ https://issues.apache.org/jira/browse/YARN-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109738#comment-14109738 ] Craig Welch commented on YARN-1198: --- I initially considered an approach like this one, bu

[jira] [Commented] (YARN-1857) CapacityScheduler headroom doesn't account for other AM's running

[ https://issues.apache.org/jira/browse/YARN-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109798#comment-14109798 ] Craig Welch commented on YARN-1857: --- [~jianhe] [~wangda] could you have a look at this pa

[jira] [Updated] (YARN-1198) Capacity Scheduler headroom calculation does not work as expected

[ https://issues.apache.org/jira/browse/YARN-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-1198: -- Attachment: YARN-1198.8.patch Patch based on my last comment which iterates/calculates headroom at the u

[jira] [Commented] (YARN-1198) Capacity Scheduler headroom calculation does not work as expected

[ https://issues.apache.org/jira/browse/YARN-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122188#comment-14122188 ] Craig Welch commented on YARN-1198: --- [~jianhe], have a look at patch 7, it takes that sor

[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127329#comment-14127329 ] Craig Welch commented on YARN-796: -- This is a bit of a detail, but the current version of t

[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129024#comment-14129024 ] Craig Welch commented on YARN-796: -- So, I'm adding code to check whether a user should be a

[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests

[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129480#comment-14129480 ] Craig Welch commented on YARN-796: -- Good, what you describe wrt the cli is what I was tryin

[jira] [Updated] (YARN-1857) CapacityScheduler headroom doesn't account for other AM's running

[ https://issues.apache.org/jira/browse/YARN-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-1857: -- Attachment: YARN-1857.2.patch I went ahead and put in a comment wrt a headroom value in the test and chan

[jira] [Commented] (YARN-2925) Internal fields in LeafQueue access should be protected when accessed from FiCaSchedulerApp to calculate Headroom

[ https://issues.apache.org/jira/browse/YARN-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235924#comment-14235924 ] Craig Welch commented on YARN-2925: --- Hey Wangda, couple questions: Have you actually see

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.15.patch Attached .15 patch which implements option 2 > maximum-am-resource-percen

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.16.patch .16 the same as .15 but with additional verification of the effect of the

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236389#comment-14236389 ] Craig Welch commented on YARN-2637: --- Something to be aware of wrt the way I implemented o

[jira] [Commented] (YARN-2925) Internal fields in LeafQueue access should be protected when accessed from FiCaSchedulerApp to calculate Headroom

[ https://issues.apache.org/jira/browse/YARN-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236461#comment-14236461 ] Craig Welch commented on YARN-2925: --- So, a few things things: I can see how we could get

[jira] [Commented] (YARN-2925) Internal fields in LeafQueue access should be protected when accessed from FiCaSchedulerApp to calculate Headroom

[ https://issues.apache.org/jira/browse/YARN-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238155#comment-14238155 ] Craig Welch commented on YARN-2925: --- Hmm, there might be an even simpler approach - if we

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.17.patch Made [~djp] 's recommended changes, thanks for taking a look. [~leftnotea

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243065#comment-14243065 ] Craig Welch commented on YARN-2637: --- I double checked - none of the findbugs warnings are

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243171#comment-14243171 ] Craig Welch commented on YARN-2495: --- Sorry if I'm jumping in late and asking redundant qu

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243197#comment-14243197 ] Craig Welch commented on YARN-2495: --- So, I see the language around the script based vs th

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243283#comment-14243283 ] Craig Welch commented on YARN-2495: --- Other comments on the patch as such, assuming we rea

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.18.patch Update with some changes based on [~leftnoteasy] 's comments. > maximum-a

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250908#comment-14250908 ] Craig Welch commented on YARN-2637: --- [~leftnoteasy] updated patch with some changes based

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.19.patch Modified patch to use minimum allocation value if application master resou

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14252620#comment-14252620 ] Craig Welch commented on YARN-2637: --- Minor update - TestAllocationFileLoaderService passe

[jira] [Commented] (YARN-1680) availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

[ https://issues.apache.org/jira/browse/YARN-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253765#comment-14253765 ] Craig Welch commented on YARN-1680: --- Go for it :-) I thought I was free to work it, and a

[jira] [Updated] (YARN-1680) availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

[ https://issues.apache.org/jira/browse/YARN-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-1680: -- Assignee: Chen He (was: Craig Welch) > availableResources sent to applicationMaster in heartbeat should

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.20.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.21.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.22.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.23.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.25.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263608#comment-14263608 ] Craig Welch commented on YARN-2637: --- bq. I think there should at least one AM can be laun

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265480#comment-14265480 ] Craig Welch commented on YARN-2637: --- bq. Regarding null checks in FiCaSchedulerApp. Sinc

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.26.patch reformatted some sections of testleafqueue, commenting the null check for

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.27.patch patch tests which fail when null check for rmcontext.getscheduler is not

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.28.patch > maximum-am-resource-percent could be violated when resource of AM is >

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267030#comment-14267030 ] Craig Welch commented on YARN-2637: --- Findbugs was the result of changing the ratio of syn

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.29.patch Take a go adding user am limit also (needs further verification/test), see

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be violated when resource of AM is > minimumAllocation

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.30.patch userAMLimit logic included as well, now with a test :-) > maximum-am-reso

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.31.patch See what happens when maxActiveApplications and maxActiveApplicationsPerUs

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.32.patch Check tests using absoluteCapacity for userAmLimit > maximum-am-resource-

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.36.patch Should be down to one failing test, let's see > maximum-am-resource-perce

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.38.patch Should be the core logic but still plan to add some additional tests & to

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.39.patch Now with web ui entries max am and max am user resource + application limi

[jira] [Commented] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275700#comment-14275700 ] Craig Welch commented on YARN-2637: --- Regarding the findbugs report for LeafQueue.lastClus

[jira] [Updated] (YARN-2637) maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.

[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-2637: -- Attachment: YARN-2637.40.patch > maximum-am-resource-percent could be respected for both LeafQueue/User w

[jira] [Commented] (YARN-1680) availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

[ https://issues.apache.org/jira/browse/YARN-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281058#comment-14281058 ] Craig Welch commented on YARN-1680: --- Thanks for the update, [~airbots], a couple thoughts

[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"

[ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284304#comment-14284304 ] Craig Welch commented on YARN-1039: --- As I understand it (and, I may be wrong on this...)

[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"

[ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284325#comment-14284325 ] Craig Welch commented on YARN-1039: --- Another thought - if we do need this kind of "flag",

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294441#comment-14294441 ] Craig Welch commented on YARN-2495: --- [~Naganarasimha] I understand the desire to have the

[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"

[ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294557#comment-14294557 ] Craig Welch commented on YARN-1039: --- [~chris.douglas] what's the proper duration for a se

[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"

[ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297783#comment-14297783 ] Craig Welch commented on YARN-1039: --- [~chris.douglas] bq. YARN shouldn't understand the

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323076#comment-14323076 ] Craig Welch commented on YARN-2495: --- My point is that everything necessary to manage labe

[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)

[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327716#comment-14327716 ] Craig Welch commented on YARN-2495: --- So, here's my proposal [~Naganarasimha] [~leftnoteas

[jira] [Commented] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335765#comment-14335765 ] Craig Welch commented on YARN-3251: --- It looks like this can occur when a call which walks

[jira] [Updated] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3251: -- Attachment: YARN-3251.1.patch > CapacityScheduler deadlock when computing absolute max avail capacity > -

[jira] [Updated] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3251: -- Attachment: YARN-3251.2-6-0.2.patch Patch against branch-2.6.0 > CapacityScheduler deadlock when computi

[jira] [Commented] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337962#comment-14337962 ] Craig Welch commented on YARN-3251: --- bq. 1) Since the target of your patch is to make a q

[jira] [Updated] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3251: -- Attachment: YARN-3251.2-6-0.3.patch Removing the csqueueutils > CapacityScheduler deadlock when computin

[jira] [Commented] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338848#comment-14338848 ] Craig Welch commented on YARN-3251: --- Sorry if that wasn't clear, to reduce risk removed t

[jira] [Updated] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3251: -- Attachment: YARN-3251.2-6-0.4.patch Minor, switch to "Internal", seems to be more common in the codebase

[jira] [Updated] (YARN-3251) CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)

[ https://issues.apache.org/jira/browse/YARN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated YARN-3251: -- Attachment: YARN-3251.2.patch Attaching an analogue of the most recent patch against trunk. I do not bel

<    1   2   3   4   5