[
https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-2056:
-----------------------------
Attachment: YARN-2056.201411132215.txt
Thank you very much [~leftnoteasy]. Here is the patch with my changes.
{quote}
1) Typo of comment in cloneQueue:
2. The logic should be correct, but I think it might be simpler to say:
untouchableExtra = max(extra - childrenPreemptable, 0) and as same as the code.
{quote}
Comment changed
{quote}
3)
bq. public double getIdealPctOfGuaranteed(TempQueue q)
The method doesn't need to be public anymore
{quote}
Changed to {{private}}
{quote}
4) Does it possible there's only one queue in getMostUnderservedQueues?If so,
you should check if q2 is null
{quote}
Deep down in {{getIdealPctOfGuaranteed}}, it eventually does the null check,
but I added a null check in {{getMostUnderservedQueues}} as well. Better safe
than sorry :-)
{quote}
1) testDisablePreemptionOverCapPlusPending
Should disable queueB instead of queueA? Currently, the test will preempt from
appB not matter if preemption disabled for queueA or not
{quote}
The point of that test was to indicate that preemption levelization will still
happen even if the thing asking for resources is the one that is untouchable.
If you think this test is unnecessary, I will take it out.
{quote}
2) changes for t{{estHierarchicalLarge}}:
I'm a little concern about this change, even if we considering round error,
appA should be taken about 9-10 resources, 9->6 seems some potential bug caused
issue, could you double check if it works as expected? (Without affect the
normal preemption logic).
3) As above for {{testSkipAMContainer}}
I suggest you can take some investigation about why some original numbers need
to be changed, if it is just a round problem, that should be fine, but we
should avoid behavior changes.
{quote}
Yes, these changes are definitely due to rounding. There are 2 things that
cause the rounding problem:
# The new algorithm in {{computeFixpointAllocation}} is different, so you would
expect there to be differences in rounding.
#* One oddity about this algorithm is that it is still using
{{normalizedGuarantee}} to calculate {{wQavail}}.
#* if you have A and B, and B is less served than A, when it gets down to the
last one or two resources, it will try to multiply 1 or 2 by
{{normalizedGuarantee}} and will offer B 0, so the last 1 or 2 go to A.
#* This shows up noticeably in the unit tests where there are a smaller total
number of resources.
#* The algorithm could do a couple of things to clean this up.
#* it could always round the wQavail
#* It could also calculate how much resources it would take to get B up to the
level of the next thing on the underserved queue, and if there are unassigned
resources, just offer them to B.
# The other thing that is causing rounding is that when {{cloneQueues}} gets
{{root.getAbsoluteUsedCapacity()}}, it often comes out as something
0.5249999761581421 instead of 0.525, and so when that is used to calculate
{{current}}, it thinks a queue's current is really one less than what it should
be.
#* Something that could be done here is we could round to the nearest 3 decimal
places, for example. 3 decimal places is what shows up in the scheduler UI
metrics.
> Disable preemption at Queue level
> ---------------------------------
>
> Key: YARN-2056
> URL: https://issues.apache.org/jira/browse/YARN-2056
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.4.0
> Reporter: Mayank Bansal
> Assignee: Eric Payne
> Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt,
> YARN-2056.201408310117.txt, YARN-2056.201409022208.txt,
> YARN-2056.201409181916.txt, YARN-2056.201409210049.txt,
> YARN-2056.201409232329.txt, YARN-2056.201409242210.txt,
> YARN-2056.201410132225.txt, YARN-2056.201410141330.txt,
> YARN-2056.201410232244.txt, YARN-2056.201410311746.txt,
> YARN-2056.201411041635.txt, YARN-2056.201411072153.txt,
> YARN-2056.201411122305.txt, YARN-2056.201411132215.txt
>
>
> We need to be able to disable preemption at individual queue level
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
