Eric Payne updated YARN-2056:
    Attachment: YARN-2056.201409210049.txt

Hi [~leftnoteasy]. Thank you for spending the time to look at this patch and 
provide helpful suggestions.

IMHO, the right place to put reserving resource logic for un-preemptable queue 
is not {{resetCapacity}}, it should in {{computeFixpointAllocation}}.
Does this make sense to you?

Yes, that makes sense, and I think it is a simpler algorithm. I updated the 
patch, so please have a look.

I have made a conscious decision to only allow disable preemption at the leaf 
queue level. This is because there may be a use case where you want to disable 
preemption at the parent level, and have other queue hierarchies leave it 
alone, but then allow preemption between children of the disabled parent. So, 
rather than solve that problem with this fix, I only allow leaf queues to 
disable preemption. Even if a leaf queue could inherit it's parent's disable 
preemption value, there will likely be cases where part of the parent queue's 
over-capacity resources are untouchable and part of them are preemptable.

So, I adjusted your suggested algorithm somewhat. 

- I collected untouchableExtra instead of preemptableExtra at the TempQueue 
level. in {{computeFixpointAllocation}},
- I looped through each queue, and if one has any untouchableExtra, then the 
queue's {{idealAssigned = guaranteed + untouchableExtra}}
- In {{TempQueue#offer}}, one of the calculations is {{current + pending - 
idealAssigned}}. I had to take into consideration that if the queue has over 
capacity, some of it may be untouchable and some may be preemptable. If some of 
it is preemptable, then {{current}} could be greater than {{idealAssigned}}, 
and {{TempQueue#offer}} would end up assigning more to that queue than it 

> Disable preemption at Queue level
> ---------------------------------
>                 Key: YARN-2056
>                 URL: https://issues.apache.org/jira/browse/YARN-2056
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Mayank Bansal
>            Assignee: Eric Payne
>         Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt, 
> YARN-2056.201408310117.txt, YARN-2056.201409022208.txt, 
> YARN-2056.201409181916.txt, YARN-2056.201409210049.txt
> We need to be able to disable preemption at individual queue level

This message was sent by Atlassian JIRA

Reply via email to