Eric Payne updated YARN-2932:
    Attachment: YARN-2932.v4.txt

Thank you [~leftnoteasy] for your review and comments.

Re 2:
 You're partially correct, queue finally calls setupQueueConfig when 
reinitialize is invoked. 
 The CapacityScheduler reinitialization is creating a new set of queues, and 
copy new parameters to your old queues via
So you need put the parameter you wants to update to setupQueueConfig as well. 
Without that, queue will not be refreshed. I didn't find any changes to 
parameter of setupQueueConfig, so I guess so, it's better to add a test to 
verify it.
I made the changes to the API for {{AbstractCSQueue#setupQueueConfigs}} to take 
the additional preemptable parameter. When it is called from 
{{[Leaf|Parent]Queue#setupQueueConfigs}}, it calls 
{{AbstractCSQueue#isQueuePathHierarchyPreemptable}} to get the preemptability 
of the queue.

I tested the fixes in both version 3 and version 4 of this patch on a one-node 
cluster and on a 10-node cluster. In both version, I was able to change the 
{{disable_preemption}} properties, refresh the queues using {{yarn rmadmin 
-refreshQueues}}, and I was able to see the updates on the Scheduler UI page. 
However, I think I see that if the new list of queues is different than the old 
list of queues, it would not pick up the parameters for the new queues without 
this change.

Re 3:
You can take a look at how AbstractCSQueue initialize labels,

I think they have similar logic – For node label is trying to get value from 
configuration, if not set, inherit from parent. With this, you can make 
getPreemptable interface without defaultVal in CapacitySchedulerConfiguration.
I did change {{CapacitySchedulerConfiguration#getQueuePreemptable}} to not take 
a default value, but in order to pass back the {{null}} information, it has to 
return a {{String}} and then the caller has to convert the {{String}} to a 
Boolean, which I think is a little awkward.

Since YARN-2056 is also planned in 2.7 (I thought it's already included in 
2.6), do you think is it better to make configuration option name to 
<queue-patch>.preemptable for consistency?
Well, that would be ideal, I think, but it isn't that simple on our side. We 
have already started using the code in YARN-2056 and are using the 
{{disable_preemption}} property.

An argument could be made that {{disable_preemption}} is better because it 
indicates that it is turning off the {{...monitor.capacity.preemption...}} 
property. If {{disable_preemption}} were changed to {{preemptable}}, someone 
may look at that property and think that the queue should have that property 
without considering the overall, system property 

How important is it to you that {{disable_preemption}} property be changed to 

> Add entry for "preemptable" status to scheduler web UI and queue 
> initialize/refresh logging
> -------------------------------------------------------------------------------------------
>                 Key: YARN-2932
>                 URL: https://issues.apache.org/jira/browse/YARN-2932
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.7.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: YARN-2932.v1.txt, YARN-2932.v2.txt, YARN-2932.v3.txt, 
> YARN-2932.v4.txt
> YARN-2056 enables the ability to turn preemption on or off on a per-queue 
> level. This JIRA will provide the preemption status for each queue in the 
> {{HOST:8088/cluster/scheduler}} UI and in the RM log during startup/queue 
> refresh.

This message was sent by Atlassian JIRA

Reply via email to