[
https://issues.apache.org/jira/browse/YARN-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-2932:
-----------------------------
Attachment: YARN-2932.v4.txt
Thank you [~leftnoteasy] for your review and comments.
{quote}
Re 2:
You're partially correct, queue finally calls setupQueueConfig when
reinitialize is invoked.
The CapacityScheduler reinitialization is creating a new set of queues, and
copy new parameters to your old queues via
{code}
setupQueueConfigs(
clusterResource,
newlyParsedLeafQueue.capacity,
...
{code}
So you need put the parameter you wants to update to setupQueueConfig as well.
Without that, queue will not be refreshed. I didn't find any changes to
parameter of setupQueueConfig, so I guess so, it's better to add a test to
verify it.
{quote}
I made the changes to the API for {{AbstractCSQueue#setupQueueConfigs}} to take
the additional preemptable parameter. When it is called from
{{[Leaf|Parent]Queue#setupQueueConfigs}}, it calls
{{AbstractCSQueue#isQueuePathHierarchyPreemptable}} to get the preemptability
of the queue.
I tested the fixes in both version 3 and version 4 of this patch on a one-node
cluster and on a 10-node cluster. In both version, I was able to change the
{{disable_preemption}} properties, refresh the queues using {{yarn rmadmin
-refreshQueues}}, and I was able to see the updates on the Scheduler UI page.
However, I think I see that if the new list of queues is different than the old
list of queues, it would not pick up the parameters for the new queues without
this change.
{quote}
Re 3:
You can take a look at how AbstractCSQueue initialize labels,
I think they have similar logic – For node label is trying to get value from
configuration, if not set, inherit from parent. With this, you can make
getPreemptable interface without defaultVal in CapacitySchedulerConfiguration.
{quote}
I did change {{CapacitySchedulerConfiguration#getQueuePreemptable}} to not take
a default value, but in order to pass back the {{null}} information, it has to
return a {{String}} and then the caller has to convert the {{String}} to a
Boolean, which I think is a little awkward.
{quote}
Since YARN-2056 is also planned in 2.7 (I thought it's already included in
2.6), do you think is it better to make configuration option name to
<queue-patch>.preemptable for consistency?
{quote}
Well, that would be ideal, I think, but it isn't that simple on our side. We
have already started using the code in YARN-2056 and are using the
{{disable_preemption}} property.
An argument could be made that {{disable_preemption}} is better because it
indicates that it is turning off the {{...monitor.capacity.preemption...}}
property. If {{disable_preemption}} were changed to {{preemptable}}, someone
may look at that property and think that the queue should have that property
without considering the overall, system property
{{...monitor.capacity.preemption...}}.
How important is it to you that {{disable_preemption}} property be changed to
{{preemptable}}?
> Add entry for "preemptable" status to scheduler web UI and queue
> initialize/refresh logging
> -------------------------------------------------------------------------------------------
>
> Key: YARN-2932
> URL: https://issues.apache.org/jira/browse/YARN-2932
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.0.0, 2.7.0
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: YARN-2932.v1.txt, YARN-2932.v2.txt, YARN-2932.v3.txt,
> YARN-2932.v4.txt
>
>
> YARN-2056 enables the ability to turn preemption on or off on a per-queue
> level. This JIRA will provide the preemption status for each queue in the
> {{HOST:8088/cluster/scheduler}} UI and in the RM log during startup/queue
> refresh.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
