[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357871#comment-15357871 ] Jonathan Hung commented on YARN-5293: - [~sunilg] and [~leftnoteasy], thanks for the clarifications. I think these make sense. One concern I have is regarding when label exclusivity is disabled. If I understand correctly, [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] keeps track of the containers which ignore partition exclusivity. So if we disable preemption for a queue-label, we can still preempt the containers with NO_LABEL running on a partition. That way specifying preemption disabled for a queue-label only disables preemption for containers with that label so we can avoid the scenario of indirectly disabling preemption for NO_LABEL containers on non-default partitions. > Disable preemption per queue per partition > -- > > Key: YARN-5293 > URL: https://issues.apache.org/jira/browse/YARN-5293 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Jonathan Hung > > [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for > disabling preemption at queue level. > [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for > per partition preemption. It would be useful to be able to disable preemption > per partition per queue, so that if a queue Q has access to two labels A and > B, we can disable preemption for Q in A but not in B. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357609#comment-15357609 ] Wangda Tan commented on YARN-5293: -- Thanks for discussion, [~jhung], [~sunilg], [~Naganarasimha]. I can understand the use case for per-queue-per-partition preemption disable. In my mind, node partition should be considered as an individual cluster, if we allow preemption disable for default partition, we should allow preemption disable for other queues. One straightforward use case is, one queue can access multi partitions, applications in some partitions are more important than others. I don't think we should add per-partition preemption, this brings more issues about which configuration will be honored (assume we have queue.disable-preemption / queue.label.disable-preemption / label.disable-preemption). For configuration: Existing queue.disable-preemption means *disable preemption for all partitions of this queue*. We should keep this semantic, otherwise it becomes a incompatible change. In my mind: queue.label.disable-preemption should be able to overwrite queue.disable-preemption. For sequence of configuration override: - non-default-partition: this-queue.label.disable-preemption > this-queue.disable-preemption > parent-queue.label.disable-preemption > parent-queue.disable-preemption. - default-partition: this-queue.disable-preemption > parent-queue.disable-preemption (as same as now). Thoughts? > Disable preemption per queue per partition > -- > > Key: YARN-5293 > URL: https://issues.apache.org/jira/browse/YARN-5293 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Jonathan Hung > > [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for > disabling preemption at queue level. > [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for > per partition preemption. It would be useful to be able to disable preemption > per partition per queue, so that if a queue Q has access to two labels A and > B, we can disable preemption for Q in A but not in B. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356832#comment-15356832 ] Sunil G commented on YARN-5293: --- Thanks [~jhung] for a detailed idea about the configurations and its priorities. I am thinking about the advantages of disabling preemption for per-label-per-queue level. Since labels can give a physical categorization of nodes based on certain types (upto the admin to decide), we can think that one specific queue capacity can be splitted/viewed as few certain types of nods. So my *highGPU* label nodes for the queue can be disabled for preemption when compared with my *lowGPU*, thus keeping only some low priority resources for preemption. This can be viewed from both angle. I am saving some of my critical resources from getting preempted but still gives a certain percentage of resource which are of low priority to preempt. But there is a fair level balance needed, and it may be tough to maintain. Few points - *labelA* for queueA is now disabled for preemption and consider this queue is over-utilizing resources from *labelA*. *labelA* is configured in queueB also which is under utilized. So ideally *queueA-labelA* has to be preempted to satisfy queueB's need. This will be blocked now - If only one label is configured per-queue, its same queue level disabling - With label exclusivity disabled, same label can be used for NO_LABEL cases too for a queue. So if this label is disabled from preemption, indirectly NO_LABEL resources also will be saved. Such cases are hard to explain or propagate to user/admin level for clarity. I think more discussions are good here. cc/[~leftnoteasy], [~Naganarasimha Garla], [~rohithsharma] > Disable preemption per queue per partition > -- > > Key: YARN-5293 > URL: https://issues.apache.org/jira/browse/YARN-5293 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Jonathan Hung > > [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for > disabling preemption at queue level. > [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for > per partition preemption. It would be useful to be able to disable preemption > per partition per queue, so that if a queue Q has access to two labels A and > B, we can disable preemption for Q in A but not in B. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[
https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351650#comment-15351650
]
Jonathan Hung commented on YARN-5293:
-
Thanks for the comments, [~Naganarasimha] and [~sunilg]. My initial idea was to
keep the preemption disabled a queue level configuration. So we would have
something like
{{yarn.scheduler.capacity.root.queueQ.accessible-node-labels.labelA.disable_preemption}}
(say this is set to true). This would override the configuration
{{yarn.scheduler.capacity.root.queueQ.disable_preemption}} (say this is set to
false) for labelA, so queueQ should not be preempted for labelA but will be
preempted for other labels.
For the case where
{{yarn.scheduler.capacity.root.queueQ.accessible-node-labels.labelA.disable_preemption}}
is not specified, it should inherit from its parent's per-queue-per-label
configuration (in this case,
{{yarn.scheduler.capacity.root.accessible-node-labels.labelA.disable_preemption}}),
just as it does in the existing code for unlabeled disable_preemption.
Can't currently think of any use cases for disabling preemption for a label. If
we wanted to do this, we could have a configuration like
{{yarn.scheduler.capacity.labelA.disable_preemption}}. If neither
{{yarn.scheduler.capacity.root.queueQ.accessible-node-labels.labelA.disable_preemption}}
or
{{yarn.scheduler.capacity.root.accessible-node-labels.labelA.disable_preemption}}
is specified, it will inherit from
{{yarn.scheduler.capacity.labelA.disable_preemption}}.
One of the results of this is that
{{yarn.scheduler.capacity.root.queueQ.disable_preemption}} no longer has any
effect on disabling preemption for queueQ in labeled partitions. So if a user
wanted to disable preemption for a queue (across all partitions), they would
have to specify this for each partition.
> Disable preemption per queue per partition
> --
>
> Key: YARN-5293
> URL: https://issues.apache.org/jira/browse/YARN-5293
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>
> [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for
> disabling preemption at queue level.
> [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for
> per partition preemption. It would be useful to be able to disable preemption
> per partition per queue, so that if a queue Q has access to two labels A and
> B, we can disable preemption for Q in A but not in B.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350860#comment-15350860 ] Sunil G commented on YARN-5293: --- Adding to what [~Naganarasimha Garla] has mentioned and based on the example mentioned in the description, label A may also be used in some other queue's. So if we are disabling for a label, then we may automatically disable preemption for some other queues automatically. I think think we need to define the order of disabling in correct way. Do we see any other use case advantages of disabling preemption per label? I think its better if we can carefully look into with more usability aspects also. Else this may add more complexity for user. > Disable preemption per queue per partition > -- > > Key: YARN-5293 > URL: https://issues.apache.org/jira/browse/YARN-5293 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung > > [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for > disabling preemption at queue level. > [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for > per partition preemption. It would be useful to be able to disable preemption > per partition per queue, so that if a queue Q has access to two labels A and > B, we can disable preemption for Q in A but not in B. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5293) Disable preemption per queue per partition
[ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350818#comment-15350818 ] Naganarasimha G R commented on YARN-5293: - [~jhung], Makes Sense to disable Per queue per partition but better to take it as override if required else follow the default behavior. > Disable preemption per queue per partition > -- > > Key: YARN-5293 > URL: https://issues.apache.org/jira/browse/YARN-5293 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung > > [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for > disabling preemption at queue level. > [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added support for > per partition preemption. It would be useful to be able to disable preemption > per partition per queue, so that if a queue Q has access to two labels A and > B, we can disable preemption for Q in A but not in B. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
