[jira] [Commented] (YARN-4362) Too many preemption activity when nodelabels are non exclusive
[ https://issues.apache.org/jira/browse/YARN-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008774#comment-15008774 ] Varun Saxena commented on YARN-4362: bq. We should restrict assigning to partition 2. I think it depends on how to treat non exclusive partition. Does it make sense to have a non exclusive partition without it being accessible from any queue and assign to it ? Not sure if there is a use case for this. If we consider it as free for all kind of partition and its alright to assign containers to such a partition, we need to fix the preemption logic. Because such assignments will be preempted despite only one app running. However, if we consider that such a partition has no real meaning until and unless its accessible from a queue, you are correct that we should not assign to it. Although in this case we can argue that why have a node assigned to a partition if its not accessible from any queue. But I guess there can be some scenarios where such a situation is possible. Wangda, thoughts on this ? > Too many preemption activity when nodelabels are non exclusive > -- > > Key: YARN-4362 > URL: https://issues.apache.org/jira/browse/YARN-4362 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Varun Saxena >Priority: Critical > Attachments: Preemptedpartition.log, ProportionalDefaultQueue.log, > ProportionalPolicy.log, capacity-scheduler.xml > > > Steps to reproduce > === > 1.Configure HA cluster with 6 nodes and 3 partition(1,2,3) all non exclusive > *Partition configuration is as follows* > 1,2 NM's mapped with Label 1 > NM 3 to label 2 > 4,5 NM's mapped to Label 3 > NM 6 in DEFAULT partition > In capacity scheduler the queue are linked only to 1,3 partition. > The NM 3 with label 2 is a backup node for any partition whenever required > will change the label. > Submit and application/job with 200 containers to default queue. > All containers that gets assigned to partition 2 gets preempted > The application/map task execution is taking more time since 30-40 task gets > assigned to partition 2 then gets preempted and all of them needs to be > relaunched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4362) Too many preemption activity when nodelabels are non exclusive
[ https://issues.apache.org/jira/browse/YARN-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008812#comment-15008812 ] Sunil G commented on YARN-4362: --- Ideally at first thought we should fix at preemption logic as more preemption is happening there. But in this specific case, some non-used partition has resources in it, and preemption logic will always mark this as over-used resources. I am not very much inclined in having a new non-exclusive partition which makes allocation more complex (we have no_label already for general cases). However I feel we can come with a default allocation limit/unit per partition even though it is not used/configured. Hence in cases like this, guaranteed resources wont be Zero. But when real allocation starts happening to that label, this default allocation must be removed/shrinked. With this we can get some flexibility in using these free nodes w/o violating node label strict labeling concept (only no_label can use this limit). If we do not allocate to these labels, it will be more stricter and these resource will be non-usable. This is also acceptable but comes with the cost of some non-usable nodes. > Too many preemption activity when nodelabels are non exclusive > -- > > Key: YARN-4362 > URL: https://issues.apache.org/jira/browse/YARN-4362 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Varun Saxena >Priority: Critical > Attachments: Preemptedpartition.log, ProportionalDefaultQueue.log, > ProportionalPolicy.log, capacity-scheduler.xml > > > Steps to reproduce > === > 1.Configure HA cluster with 6 nodes and 3 partition(1,2,3) all non exclusive > *Partition configuration is as follows* > 1,2 NM's mapped with Label 1 > NM 3 to label 2 > 4,5 NM's mapped to Label 3 > NM 6 in DEFAULT partition > In capacity scheduler the queue are linked only to 1,3 partition. > The NM 3 with label 2 is a backup node for any partition whenever required > will change the label. > Submit and application/job with 200 containers to default queue. > All containers that gets assigned to partition 2 gets preempted > The application/map task execution is taking more time since 30-40 task gets > assigned to partition 2 then gets preempted and all of them needs to be > relaunched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4362) Too many preemption activity when nodelabels are non exclusive
[ https://issues.apache.org/jira/browse/YARN-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008072#comment-15008072 ] Bibin A Chundatt commented on YARN-4362: Attached logs and xml. Looks like the guaranteed resource for partition 2 for queue default will be always zero. So any container assigned to partition 2 will get preempted from ProportionalCapacityPreemptionPolicy even when no other application is running. We should restrict assigning to partition 2. Thoughts? > Too many preemption activity when nodelabels are non exclusive > -- > > Key: YARN-4362 > URL: https://issues.apache.org/jira/browse/YARN-4362 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Varun Saxena >Priority: Critical > Attachments: Preemptedpartition.log, ProportionalDefaultQueue.log, > ProportionalPolicy.log, capacity-scheduler.xml > > > Steps to reproduce > === > 1.Configure HA cluster with 6 nodes and 3 partition(1,2,3) all non exclusive > *Partition configuration is as follows* > 1,2 NM's mapped with Label 1 > NM 3 to label 2 > 4,5 NM's mapped to Label 3 > NM 6 in DEFAULT partition > In capacity scheduler the queue are linked only to 1,3 partition. > The NM 3 with label 2 is a backup node for any partition whenever required > will change the label. > Submit and application/job with 200 containers to default queue. > All containers that gets assigned to partition 2 gets preempted > The application/map task execution is taking more time since 30-40 task gets > assigned to partition 2 then gets preempted and all of them needs to be > relaunched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
