[jira] [Commented] (YARN-5542) Scheduling of opportunistic containers
[ https://issues.apache.org/jira/browse/YARN-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011829#comment-17011829 ] Konstantinos Karanasos commented on YARN-5542: -- Cool, thanks [~brahmareddy] and [~abmodi]. > Scheduling of opportunistic containers > -- > > Key: YARN-5542 > URL: https://issues.apache.org/jira/browse/YARN-5542 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Konstantinos Karanasos >Priority: Major > Fix For: 3.3.0 > > > This JIRA groups all efforts related to the scheduling of opportunistic > containers. > It includes the scheduling of opportunistic container through the central RM > (YARN-5220), through distributed scheduling (YARN-2877), as well as the > scheduling of containers based on actual node utilization (YARN-1011) and the > container promotion/demotion (YARN-5085). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5542) Scheduling of opportunistic containers
[ https://issues.apache.org/jira/browse/YARN-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011026#comment-17011026 ] Konstantinos Karanasos commented on YARN-5542: -- Hi [~brahmareddy], we could move the JIRAs that are still open to a new umbrella JIRA so that we can close this. Would that make sense? [~abmodi], are you actively working on any of these subtasks? > Scheduling of opportunistic containers > -- > > Key: YARN-5542 > URL: https://issues.apache.org/jira/browse/YARN-5542 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Konstantinos Karanasos >Priority: Major > > This JIRA groups all efforts related to the scheduling of opportunistic > containers. > It includes the scheduling of opportunistic container through the central RM > (YARN-5220), through distributed scheduling (YARN-2877), as well as the > scheduling of containers based on actual node utilization (YARN-1011) and the > container promotion/demotion (YARN-5085). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9737) Performance degradation, Distributed Opportunistic Scheduling
[ https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933869#comment-16933869 ] Konstantinos Karanasos commented on YARN-9737: -- Hi [~Babbleshack], just saw this. The performance of opportunistic containers depends on a lot of things -- here are a few to consider: * Note that distributed scheduling is most probably not what affects you here, but instead the use of opportunistic containers. So, I am pretty sure you would get the same results for centralized scheduling of opportunistic containers. * Opportunistic containers can be killed by guaranteed containers, therefore execution is much more sensitive as cluster utilization increases. So if your gridmix script ends up leading to high cluster utilization, you might end up getting excessive killing or queuing, hence the slowdown you observe. * Along the same lines, you might want to decrease the number of concurrent applications, especially if your utilization is high. Opportunistic containers do not count towards actually used resources for that matter, so if you are not careful you will end up launching too many jobs, therefore too many AMs, which use guaranteed containers and will be killing running opportunistic ones. * The fact that you allow at most two containers per node does not help either (and it is not a very common set-up in practice). It means that when a guaranteed container arrives at that node, at least 50% of the node's opportunistic containers will be killed (just by killing a single container). * You might also try to see what happens if you decrease the size of your queue to 0 or 1 (you will not avoid killing but you will avoid queuing). * How are you setting the percentage of opportunistic containers? I guess in the AM? Note that even when you set it to 100%, the AM will still be launched with a guaranteed container. Hope this helps. Then there are a lot of other things to tune for the scheduling of opportunistic containers, but I would start from the above list. cc: [~abmodi] that is currently working on opportunistic containers actively. > Performance degradation, Distributed Opportunistic Scheduling > - > > Key: YARN-9737 > URL: https://issues.apache.org/jira/browse/YARN-9737 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 > 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB > ssd. > 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB > ssd. > 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot > yarn-site: [^yarn-site.xml] > yarn-client-yarn-site: [^yarn-client.yarn-site.xml] > >Reporter: Babble Shack >Priority: Major > Labels: performance, scheduler, scheduling > Attachments: jct_cdf_100j_100t_1500.svg, > jct_cdf_100j_50t_1500_with_outliers.svg, jet_boxplot_j100_50t_1500.svg, > jet_boxplot_j100_50t_1500_with_outliers.svg, > task_throughput_boxplot_100j_50t_1500.svg, yarn-client.yarn-site.xml, > yarn-site.xml > > > Opportunistic scheduling is supposed to provide lower scheduling time, and > thus higher task throughput and lower job completion times for short > jobs/tasks. > Through my experiments I have found distributed scheduling can degrade > performance. > I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run > time was 1523ms. > Findings: > * Job completion time, the time take from submitting a job to job > completion, may degrade by over 200% > [^jct_cdf_100j_100t_1500.svg] > [^jct_cdf_100j_50t_1500_with_outliers.svg] > * Job execution time may increase by up to 300% > [^jet_boxplot_j100_50t_1500.svg] > [^jet_boxplot_j100_50t_1500_with_outliers.svg] > * Task throughput decreased by 100% > ^[^task_throughput_boxplot_100j_50t_1500.svg]^ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766325#comment-16766325 ] Konstantinos Karanasos commented on YARN-999: - Give a look at YARN-7934 – we had refactored some stuff in preemption for the federation code (for the glabal queues in particular). The umbrella Jira is not finished, but I think this Jira will point you to some useful classes. I am not sure how exactly the reduction of node resources is implemented, but for the opportunistic containers, you can kill stuff locally at the NMs. So if you need to free up resources due to resource reduction, you can go over the opportunistic containers running and kill the long-running ones). As far as I remember, the regular preemption code in the RM will not touch opportunistic containers. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Priority: Major > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be hired here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766325#comment-16766325 ] Konstantinos Karanasos edited comment on YARN-999 at 2/12/19 6:53 PM: -- Give a look at YARN-7934 – we had refactored some stuff in preemption for the federation code (for the glabal queues in particular). The umbrella Jira is not finished, but I think this Jira will point you to some useful classes. I am not sure how exactly the reduction of node resources is implemented, but for the opportunistic containers, you can kill stuff locally at the NMs. So if you need to free up resources due to resource reduction, you can go over the opportunistic containers running and kill the long-running ones. As far as I remember, the regular preemption code in the RM will not touch opportunistic containers. was (Author: kkaranasos): Give a look at YARN-7934 – we had refactored some stuff in preemption for the federation code (for the glabal queues in particular). The umbrella Jira is not finished, but I think this Jira will point you to some useful classes. I am not sure how exactly the reduction of node resources is implemented, but for the opportunistic containers, you can kill stuff locally at the NMs. So if you need to free up resources due to resource reduction, you can go over the opportunistic containers running and kill the long-running ones). As far as I remember, the regular preemption code in the RM will not touch opportunistic containers. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Priority: Major > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be hired here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766305#comment-16766305 ] Konstantinos Karanasos commented on YARN-999: - Hi [~elgoiri], if it is an opportunistic container, it will already be killed fast, so I think you don't need a distinction between guaranteed/opportunistic (you will do preemption only in the guaranteed after the timeout). I think [~asuresh] might have worked on something related to preemption recently, but not sure. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Priority: Major > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be hired here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5886) Dynamically prioritize execution of opportunistic containers (NM queue reordering)
[
https://issues.apache.org/jira/browse/YARN-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729564#comment-16729564
]
Konstantinos Karanasos commented on YARN-5886:
--
Hi [~abmodi], I am not currently working on this – feel free to take over.
> Dynamically prioritize execution of opportunistic containers (NM queue
> reordering)
> --
>
> Key: YARN-5886
> URL: https://issues.apache.org/jira/browse/YARN-5886
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Major
>
> Currently the {{ContainerScheduler}} in the NM picks the next queued
> opportunistic container to be executed in a FIFO manner. That is, we first
> execute containers that arrived first at the NM.
> This JIRA proposes to add pluggable queue reordering strategies at the NM
> that will dynamically determine which opportunistic container will be
> executed next.
> For example, we can choose to prioritize containers that belong to jobs which
> are closer to completion, or containers that are short-running (if such
> information is available).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[ https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695464#comment-16695464 ] Konstantinos Karanasos commented on YARN-8984: -- Looks good to me, thanks [~fly_in_gis]. > AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty > -- > > Key: YARN-8984 > URL: https://issues.apache.org/jira/browse/YARN-8984 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Critical > Attachments: YARN-8984-001.patch, YARN-8984-002.patch, > YARN-8984-003.patch, YARN-8984-004.patch, YARN-8984-005.patch > > > In AMRMClient, outstandingSchedRequests should be removed or decreased when > container allocated. However, it could not work when allocation tag is null > or empty. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[
https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680181#comment-16680181
]
Konstantinos Karanasos commented on YARN-8984:
--
{quote}So please keep the null check here. You can remove the isEmpty() check
if needed.
{quote}
+1.
> AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
> --
>
> Key: YARN-8984
> URL: https://issues.apache.org/jira/browse/YARN-8984
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Critical
> Attachments: YARN-8984-001.patch, YARN-8984-002.patch,
> YARN-8984-003.patch
>
>
> In AMRMClient, outstandingSchedRequests should be removed or decreased when
> container allocated. However, it could not work when allocation tag is null
> or empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[
https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678649#comment-16678649
]
Konstantinos Karanasos commented on YARN-8984:
--
If I remember correctly, the Scheduling Requests are used only in case we have
placement constraints (at least this was the initial design, not sure if things
have changed recently).
Given that, at successful container allocation, the Container Request of
unconstrained requests will be properly removed through a different code-path.
That said, if we want to start using Scheduling Request objects even without
constraints (I don't see a reason why do this urgently, but we can do it in the
long run), then I think we should fix the code. As [~cheersyang] said, I don't
think the current fix will work, since the {{schedReqs}} will be null when
there are no tags, and the map of {{outstandingSchedRequests}} has the
allocation tags as key.
Adding [~asuresh] (Arun, I think you worked on that code last, so let me know
if I am missing something) and [~leftnoteasy].
> AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
> --
>
> Key: YARN-8984
> URL: https://issues.apache.org/jira/browse/YARN-8984
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Critical
> Attachments: YARN-8984-001.patch, YARN-8984-002.patch,
> YARN-8984-003.patch
>
>
> In AMRMClient, outstandingSchedRequests should be removed or decreased when
> container allocated. However, it could not work when allocation tag is null
> or empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580321#comment-16580321 ] Konstantinos Karanasos commented on YARN-7953: -- Looks good to me, +1. Thanks [~abmodi] and [~botong]! > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, > YARN-7953-YARN-7402.v8.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574418#comment-16574418
]
Konstantinos Karanasos commented on YARN-7953:
--
Thanks [~abmodi] for working on this. A few comments on the latest patch:
* Can we improve a bit the {{propagate}} method of the {{FederationQueue}}?
The name is not very informative and also it does two things, that is,
propagate total capacity top-down, and propagate the rest of the metrics
bottom-up. Let's split that into two methods, one for bottom-up and one for
top-down. Then we can keep the propagate (call it propagateCapacities?) that
will call both.
* Some public methods of FederationQueue do not have comments, including
propagate, rollupIdealAssignedFromChildren, countQueues, etc. Same for the
FedQueueIterator class (it creates an iterator to go over all queues of a
FederationQueue hierarchy).
* [minor] In the first two classes of the patch: This class represent -> This
class represents
* Do we need all three json files for the tests?
> [GQ] Data structures for federation global queues calculations
> --
>
> Key: YARN-7953
> URL: https://issues.apache.org/jira/browse/YARN-7953
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-7953-YARN-7402.v1.patch,
> YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch,
> YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch,
> YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, YARN-7953.v1.patch
>
>
> This Jira tracks data structures and helper classes used by the core
> algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8511) When AM releases a container, RM removes allocation tags before it is released by NM
[ https://issues.apache.org/jira/browse/YARN-8511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544861#comment-16544861 ] Konstantinos Karanasos commented on YARN-8511: -- Good catch – patch looks good, +1. > When AM releases a container, RM removes allocation tags before it is > released by NM > > > Key: YARN-8511 > URL: https://issues.apache.org/jira/browse/YARN-8511 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.1.0 >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8511.001.patch, YARN-8511.002.patch, > YARN-8511.003.patch, YARN-8511.004.patch > > > User leverages PC with allocation tags to avoid port conflicts between apps, > we found sometimes they still get port conflicts. This is a similar issue > like YARN-4148. Because RM immediately removes allocation tags once > AM#allocate asks to release a container, however container on NM has some > delay until it actually gets killed and released the port. We should let RM > remove allocation tags AFTER NM confirms the containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8250) Create another implementation of ContainerScheduler to support NM overallocation
[ https://issues.apache.org/jira/browse/YARN-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510299#comment-16510299 ] Konstantinos Karanasos commented on YARN-8250: -- Just went through your comments. I am skeptical about adding a new implementation of the Container Scheduler, as [~asuresh] and [~leftnoteasy] also pointed out. It seems we can achieve the required behavior just with some new policies/functions rather than a new implementation. >From what I understand, the main points are (1) how to start guaranteed >containers (do we start them and then kill opportunistic, or do we kill >opportunistic and then start them) and (2) how/when to start opportunistic >containers (at regular intervals or when another container finishes). *How to start guaranteed:* For (1), just to make sure I understand: why can't we know the resources utilized by each opportunistic container? But even if we don't know, can't we just use the allocated resources of the container in our calculation? I understand that this might lead to more opportunistic containers killed than might be needed, but for sure it will be a good first version (and much better than what we have today). At the same time, if we are not very aggressive with starting opportunistic containers, it should not lead to too many containers killed. So my take is to have a first version that does the starting as it is today (first queue, then kill), even if that leads to more opportunistic containers killed. And be less aggressive in starting opportunistic for that matter. *How to start opportunistic:* Why not keep existing behavior and add the new one too? That is, we still let opportunistic start when there is a container finish event, but do so not with utilized but allocated resources. This will keep the current behavior. On top of that, if over-allocation is enabled, we can have a thread that every so often checks utilized resources and starts opportunistic containers on over-allocated resources. > Create another implementation of ContainerScheduler to support NM > overallocation > > > Key: YARN-8250 > URL: https://issues.apache.org/jira/browse/YARN-8250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8250-YARN-1011.00.patch, > YARN-8250-YARN-1011.01.patch, YARN-8250-YARN-1011.02.patch > > > YARN-6675 adds NM over-allocation support by modifying the existing > ContainerScheduler and providing a utilizationBased resource tracker. > However, the implementation adds a lot of complexity to ContainerScheduler, > and future tweak of over-allocation strategy based on how much containers > have been launched is even more complicated. > As such, this Jira proposes a new ContainerScheduler that always launch > guaranteed containers immediately and queues opportunistic containers. It > relies on a periodical check to launch opportunistic containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8394) Improve data locality documentation for Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509931#comment-16509931
]
Konstantinos Karanasos commented on YARN-8394:
--
+1, thanks [~cheersyang].
> Improve data locality documentation for Capacity Scheduler
> --
>
> Key: YARN-8394
> URL: https://issues.apache.org/jira/browse/YARN-8394
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8394.001.patch, YARN-8394.002.patch
>
>
> YARN-6344 introduces a new parameter
> {{yarn.scheduler.capacity.rack-locality-additional-delay}} in
> capacity-scheduler.xml, we need to add some documentation in
> {{CapacityScheduler.md}} accordingly.
> Moreover, we are seeing more and more clusters are separating storage and
> computation where file system is always remote, in such cases we need to
> introduce how to compromise data locality in CS otherwise MR jobs are
> suffering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8394) Improve data locality documentation for Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506651#comment-16506651
]
Konstantinos Karanasos commented on YARN-8394:
--
Looks good to me. A couple of things to fix before committing:
* that tries best efforts to honor task locality constraint -> to honor task
locality constraints
* losing the locality constraint -> relaxing the locality constraint
* when additional is -1, you can say that it is calculated based on the
formula L * C / N, capped by the cluster size, where L is number of locations
(nodes or racks) specified in the resource request, C is the number of
requested containers, and N is the size of the cluster.
> Improve data locality documentation for Capacity Scheduler
> --
>
> Key: YARN-8394
> URL: https://issues.apache.org/jira/browse/YARN-8394
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8394.001.patch
>
>
> YARN-6344 introduces a new parameter
> {{yarn.scheduler.capacity.rack-locality-additional-delay}} in
> capacity-scheduler.xml, we need to add some documentation in
> {{CapacityScheduler.md}} accordingly.
> Moreover, we are seeing more and more clusters are separating storage and
> computation where file system is always remote, in such cases we need to
> introduce how to compromise data locality in CS otherwise MR jobs are
> suffering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8346) Upgrading to 3.1 kills running containers with error "Opportunistic container queue is full"
[
https://issues.apache.org/jira/browse/YARN-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487803#comment-16487803
]
Konstantinos Karanasos commented on YARN-8346:
--
Thanks for the patch, [~jlowe].
Indeed you are right – the problem is the lack of execution type. The queue
size should remain 0 given that opportunistic containers are disabled in this
case.
+1 for the patch.
> Upgrading to 3.1 kills running containers with error "Opportunistic container
> queue is full"
>
>
> Key: YARN-8346
> URL: https://issues.apache.org/jira/browse/YARN-8346
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0, 3.0.2
>Reporter: Rohith Sharma K S
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: YARN-8346.001.patch
>
>
> It is observed while rolling upgrade from 2.8.4 to 3.1 release, all the
> running containers are killed and second attempt is launched for that
> application. The diagnostics message is "Opportunistic container queue is
> full" which is the reason for container killed.
> In NM log, I see below logs for after container is recovered.
> {noformat}
> 2018-05-23 17:18:50,655 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
> Opportunistic container [container_e06_1527075664705_0001_01_01] will
> not be queued at the NMsince max queue length [0] has been reached
> {noformat}
> Following steps are executed for rolling upgrade
> # Install 2.8.4 cluster and launch a MR job with distributed cache enabled.
> # Stop 2.8.4 RM. Start 3.1.0 RM with same configuration.
> # Stop 2.8.4 NM batch by batch. Start 3.1.0 NM batch by batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8113) Update placement constraints doc with application namespaces and inter-app constraints
[ https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-8113: - Summary: Update placement constraints doc with application namespaces and inter-app constraints (was: Update placement constraints doc with application namespace and inter-app constraints) > Update placement constraints doc with application namespaces and inter-app > constraints > -- > > Key: YARN-8113 > URL: https://issues.apache.org/jira/browse/YARN-8113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: AllocationTag_Namespace.png, DS_update.png, > YARN-8113.001.patch, YARN-8113.002.patch, YARN-8113.003.patch, > YARN-8113.004.patch > > > Once YARN-8013 is done, we will support all type of application namespace > types for inter-app constraints, accordingly we need to update the doc. Also > make sure API doc will be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8113) Update placement constraints doc with application namespace and inter-app constraints
[ https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459989#comment-16459989 ] Konstantinos Karanasos commented on YARN-8113: -- Hi [~cheersyang]. I did some changes and uploaded a new version of the patch. Give it a look. The most important changes I made: * Fixed a bit the PlacementSpec grammar, because it was assuming only two constraints within an AND/OR constraints, while I guess it is a list. * Explained how to specify tag namespaces in the placementspec -- please check it is correct. * Added examples of tag namespace usage. BTW the current grammar assumes no nesting of constraints (as in an AND is a child of an OR, etc.). Do we allow nesting in the latest version? If so, we have to update the grammar, because at the moment it assumes a flat hierarchy. > Update placement constraints doc with application namespace and inter-app > constraints > - > > Key: YARN-8113 > URL: https://issues.apache.org/jira/browse/YARN-8113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: AllocationTag_Namespace.png, DS_update.png, > YARN-8113.001.patch, YARN-8113.002.patch, YARN-8113.003.patch > > > Once YARN-8013 is done, we will support all type of application namespace > types for inter-app constraints, accordingly we need to update the doc. Also > make sure API doc will be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8113) Update placement constraints doc with application namespace and inter-app constraints
[ https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-8113: - Attachment: YARN-8113.003.patch > Update placement constraints doc with application namespace and inter-app > constraints > - > > Key: YARN-8113 > URL: https://issues.apache.org/jira/browse/YARN-8113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: AllocationTag_Namespace.png, DS_update.png, > YARN-8113.001.patch, YARN-8113.002.patch, YARN-8113.003.patch > > > Once YARN-8013 is done, we will support all type of application namespace > types for inter-app constraints, accordingly we need to update the doc. Also > make sure API doc will be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8113) Update placement constraints doc with application namespace and inter-app constraints
[
https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458952#comment-16458952
]
Konstantinos Karanasos commented on YARN-8113:
--
Thanks for working on this, [~cheersyang].
A couple of comments:
* I think you removed from the scheduler handler description the sentence "It
currently supports anti-affinity constraints (no affinity or cardinality)". I
thought we still do not have support inside the scheduler for that (only the
processor can do this). If this is the case, we should keep this sentence.
* Can we instead of "server" and "processor" in your AND example, use
something like storm, zk, spark? It took me a moment to understand that
processor and server are just tags.
* In the {{PlacementSpec}} description, we should add the syntax for AND and
OR.
* Minor: You can put the "Allocation tags namespace" paragraph above the
"Differences between node labels, attributes, and tags" para.
> Update placement constraints doc with application namespace and inter-app
> constraints
> -
>
> Key: YARN-8113
> URL: https://issues.apache.org/jira/browse/YARN-8113
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: AllocationTag_Namespace.png, DS_update.png,
> YARN-8113.001.patch
>
>
> Once YARN-8013 is done, we will support all type of application namespace
> types for inter-app constraints, accordingly we need to update the doc. Also
> make sure API doc will be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8195) Fix constraint cardinality check in the presence of multiple target allocation tags
[
https://issues.apache.org/jira/browse/YARN-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8195:
-
Summary: Fix constraint cardinality check in the presence of multiple
target allocation tags (was: Multiple target allocation tags in placement
constraint is not working as expected)
> Fix constraint cardinality check in the presence of multiple target
> allocation tags
> ---
>
> Key: YARN-8195
> URL: https://issues.apache.org/jira/browse/YARN-8195
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: YARN-8195.001.patch, YARN-8195.002.patch
>
>
> While testing PC with multiple allocation tags, I found the behavior is not
> working as expected. To demonstrate this issue, imagine we have following PC
> {code}
> X=1, notin,node, A, B
> {code}
> this is to ask for a placement on a node that has no A or B.
> now assume we have A, B tags on a node with following numbers of allocations,
> {code}
> A(0), B(1)
> {code}
> expectation, as B has 1 allocation on this node, this node should not satisfy
> the PC. However in actual, it passes the check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8195) Multiple target allocation tags in placement constraint is not working as expected
[
https://issues.apache.org/jira/browse/YARN-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458859#comment-16458859
]
Konstantinos Karanasos commented on YARN-8195:
--
+1, thanks [~cheersyang].
Test failure does not seem related. I will fix the checkstyle issues (it just
complains about a couple variable names) and commit it.
> Multiple target allocation tags in placement constraint is not working as
> expected
> --
>
> Key: YARN-8195
> URL: https://issues.apache.org/jira/browse/YARN-8195
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: YARN-8195.001.patch, YARN-8195.002.patch
>
>
> While testing PC with multiple allocation tags, I found the behavior is not
> working as expected. To demonstrate this issue, imagine we have following PC
> {code}
> X=1, notin,node, A, B
> {code}
> this is to ask for a placement on a node that has no A or B.
> now assume we have A, B tags on a node with following numbers of allocations,
> {code}
> A(0), B(1)
> {code}
> expectation, as B has 1 allocation on this node, this node should not satisfy
> the PC. However in actual, it passes the check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints API by removing allocationTagToIntraApp
[ https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446245#comment-16446245 ] Konstantinos Karanasos commented on YARN-8111: -- Just committed to trunk and branch-3.1. Thanks for working on this, [~cheersyang]. > Simplify PlacementConstraints API by removing allocationTagToIntraApp > - > > Key: YARN-8111 > URL: https://issues.apache.org/jira/browse/YARN-8111 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: YARN-8111.001.patch, YARN-8111.002.patch > > > # Per discussion in YARN-8013, we agree to disallow null value for namespace > and default it to SELF. > # Remove PlacementConstraints#allocationTagToIntraApp -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8178) [Umbrella] Resource Over-commitment Based on Opportunistic Container Preemption
[ https://issues.apache.org/jira/browse/YARN-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443268#comment-16443268 ] Konstantinos Karanasos commented on YARN-8178: -- This sounds interesting, [~Zian Chen]! I am curious to learn more details, especially about when exactly demotion will kick in and which containers will be chosen for demotion. > [Umbrella] Resource Over-commitment Based on Opportunistic Container > Preemption > --- > > Key: YARN-8178 > URL: https://issues.apache.org/jira/browse/YARN-8178 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacity scheduler >Reporter: Zian Chen >Priority: Major > > We want to provide an opportunistic container-based solution to achieve more > aggressive preemption with shorter preemption monitoring interval. > Instead of allowing applications to allocate resources with a mix of > guaranteed and opportunistic containers, we allow newly submitted > applications to only contain guaranteed containers. Meanwhile, we change the > preemption logic to, instead of killing containers, demote guaranteed > containers into opportunistic ones, so that when there are new applications > submitted, we can ensure that these containers can be launched by preempting > opportunistic containers. > This approach is related to YARN-1011 but achieves over-commitment in a > different way. However, we rely on opportunistic container part implemented > in YARN-1011 to make our design work well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints API by removing allocationTagToIntraApp
[
https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426200#comment-16426200
]
Konstantinos Karanasos commented on YARN-8111:
--
{quote}[~kkaranasos], the conflict should be caused by YARN-7142. Let me check
if it is possible to bring that to branch-3.1
{quote}
Thanks [~leftnoteasy] – let me know how it goes.
> Simplify PlacementConstraints API by removing allocationTagToIntraApp
> -
>
> Key: YARN-8111
> URL: https://issues.apache.org/jira/browse/YARN-8111
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: YARN-8111.001.patch, YARN-8111.002.patch
>
>
> # Per discussion in YARN-8013, we agree to disallow null value for namespace
> and default it to SELF.
> # Remove PlacementConstraints#allocationTagToIntraApp
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints API by removing allocationTagToIntraApp
[
https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425960#comment-16425960
]
Konstantinos Karanasos commented on YARN-8111:
--
[~cheersyang], this is also not applying to branch-3.1.
The {{Component}} class is complaining. I guess there is another patch that has
was committed only to trunk. Looping in [~leftnoteasy].
In general, we will keep committing all this to both trunk and branch-3.1,
right?
> Simplify PlacementConstraints API by removing allocationTagToIntraApp
> -
>
> Key: YARN-8111
> URL: https://issues.apache.org/jira/browse/YARN-8111
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: YARN-8111.001.patch, YARN-8111.002.patch
>
>
> # Per discussion in YARN-8013, we agree to disallow null value for namespace
> and default it to SELF.
> # Remove PlacementConstraints#allocationTagToIntraApp
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8111) Simplify PlacementConstraints API by removing allocationTagToIntraApp
[ https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-8111: - Summary: Simplify PlacementConstraints API by removing allocationTagToIntraApp (was: Simplify PlacementConstraints class API) > Simplify PlacementConstraints API by removing allocationTagToIntraApp > - > > Key: YARN-8111 > URL: https://issues.apache.org/jira/browse/YARN-8111 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: YARN-8111.001.patch, YARN-8111.002.patch > > > # Per discussion in YARN-8013, we agree to disallow null value for namespace > and default it to SELF. > # Remove PlacementConstraints#allocationTagToIntraApp -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8112) Fix min cardinality check for same source and target tags in intra-app constraints
[ https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos reassigned YARN-8112: Assignee: Konstantinos Karanasos > Fix min cardinality check for same source and target tags in intra-app > constraints > -- > > Key: YARN-8112 > URL: https://issues.apache.org/jira/browse/YARN-8112 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Konstantinos Karanasos >Priority: Major > > The min cardinality constraint (min cardinality = _k_) ensures that a > container is placed at a node that has already k occurrences of the target > tag. For example, a constraint _zk=3,CARDINALITY,NODE,hb,2,10_ will place > each of the three zk containers on a node with at least 2 hb instances (and > no more than 10 for the max cardinality). > Affinity constraints is a special case of this where min cardinality is 1. > Currently we do not support min cardinality when the source and the target of > the constraint are the same in an intra-app constraint. > Therefore, zk=3,CARDINALITY,NODE,zk,2,10 is not supported, and neither is > zk=3,IN,NODE,zk. > This Jira will address this problem by placing the first k containers on the > same node (or any other specified scope, e.g., rack), so that min cardinality > can be met when placing the subsequent containers with the same tag. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support application tags when defining application namespaces for placement constraints
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424801#comment-16424801
]
Konstantinos Karanasos commented on YARN-8013:
--
[~cheersyang], can you please post a patch for branch-3.1, too?
There are a few merge conflicts due to the renames and I want to make sure we
are doing introducing any errors.
> Support application tags when defining application namespaces for placement
> constraints
> ---
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds the concept of *Application Tags* to Yarn applications. In
> particular, a user is able to annotate application with multiple tags to
> classify apps. We can leverage this to define application namespaces based on
> application tags for placement constraints.
> Below is a typical use case.
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8013) Support application tags when defining application namespaces for placement constraints
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8013:
-
Description:
YARN-1461 adds the concept of *Application Tags* to Yarn applications. In
particular, a user is able to annotate application with multiple tags to
classify apps. We can leverage this to define application namespaces based on
application tags for placement constraints.
Below is a typical use case.
There are a lot of TF jobs running on Yarn, and some of them are consuming
resources heavily. So we want to limit number of PS on each node for such BIG
players but ignore those SMALL ones. To achieve this, we can do following steps:
# Add application tag "big-tf" to these big TF jobs
# For each PS request, we add "ps" source tag and map it to constraint
"{color:#d04437}notin, node, tensorflow/ps{color}" or
"{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
2{color}" for finer grained controls.
was:
YARN-1461 adds *Application Tag* concept to Yarn applications, user is able to
annotate application with multiple tags to classify apps. We can leverage this
to represent a namespace for a certain group of apps. So instead of calling
*app-label*, propose to call it *app-tag*.
A typical use case is,
There are a lot of TF jobs running on Yarn, and some of them are consuming
resources heavily. So we want to limit number of PS on each node for such BIG
players but ignore those SMALL ones. To achieve this, we can do following steps:
# Add application tag "big-tf" to these big TF jobs
# For each PS request, we add "ps" source tag and map it to constraint
"{color:#d04437}notin, node, tensorflow/ps{color}" or
"{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
2{color}" for finer grained controls.
> Support application tags when defining application namespaces for placement
> constraints
> ---
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds the concept of *Application Tags* to Yarn applications. In
> particular, a user is able to annotate application with multiple tags to
> classify apps. We can leverage this to define application namespaces based on
> application tags for placement constraints.
> Below is a typical use case.
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8013) Support application tags when defining application namespaces for placement constraints
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8013:
-
Summary: Support application tags when defining application namespaces for
placement constraints (was: Support application namespaces defined with
application tags for placement constraints)
> Support application tags when defining application namespaces for placement
> constraints
> ---
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8013) Support application namespaces defined with application tags for placement constraints
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8013:
-
Summary: Support application namespaces defined with application tags for
placement constraints (was: Support APP-TAG namespace for allocation tags)
> Support application namespaces defined with application tags for placement
> constraints
> --
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424766#comment-16424766
]
Konstantinos Karanasos commented on YARN-8013:
--
Sure, will commit this now.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8112) Fix min cardinality check for same source and target tags in intra-app constraints
[ https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424188#comment-16424188 ] Konstantinos Karanasos commented on YARN-8112: -- [~cheersyang], I updated the description so that we capture the more generic case. I can take over it, if that's ok with you. > Fix min cardinality check for same source and target tags in intra-app > constraints > -- > > Key: YARN-8112 > URL: https://issues.apache.org/jira/browse/YARN-8112 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Priority: Major > > The min cardinality constraint (min cardinality = _k_) ensures that a > container is placed at a node that has already k occurrences of the target > tag. For example, a constraint _zk=3,CARDINALITY,NODE,hb,2,10_ will place > each of the three zk containers on a node with at least 2 hb instances (and > no more than 10 for the max cardinality). > Affinity constraints is a special case of this where min cardinality is 1. > Currently we do not support min cardinality when the source and the target of > the constraint are the same in an intra-app constraint. > Therefore, zk=3,CARDINALITY,NODE,zk,2,10 is not supported, and neither is > zk=3,IN,NODE,zk. > This Jira will address this problem by placing the first k containers on the > same node (or any other specified scope, e.g., rack), so that min cardinality > can be met when placing the subsequent containers with the same tag. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8112) Fix min cardinality check for same source and target tags in intra-app constraints
[
https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8112:
-
Description:
The min cardinality constraint (min cardinality = _k_) ensures that a container
is placed at a node that has already k occurrences of the target tag. For
example, a constraint _zk=3,CARDINALITY,NODE,hb,2,10_ will place each of the
three zk containers on a node with at least 2 hb instances (and no more than 10
for the max cardinality).
Affinity constraints is a special case of this where min cardinality is 1.
Currently we do not support min cardinality when the source and the target of
the constraint are the same in an intra-app constraint.
Therefore, zk=3,CARDINALITY,NODE,zk,2,10 is not supported, and neither is
zk=3,IN,NODE,zk.
This Jira will address this problem by placing the first k containers on the
same node (or any other specified scope, e.g., rack), so that min cardinality
can be met when placing the subsequent containers with the same tag.
was:
The min cardinality constraint is
Currently following PC doesn't work
{code:java}
-placement_spec zk=3,IN,NODE,zk
-placement_spec zk=1,IN,NODE,zk
{code}
Expectation: place 3/1 zk containers in any node of the cluster.
This is because when we do cardinality check, PC cannot be satisfied as there
is no "zk" tag placed on any node. But this is a very common user-case, hence
we need to support.
> Fix min cardinality check for same source and target tags in intra-app
> constraints
> --
>
> Key: YARN-8112
> URL: https://issues.apache.org/jira/browse/YARN-8112
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Priority: Major
>
> The min cardinality constraint (min cardinality = _k_) ensures that a
> container is placed at a node that has already k occurrences of the target
> tag. For example, a constraint _zk=3,CARDINALITY,NODE,hb,2,10_ will place
> each of the three zk containers on a node with at least 2 hb instances (and
> no more than 10 for the max cardinality).
> Affinity constraints is a special case of this where min cardinality is 1.
> Currently we do not support min cardinality when the source and the target of
> the constraint are the same in an intra-app constraint.
> Therefore, zk=3,CARDINALITY,NODE,zk,2,10 is not supported, and neither is
> zk=3,IN,NODE,zk.
> This Jira will address this problem by placing the first k containers on the
> same node (or any other specified scope, e.g., rack), so that min cardinality
> can be met when placing the subsequent containers with the same tag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8112) Fix min cardinality check for same source and target tags in intra-app constraints
[
https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8112:
-
Description:
The min cardinality constraint is
Currently following PC doesn't work
{code:java}
-placement_spec zk=3,IN,NODE,zk
-placement_spec zk=1,IN,NODE,zk
{code}
Expectation: place 3/1 zk containers in any node of the cluster.
This is because when we do cardinality check, PC cannot be satisfied as there
is no "zk" tag placed on any node. But this is a very common user-case, hence
we need to support.
was:
Currently following PC doesn't work
{code:java}
-placement_spec zk=3,IN,NODE,zk
-placement_spec zk=1,IN,NODE,zk
{code}
Expectation: place 3/1 zk containers in any node of the cluster.
This is because when we do cardinality check, PC cannot be satisfied as there
is no "zk" tag placed on any node. But this is a very common user-case, hence
we need to support.
> Fix min cardinality check for same source and target tags in intra-app
> constraints
> --
>
> Key: YARN-8112
> URL: https://issues.apache.org/jira/browse/YARN-8112
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Priority: Major
>
> The min cardinality constraint is
>
> Currently following PC doesn't work
> {code:java}
> -placement_spec zk=3,IN,NODE,zk
> -placement_spec zk=1,IN,NODE,zk
> {code}
> Expectation: place 3/1 zk containers in any node of the cluster.
> This is because when we do cardinality check, PC cannot be satisfied as there
> is no "zk" tag placed on any node. But this is a very common user-case, hence
> we need to support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8112) Fix min cardinality check for same source and target tags in intra-app constraints
[
https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8112:
-
Summary: Fix min cardinality check for same source and target tags in
intra-app constraints (was: Fix min cardinality check when source==target tag
in intra-app constraints)
> Fix min cardinality check for same source and target tags in intra-app
> constraints
> --
>
> Key: YARN-8112
> URL: https://issues.apache.org/jira/browse/YARN-8112
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Priority: Major
>
> Currently following PC doesn't work
> {code:java}
> -placement_spec zk=3,IN,NODE,zk
> -placement_spec zk=1,IN,NODE,zk
> {code}
> Expectation: place 3/1 zk containers in any node of the cluster.
> This is because when we do cardinality check, PC cannot be satisfied as there
> is no "zk" tag placed on any node. But this is a very common user-case, hence
> we need to support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8112) Fix min cardinality check when source==target tag in intra-app constraints
[
https://issues.apache.org/jira/browse/YARN-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-8112:
-
Summary: Fix min cardinality check when source==target tag in intra-app
constraints (was: Relax cardinality check for intra-app affinity placement
constraints)
> Fix min cardinality check when source==target tag in intra-app constraints
> --
>
> Key: YARN-8112
> URL: https://issues.apache.org/jira/browse/YARN-8112
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Priority: Major
>
> Currently following PC doesn't work
> {code:java}
> -placement_spec zk=3,IN,NODE,zk
> -placement_spec zk=1,IN,NODE,zk
> {code}
> Expectation: place 3/1 zk containers in any node of the cluster.
> This is because when we do cardinality check, PC cannot be satisfied as there
> is no "zk" tag placed on any node. But this is a very common user-case, hence
> we need to support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8113) Update placement constraints doc with application namespace and inter-app constraints
[ https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-8113: - Summary: Update placement constraints doc with application namespace and inter-app constraints (was: Add doc about allocation tag namespace and inter-app constraints) > Update placement constraints doc with application namespace and inter-app > constraints > - > > Key: YARN-8113 > URL: https://issues.apache.org/jira/browse/YARN-8113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > Once YARN-8013 is done, we will support all type of namespace types for > inter-app constraints, accordingly we need to update the doc. Also make sure > API doc will be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8113) Update placement constraints doc with application namespace and inter-app constraints
[ https://issues.apache.org/jira/browse/YARN-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-8113: - Description: Once YARN-8013 is done, we will support all type of application namespace types for inter-app constraints, accordingly we need to update the doc. Also make sure API doc will be updated. (was: Once YARN-8013 is done, we will support all type of namespace types for inter-app constraints, accordingly we need to update the doc. Also make sure API doc will be updated.) > Update placement constraints doc with application namespace and inter-app > constraints > - > > Key: YARN-8113 > URL: https://issues.apache.org/jira/browse/YARN-8113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > Once YARN-8013 is done, we will support all type of application namespace > types for inter-app constraints, accordingly we need to update the doc. Also > make sure API doc will be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints class API
[ https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424180#comment-16424180 ] Konstantinos Karanasos commented on YARN-8111: -- Makes sense, will commit this today. Thanks [~cheersyang]. > Simplify PlacementConstraints class API > --- > > Key: YARN-8111 > URL: https://issues.apache.org/jira/browse/YARN-8111 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: YARN-8111.001.patch, YARN-8111.002.patch > > > # Per discussion in YARN-8013, we agree to disallow null value for namespace > and default it to SELF. > # Remove PlacementConstraints#allocationTagToIntraApp -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints class API
[ https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423587#comment-16423587 ] Konstantinos Karanasos commented on YARN-8111: -- I checked the code again, yes, makes sense. BTW you can call the allocationTagWithNamespace inside the allocationTag, so that all allocation tag creations end up in the same method. I think we also need to update the documentation (if I remember well the allocationTagToIntraApp is mentioned there). > Simplify PlacementConstraints class API > --- > > Key: YARN-8111 > URL: https://issues.apache.org/jira/browse/YARN-8111 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: YARN-8111.001.patch > > > # Per discussion in YARN-8013, we agree to disallow null value for namespace > and default it to SELF. > # Remove PlacementConstraints#allocationTagToIntraApp -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423577#comment-16423577
]
Konstantinos Karanasos commented on YARN-8013:
--
+1, thanks [~cheersyang].
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8111) Simplify PlacementConstraints class API
[ https://issues.apache.org/jira/browse/YARN-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423575#comment-16423575 ] Konstantinos Karanasos commented on YARN-8111: -- Thanks for the patch, [~cheersyang]. Looks good. I suggest to keep the allocationTagToIntraApp so that we can later easily change the default behavior if we want. This way an app can also make sure it really requests intra-app constraints. > Simplify PlacementConstraints class API > --- > > Key: YARN-8111 > URL: https://issues.apache.org/jira/browse/YARN-8111 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: YARN-8111.001.patch > > > # Per discussion in YARN-8013, we agree to disallow null value for namespace > and default it to SELF. > # Remove PlacementConstraints#allocationTagToIntraApp -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423394#comment-16423394
]
Konstantinos Karanasos commented on YARN-8013:
--
+1, thanks [~cheersyang].
(There seem to be a couple of checkstyle issues and a failing test, but
otherwise +1 from me).
BTW, since you are on this part of the code, I was checking some things today
and found that we allow the application namespace to be null. Then once we
check it, we substitute it with Self as the default in case of null. I think we
should instead make it Self by default in the PlacementConstraints and never
let it be null. This way also simplify the checking code. I just mention it
here in case you are at it -- of course it is not part of this JIRA.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422820#comment-16422820
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks for the new patch, [~cheersyang].
I think it is cleaner that you call the evaluate inside the tags manager in
this version.
{quote}I was hoping the java doc explains the bit. Any suggested name?
{quote}
Javadoc is totally fine. I would call the class
{{TargetApplicationsNamespace}}. How does this sound?
{quote}I can even remove this class by letting namespace to be eval'd by two
args 1) current appId 2) all appIds.
{quote}
Now I see what this class does, yes. It is small enough to be folded into the
evaluate to be honest, and I don't think the functionality of this class will
be extended. Makes sense?
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421040#comment-16421040
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks again for working on this, [~cheersyang].
Patch looks mostly fine, but from a quick look it feels that the
AllocationTagsManager exposes a bit too much information at the moment. So we
have the {{getApplicationIdToTags}} method that returns all existing
applications and their tags (which can be too many). Then we create
{{TargetApplications}} with all this information, although I think the nsScope
is given, right?
Why not have a method inside the tags manager that takes an nsScope and returns
only the target applications for that nsScope, rather than exposing all this?
It seems that this will simplify the code of {{TargetApplications}} too.
Other than that, a couple of minor things:
* AllocationTagNamespace is a slightly misleading name, as it does not reveal
that it essentially defines the applications to which a constraint will refer
* When defining {{getApplicationIdToTags}}:
perAppNodeMappings.keySet().contains() -> perAppNodeMappings.containsKey()
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418099#comment-16418099
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks [~cheersyang] for working on this and [~leftnoteasy] for the review.
Please give me a day to check this too.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8034) Clarification on preferredHost request with relaxedLocality
[ https://issues.apache.org/jira/browse/YARN-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403185#comment-16403185 ] Konstantinos Karanasos edited comment on YARN-8034 at 3/17/18 1:22 AM: --- Hi [~jagadish1...@gmail.com], As [~jlowe] mentioned, this is very related to YARN-6344 for the capacity scheduler. What you should look at is the "yarn.scheduler.capacity.rack-locality-additional-delay" parameter. Since you have only one (or very few) container requests, the current logic (if you let the above parameter to its default value) will lead to relaxing locality almost immediately. If you set that parameter to a positive value, you should achieve your desired behavior. was (Author: kkaranasos): Hi [~jagadish1...@gmail.com], As [~jlowe] mentioned, this is very related to YARN-6344 for the capacity scheduler. What you should look at is the "yarn.scheduler.capacity.rack-locality-additional-delay" parameter. Since you have only one (or very few) container requests, the current logic (if you let the above parameter to its default value) value will lead to relaxing locality almost immediately. If you set that parameter to a positive value, you should achieve your desired behavior. > Clarification on preferredHost request with relaxedLocality > --- > > Key: YARN-8034 > URL: https://issues.apache.org/jira/browse/YARN-8034 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jagadish >Priority: Major > > I work on Apache Samza, a stateful stream-processing framework that leverages > Yarn for resource management. The Samza AM requests resources on specific > hosts to schedule stateful jobs. We set relaxLocality = true in these > requests we make to Yarn. Often we have observed that we don't get containers > on the hosts that we requested them on and the Yarn RM returns containers on > arbitrary hosts. > Do you know what the behavior of the FairScheduler/CapacityScheduler is when > setting "relaxLocality = true".I did play around by setting a high value for > yarn.scheduler.capacity.node-locality-delay but it did not seem to matter. > However, when setting relaxLocality = false, we get resources on the exact > hosts we requested on. > The behavior I want from Yarn is "Honor locality to the best possible extent > and only return a container on an arbitrary host if the requested host is > down". Is there a way to accomplish this? > If you can point me to the Scheduler code, I'm happy to look at it as well. > For context, we have continuous scheduling enabled in our clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8034) Clarification on preferredHost request with relaxedLocality
[ https://issues.apache.org/jira/browse/YARN-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403185#comment-16403185 ] Konstantinos Karanasos commented on YARN-8034: -- Hi [~jagadish1...@gmail.com], As [~jlowe] mentioned, this is very related to YARN-6344 for the capacity scheduler. What you should look at is the "yarn.scheduler.capacity.rack-locality-additional-delay" parameter. Since you have only one (or very few) container requests, the current logic (if you let the above parameter to its default value) value will lead to relaxing locality almost immediately. If you set that parameter to a positive value, you should achieve your desired behavior. > Clarification on preferredHost request with relaxedLocality > --- > > Key: YARN-8034 > URL: https://issues.apache.org/jira/browse/YARN-8034 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jagadish >Priority: Major > > I work on Apache Samza, a stateful stream-processing framework that leverages > Yarn for resource management. The Samza AM requests resources on specific > hosts to schedule stateful jobs. We set relaxLocality = true in these > requests we make to Yarn. Often we have observed that we don't get containers > on the hosts that we requested them on and the Yarn RM returns containers on > arbitrary hosts. > Do you know what the behavior of the FairScheduler/CapacityScheduler is when > setting "relaxLocality = true".I did play around by setting a high value for > yarn.scheduler.capacity.node-locality-delay but it did not seem to matter. > However, when setting relaxLocality = false, we get resources on the exact > hosts we requested on. > The behavior I want from Yarn is "Honor locality to the best possible extent > and only return a container on an arbitrary host if the requested host is > down". Is there a way to accomplish this? > If you can point me to the Scheduler code, I'm happy to look at it as well. > For context, we have continuous scheduling enabled in our clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7921) Transform a PlacementConstraint to a string expression
[
https://issues.apache.org/jira/browse/YARN-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-7921:
-
Fix Version/s: 3.1.0
> Transform a PlacementConstraint to a string expression
> --
>
> Key: YARN-7921
> URL: https://issues.apache.org/jira/browse/YARN-7921
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: Placement Constraint Expression Syntax
> Specification.pdf, YARN-7921.001.patch, YARN-7921.002.patch
>
>
> Purpose:
> Let placement constraint viewable on UI or log, e.g print app placement
> constraint in RM app page. Help user to use constraints and analysis
> placement issues easier.
> Propose:
> Like what was added for DS, toString is a reversed process of
> {{PlacementConstraintParser}} that transforms a PlacementConstraint to a
> string, using same syntax. E.g
> {code}
> AbstractConstraint constraintExpr = targetIn(NODE, allocationTag("hbase-m"));
> constraint.toString();
> // This prints: IN,NODE,hbase-m
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7921) Transform a PlacementConstraint to a string expression
[
https://issues.apache.org/jira/browse/YARN-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375853#comment-16375853
]
Konstantinos Karanasos commented on YARN-7921:
--
{quote}from the latest discussion in YARN-3409, we plan to use DNS format for
namespace
{quote}
Okay, we are good on that front then.
{quote}we are using ":" to separate multiple child constraint in a AND or OR
constraint
{quote}
Yes, I meant using ";" there too.
I find ";" more intuitive for delimiting, but we can do it in a separate Jira
later if you prefer.
I will commit this Monday morning.
> Transform a PlacementConstraint to a string expression
> --
>
> Key: YARN-7921
> URL: https://issues.apache.org/jira/browse/YARN-7921
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Placement Constraint Expression Syntax
> Specification.pdf, YARN-7921.001.patch, YARN-7921.002.patch
>
>
> Purpose:
> Let placement constraint viewable on UI or log, e.g print app placement
> constraint in RM app page. Help user to use constraints and analysis
> placement issues easier.
> Propose:
> Like what was added for DS, toString is a reversed process of
> {{PlacementConstraintParser}} that transforms a PlacementConstraint to a
> string, using same syntax. E.g
> {code}
> AbstractConstraint constraintExpr = targetIn(NODE, allocationTag("hbase-m"));
> constraint.toString();
> // This prints: IN,NODE,hbase-m
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7921) Transform a PlacementConstraint to a string expression
[
https://issues.apache.org/jira/browse/YARN-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375780#comment-16375780
]
Konstantinos Karanasos commented on YARN-7921:
--
Thanks [~cheersyang]. Patch looks good.
Specification is the same as we have in the documentation, right?
Only thing I was wondering is whether we want to use ";" instead of ":" as a
delimiter for constraints. Maybe we should keep ":" for the namespaces for node
attributes.
What do you think?
> Transform a PlacementConstraint to a string expression
> --
>
> Key: YARN-7921
> URL: https://issues.apache.org/jira/browse/YARN-7921
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Placement Constraint Expression Syntax
> Specification.pdf, YARN-7921.001.patch, YARN-7921.002.patch
>
>
> Purpose:
> Let placement constraint viewable on UI or log, e.g print app placement
> constraint in RM app page. Help user to use constraints and analysis
> placement issues easier.
> Propose:
> Like what was added for DS, toString is a reversed process of
> {{PlacementConstraintParser}} that transforms a PlacementConstraint to a
> string, using same syntax. E.g
> {code}
> AbstractConstraint constraintExpr = targetIn(NODE, allocationTag("hbase-m"));
> constraint.toString();
> // This prints: IN,NODE,hbase-m
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373541#comment-16373541
]
Konstantinos Karanasos commented on YARN-7953:
--
Thanks for working on this and for splitting the patches, [~curino].
The data structures make sense. Some comments below:
In {{FedQueue}}:
* Should idealalloc be part of the FedQueue? Maybe targetAlloc? But still not
sure if it should be part of this class.
* Should we abstract out the creation of the JAXB context, the toString, and
the reading from JSON in a Util class, given we are using it in various
classes? JAXB context and toString are currently in FedQueue and
FederationGlobalView, the reading is in the test classes. The loadFile method
from the test classes can be put in there too.
* Split the propagate to two methods, one for the top-down traversal and one
for the bottom-up.
* Not clear what mergeQueues does. Does it take multiple subcluster queues and
combines them in a global queue?
* Is recursiveSetOfTotCap used? I guess it could also be used in the propagate
(especially after we break it)?
In {{FederationGlobalView}}:
* Does this class need a resource calculator?
* Shouldn't we have a method that builds the global view inside this class?
Nits (all but the first are in FedQueue):
* Is the org.ojalgo dependency needed in the pom?
* Better javadoc for class (explain GPG and global policies a bit)
* FedQueue -> FederatedQueue or FederationQueue
* scope -> subcluster?
* getChildrenByName -> getChildByName
> [GQ] Data structures for federation global queues calculations
> --
>
> Key: YARN-7953
> URL: https://issues.apache.org/jira/browse/YARN-7953
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Major
> Attachments: YARN-7953.v1.patch
>
>
> This Jira tracks data structures and helper classes used by the core
> algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7403) [GQ] Compute global and local "IdealAllocation"
[ https://issues.apache.org/jira/browse/YARN-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370912#comment-16370912 ] Konstantinos Karanasos commented on YARN-7403: -- Thanks for the patch, [~curino]. I would suggest to split it in two parts to make it easier to follow. The first can be the data structures to model the federated cluster and the second the algos for rebalancing. Another way would be to add the data structures and the abstract rebalancer, and then add the various implementations as the second jira. > [GQ] Compute global and local "IdealAllocation" > --- > > Key: YARN-7403 > URL: https://issues.apache.org/jira/browse/YARN-7403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: YARN-7403.draft.patch, YARN-7403.draft2.patch, > YARN-7403.draft3.patch, YARN-7403.v1.patch, YARN-7403.v2.patch, > global-queues-preemption.PNG > > > This JIRA tracks algorithmic effort to combine the local queue views of > capacity guarantee/use/demand and compute the global ideal allocation, and > the respective local allocations. This will inform the RMs in each > sub-clusters on how to allocate more containers to each queues (allowing for > temporary over/under allocations that are locally excessive, but globally > correct). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7920) Simplify configuration for PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7920: - Summary: Simplify configuration for PlacementConstraints (was: Cleanup configuration of PlacementConstraints) > Simplify configuration for PlacementConstraints > --- > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch, YARN-7920.005.patch, > YARN-7920.006.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366168#comment-16366168 ] Konstantinos Karanasos commented on YARN-7920: -- Patch looks good, thanks [~leftnoteasy]. There are a few checkstyle issues left. Anyway, I will fix them and commit in a bit, because I will not be available to commit this tomorrow. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch, YARN-7920.005.patch, > YARN-7920.006.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364783#comment-16364783 ] Konstantinos Karanasos commented on YARN-7920: -- Re: configuration, I think it is simpler at the moment to leave it as is (i.e., placement-processor/scheduler). I chatted offline with Arun and he is fine with is as well. Re: the .md.vm format, I remember [~cheersyang] used this format for the opportunistic containers doc that I had written, so I thought it is preferable. Weiwei, any advantages in this format? Re: the naming, I think all is fine now apart from the NonePlacementProcessor that sounds weird. Looking at it better, indeed Default is not okay (initially I thought it is a no-op, but in fact it rejects scheduling requests). Any of the following is fine with me: DisabledConstraintProcessor, UnsupportedConstraintProcessor, RejectConstraintProcessor? >From a quick look, documentation looks fine. I can make a pass over it before >I commit the patch. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363348#comment-16363348
]
Konstantinos Karanasos commented on YARN-7920:
--
Hi [~leftnoteasy],
{quote}I would still prefer to "scheduler", otherwise it will be a duplicated
config to yarn.resourcemanager.scheduler, and once FS want to support the
feature, we need to add a new option and document, etc.
{quote}
Sure, makes sense.
Re: the patch, I will check in more detail the implementation, but a first few
comments about the naming:
* The naming external processor is a bit redundant and not very descriptive.
Let's call it {{PlacementConstraintProcessor}}, since this is what it does.
* Similarly, in the comments of YarnConfiguration, "external which sits
outside of the scheduler" is not very helpful about why this should be used.
Let's say "Handle placement constraints by processor that is agnostic of the
scheduler implementation".
* Also, shall we call the {{NoneProcessor}} -> {{DefaultProcessor}} or
something along these lines?
* At some places you use the term "placement requests". Maybe say scheduling
requests?
Also, I agree with [~sunilg] to update the doc in the same Jira, it should be
very few changes.
I would also like to hear from [~asuresh], since he added the processor.
> Cleanup configuration of PlacementConstraints
> -
>
> Key: YARN-7920
> URL: https://issues.apache.org/jira/browse/YARN-7920
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7920.001.patch, YARN-7920.002.patch
>
>
> Currently it is very confusing to have the two configs in two different files
> (yarn-site.xml and capacity-scheduler.xml).
>
> Maybe a better approach is: we can delete the scheduling-request.allowed in
> CS, and update placement-constraints configs in yarn-site.xml a bit:
>
> - Remove placement-constraints.enabled, and add a new
> placement-constraints.handler, by default is none, and other acceptable
> values are a. external-processor (since algorithm is too generic to me), b.
> scheduler.
> - And add a new PlacementProcessor just to pass SchedulingRequest to
> scheduler without any modifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7921) Transform a PlacementConstraint to a string expression
[
https://issues.apache.org/jira/browse/YARN-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363329#comment-16363329
]
Konstantinos Karanasos commented on YARN-7921:
--
Hi [~cheersyang],
.bq what I was trying to do with this task is to implement 2). Which will be
done by implementing {{AbstractConstraint#toString}} methods.
Agreed. I was thinking that we could instead create a visitor, so that we can
have different toString representations and we can also take advantage of the
transformations more easily (for example you can first call a transformation
and then do the string representation). But I guess we can start by just
overriding the toString method.
> Transform a PlacementConstraint to a string expression
> --
>
> Key: YARN-7921
> URL: https://issues.apache.org/jira/browse/YARN-7921
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> Purpose:
> Let placement constraint viewable on UI or log, e.g print app placement
> constraint in RM app page. Help user to use constraints and analysis
> placement issues easier.
> Propose:
> Like what was added for DS, toString is a reversed process of
> {{PlacementConstraintParser}} that transforms a PlacementConstraint to a
> string, using same syntax. E.g
> {code}
> AbstractConstraint constraintExpr = targetIn(NODE, allocationTag("hbase-m"));
> constraint.toString();
> // This prints: IN,NODE,hbase-m
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360261#comment-16360261 ] Konstantinos Karanasos commented on YARN-7920: -- Hi [~leftnoteasy], +1 for simplifying the configuration. I think it is a good idea to use just one conf. I would call the scheduler choice "capacity-scheduler", given that it would not work for the fair scheduler. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7921) Transform a PlacementConstraint to a string expression
[
https://issues.apache.org/jira/browse/YARN-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360255#comment-16360255
]
Konstantinos Karanasos commented on YARN-7921:
--
Hi [~cheersyang], I think this will be certainly useful.
We can indeed use the same syntax we described in the documentation. Also, you
just need to support SingleConstraint and then the combined constraints. You
don't need a special case for the Cardinality and Target constraints, as you
can use the transformer to transform those to SingleConstraints.
Also, let's try to use the visitor pattern that we had created for the
transformers too.
> Transform a PlacementConstraint to a string expression
> --
>
> Key: YARN-7921
> URL: https://issues.apache.org/jira/browse/YARN-7921
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> Purpose:
> Let placement constraint viewable on UI or log, e.g print app placement
> constraint in RM app page. Help user to use constraints and analysis
> placement issues easier.
> Propose:
> Like what was added for DS, toString is a reversed process of
> {{PlacementConstraintParser}} that transforms a PlacementConstraint to a
> string, using same syntax. E.g
> {code}
> AbstractConstraint constraintExpr = targetIn(NODE, allocationTag("hbase-m"));
> constraint.toString();
> // This prints: IN,NODE,hbase-m
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7886) [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation
Konstantinos Karanasos created YARN-7886: Summary: [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation Key: YARN-7886 URL: https://issues.apache.org/jira/browse/YARN-7886 Project: Hadoop YARN Issue Type: Sub-task Reporter: Konstantinos Karanasos Assignee: Konstantinos Karanasos Given a federated cluster, this Jira will enable us to compare the allocation achieved by our rebalancing algorithms, when compared to the allocation that the Capacity Scheduler would achieve if it were operating over a single big cluster having the same total resources as the federated cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7885) [GQ] Generator for queue hierarchies over federated clusters
Konstantinos Karanasos created YARN-7885: Summary: [GQ] Generator for queue hierarchies over federated clusters Key: YARN-7885 URL: https://issues.apache.org/jira/browse/YARN-7885 Project: Hadoop YARN Issue Type: Sub-task Reporter: Konstantinos Karanasos Assignee: Konstantinos Karanasos This Jira will focus on generating random queue hierarchies with different total/used/pending resources across the sub-clusters of a federated cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7778) Merging of placement constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7778: - Summary: Merging of placement constraints defined at different levels (was: Merging of constraints defined at different levels) > Merging of placement constraints defined at different levels > > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349924#comment-16349924 ] Konstantinos Karanasos commented on YARN-7778: -- Thanks [~cheersyang], will commit it tomorrow morning. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349178#comment-16349178 ] Konstantinos Karanasos commented on YARN-7778: -- +1 on latest patch, thanks [~cheersyang]. I will do a minor fix to say that the "constraint" is coming from the SchedulingRequest when I commit the patch, if you don't mind. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347514#comment-16347514 ] Konstantinos Karanasos commented on YARN-7778: -- Thanks for the patch, [~cheersyang]. A couple of comments: * Let's rename the getResourceConstraint() to something like getMultilevelConstraint? Also the initial part of the comment "Retrieve the placement constraint of a given scheduling request from the PCM point of view" could be made more like "Consider all levels of constraints (resource request, app, cluster) and return a merged constraint". This is because we can also return a merged constraint of app and cluster level, even if there are no resource request constraints. * I would also change the signature of the method to add directly a Constraint coming from the resource request and a set of tags, rather than pass the SchedulingRequest object. This way we make clear that the set of tags is separate from the specific constraint. * Do we need the "Remove all null or duplicate constraints"? I was thinking that this should belong to a separate step that will do constraint simplification/minimization. If we keep it, I would suggest to put it in a separate method in the PCM, so that we can use it elsewhere too or extend it in the future easily. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7863) Modify placement constraints to support node attributes
[ https://issues.apache.org/jira/browse/YARN-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347372#comment-16347372 ] Konstantinos Karanasos commented on YARN-7863: -- Ideally there should be no API changes for this, right? Since we already allow node attributes in the target. I agree with [~asuresh] that there are two aspects to this (support node attributes in the target and in the scope), and we should tackle them in separate JIRAs. So we can keep this one for the target only. For the scope case, I was thinking whether we should only support a subset of the node attributes or whether it is no extra effort to support any attribute. But we can continue the discussions about this in YARN-7858. > Modify placement constraints to support node attributes > --- > > Key: YARN-7863 > URL: https://issues.apache.org/jira/browse/YARN-7863 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > This Jira will track to *Modify existing placement constraints to support > node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345599#comment-16345599 ] Konstantinos Karanasos commented on YARN-7780: -- Thanks [~asuresh], [~cheersyang], [~sunilg], [~leftnoteasy]. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Fix For: YARN-6592 > > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch, YARN-7780-YARN-6592.003.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344470#comment-16344470 ] Konstantinos Karanasos commented on YARN-7780: -- Attaching new version addressing comments. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch, YARN-7780-YARN-6592.003.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7780: - Attachment: YARN-7780-YARN-6592.003.patch > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch, YARN-7780-YARN-6592.003.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344154#comment-16344154 ] Konstantinos Karanasos commented on YARN-7778: -- Hi [~cheersyang], just checked your proposal. I like the idea of taking advantage of the AND constraints, instead of actually merging the constraints. So +1 to your approach. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343904#comment-16343904 ] Konstantinos Karanasos commented on YARN-7780: -- Thanks for the comments, [~sunilg]. I will address them in the next version of the patch. Re: SchedulingRequest, I guess we can keep it given it is an API change as Arun says. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7822) Constraint satisfaction checker support for composite OR and AND constraints
[
https://issues.apache.org/jira/browse/YARN-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343897#comment-16343897
]
Konstantinos Karanasos commented on YARN-7822:
--
Patch looks good, thanks [~cheersyang].
I have only one ask, which I don't think is a big change. Can we support DNF?
This is just ORs of ANDs. This means that we can allow the following:
* AND constraints with no nesting (your patch does that already);
* OR constraints, each being either a simple constraint or an AND constraint
with no nesting. So essentially, you only have to allow this additional case in
your patch.
If we do this, then we can transform any compound constraint with any nesting
level to an equivalent DNF form (we can tackle this in a separate JIRA), and we
will be done with the canSatisfyConstraint for all possible constraints.
Regarding the max cardinality, you are right it is a bit confusing in the
design doc, but it is not a code implementation artifact.
The idea is that the constraint has to be satisfied the moment of scheduling:
* When the source and target tags are different, this happens to be the same
both before and after the placement. Assume a constraint that hb-m should be
placed at a node with at most 2 hb-rs. Say node n1 has before scheduling 2
hb-rs, so we can place hb-m there. After placement, it will still have 2 hb-rs.
* Now when the source and target tags are the same, it is more complicated.
Assume a constraint that hb-rs should be placed at a node with at most 2 hb-rs.
Say node n1 has again 2 hb-rs before scheduling. After scheduling, it will have
3.
So to unify the two cases above, we say that the constraints should be
satisfied before the placement of the new container happens. As [~asuresh]
mentioned, at some point we tried to create a special case for when
source==target, but it the semantics were not straightforward (e.g., if a user
was saying max cardinality 2, they were expecting 2 after placement; when a
user says max cardinality 0, they think anti-affinity, but it actually has to
be 1 if we consider cardinality after placement). To unify things, we opted for
cardinality check before placement in all cases.
> Constraint satisfaction checker support for composite OR and AND constraints
>
>
> Key: YARN-7822
> URL: https://issues.apache.org/jira/browse/YARN-7822
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7822-YARN-6592.001.patch,
> YARN-7822-YARN-6592.002.patch, YARN-7822-YARN-6592.003.patch,
> YARN-7822-YARN-6592.004.patch
>
>
> JIRA to track changes to {{PlacementConstraintsUtil#canSatisfyConstraints}}
> handle OR and AND Composite constaints
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7839) Check node capacity before placing in the Algorithm
[
https://issues.apache.org/jira/browse/YARN-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343850#comment-16343850
]
Konstantinos Karanasos commented on YARN-7839:
--
I think the change makes sense, [~asuresh].
However, what about the case that a node seems full but a container is about to
finish (and will be finished until the allocate is done)? Should we completely
reject such nodes, or simply give higher priority to nodes that already have
available resources?
{quote}getPreferredNodeIterator(CandidateNodeSet candidateNodeSet)
{quote}
[~cheersyang], despite the naming, as far as I know, the candidateNodeSet is
currently always only a single node...
> Check node capacity before placing in the Algorithm
> ---
>
> Key: YARN-7839
> URL: https://issues.apache.org/jira/browse/YARN-7839
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Priority: Major
>
> Currently, the Algorithm assigns a node to a requests purely based on if the
> constraints are met. It is later in the scheduling phase that the Queue
> capacity and Node capacity are checked. If the request cannot be placed
> because of unavailable Queue/Node capacity, the request is retried by the
> Algorithm.
> For clusters that are running at high utilization, we can reduce the retries
> if we perform the Node capacity check in the Algorithm as well. The Queue
> capacity check and the other user limit checks can still be handled by the
> scheduler (since queues and other limits are tied to the scheduler, and not
> scheduler agnostic)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[
https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341902#comment-16341902
]
Konstantinos Karanasos commented on YARN-7780:
--
Thanks for the comments, [~asuresh].
{quote}line 34: Think we should remove this - allocation == since container at
the moment.
{quote}
That's true, but then all over the code we use the notion allocation now
(including allocation tags). I can mention that currently allocation=container
though.
{quote}And lets move this line to the bottom - since it is specific to the
processor.
{quote}
Doesn't the allocator in the Capacity Scheduler only do hard constraints too?
I will apply the fixes Monday morning (don't have a reliable internet
connection at the moment) so that we can commit this.
> Documentation for Placement Constraints
> ---
>
> Key: YARN-7780
> URL: https://issues.apache.org/jira/browse/YARN-7780
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
>Priority: Major
> Attachments: YARN-7780-YARN-6592.001.patch,
> YARN-7780-YARN-6592.002.patch
>
>
> JIRA to track documentation for the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7780: - Attachment: (was: YARN-7780-YARN-6592.003.patch) > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7780: - Attachment: YARN-7780-YARN-6592.003.patch > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch, YARN-7780-YARN-6592.003.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341883#comment-16341883 ] Konstantinos Karanasos commented on YARN-7780: -- Thanks for checking the doc, [~jianhe]. Indeed in this first version I purposely added only the minimum parameters required for someone to experiment with the constraint placement. As we discussed with [~asuresh] and [~leftnoteasy], we are planning to do some refactoring/cleanup of the parameters after the merge, so if it is okay, I would prefer to add the rest of the parameters once this cleanup is done. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7822) Constraint satisfaction checker support for composite OR and AND constraints
[
https://issues.apache.org/jira/browse/YARN-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341147#comment-16341147
]
Konstantinos Karanasos commented on YARN-7822:
--
Hi [~cheersyang], so maxCardinality does a check of <= at the state of the
cluster before the allocation that is currently being attempted. In your case,
you have maxCard=3, so n1 is a valid node as it has 3. Indeed after the
allocation it will end up having 4, so you would have to make your maxCard=2 to
have at most 3, since we are looking at the pre-allocation state. I had
mentioned this case in a comment somewhere.
> Constraint satisfaction checker support for composite OR and AND constraints
>
>
> Key: YARN-7822
> URL: https://issues.apache.org/jira/browse/YARN-7822
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7822-YARN-6592.001.patch
>
>
> JIRA to track changes to {{PlacementConstraintsUtil#canSatisfyConstraints}}
> handle OR and AND Composite constaints
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340453#comment-16340453 ] Konstantinos Karanasos commented on YARN-7780: -- Thanks for the review, [~cheersyang]. I addressed your comments and uploaded new version of the patch. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7780: - Attachment: YARN-7780-YARN-6592.002.patch > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch, > YARN-7780-YARN-6592.002.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7780: - Attachment: YARN-7780-YARN-6592.001.patch > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > Attachments: YARN-7780-YARN-6592.001.patch > > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7788) Factor out management of temp tags from AllocationTagsManager
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7788: - Description: Instead of using the AllocationTagsManager to store the temp tags that get generated when placing containers with constraints, we will use a LocalAllocationtagsManager. (was: Splitting YARN-7783 to move the refactoring to a separate JIRA) > Factor out management of temp tags from AllocationTagsManager > - > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Fix For: YARN-6592 > > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch, YARN-7788-YARN-6592.003.patch > > > Instead of using the AllocationTagsManager to store the temp tags that get > generated when placing containers with constraints, we will use a > LocalAllocationtagsManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7788) Factor out management of temp tags from AllocationTagsManager
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7788: - Fix Version/s: YARN-6592 > Factor out management of temp tags from AllocationTagsManager > - > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Fix For: YARN-6592 > > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch, YARN-7788-YARN-6592.003.patch > > > Splitting YARN-7783 to move the refactoring to a separate JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7788) Factor out management of temp tags from AllocationTagsManager
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-7788: - Summary: Factor out management of temp tags from AllocationTagsManager (was: Refactor AllocationTagsManager to move management of temp tags to a separate class) > Factor out management of temp tags from AllocationTagsManager > - > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch, YARN-7788-YARN-6592.003.patch > > > Splitting YARN-7783 to move the refactoring to a separate JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7788) Refactor AllocationTagsManager to move management of temp tags to a separate class
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335466#comment-16335466 ] Konstantinos Karanasos commented on YARN-7788: -- Committing this shortly to the branch. Thanks for working on this [~asuresh] and for the review [~leftnoteasy]. > Refactor AllocationTagsManager to move management of temp tags to a separate > class > -- > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch, YARN-7788-YARN-6592.003.patch > > > Splitting YARN-7783 to move the refactoring to a separate JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7788) Refactor AllocationTagsManager to move management of temp tags to a separate class
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335232#comment-16335232 ] Konstantinos Karanasos commented on YARN-7788: -- Sounds good, will commit the patch tonight. > Refactor AllocationTagsManager to move management of temp tags to a separate > class > -- > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch, YARN-7788-YARN-6592.003.patch > > > Splitting YARN-7783 to move the refactoring to a separate JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7788) Refactor AllocationTagsManager to move management of temp tags to a separate class
[ https://issues.apache.org/jira/browse/YARN-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335188#comment-16335188 ] Konstantinos Karanasos commented on YARN-7788: -- Thanks for the patch, [~asuresh]. +1 for separating the refactoring from YARN-7783. Patch looks good. Only comment is that I think you don't need to instantiate the EphemeralTagsManager each time. You can keep a single copy at the DefaultPlacementAlgo and use that instance each time you call the placement algorithm. > Refactor AllocationTagsManager to move management of temp tags to a separate > class > -- > > Key: YARN-7788 > URL: https://issues.apache.org/jira/browse/YARN-7788 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Major > Attachments: YARN-7788-YARN-6592.001.patch, > YARN-7788-YARN-6592.002.patch > > > Splitting YARN-7783 to move the refactoring to a separate JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7780) Documentation for Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333138#comment-16333138 ] Konstantinos Karanasos commented on YARN-7780: -- Thanks [~cheersyang] – will upload a first patch within the next few days and will let you know if I need help on specific parts. I will certainly give some DS commands for people to easily try it out, good point. > Documentation for Placement Constraints > --- > > Key: YARN-7780 > URL: https://issues.apache.org/jira/browse/YARN-7780 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos >Priority: Major > > JIRA to track documentation for the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333136#comment-16333136 ] Konstantinos Karanasos commented on YARN-7778: -- Sure [~cheersyang], go ahead. Let me know if you want to discuss about any details. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Priority: Major > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7763) Allow Constraints specified in the SchedulingRequest to override application level constraints
[
https://issues.apache.org/jira/browse/YARN-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331979#comment-16331979
]
Konstantinos Karanasos commented on YARN-7763:
--
Thanks [~cheersyang]. Only one minor thing: the instanceof check has to be done
after we call the transformer. If we do it before, it might be a
CardinalityConstraint or a TargetConstraint, all of which get transformed to a
SingleConstraint. So we have to do it just right before the cast. Sorry for not
clarifying it properly.
> Allow Constraints specified in the SchedulingRequest to override application
> level constraints
> --
>
> Key: YARN-7763
> URL: https://issues.apache.org/jira/browse/YARN-7763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-7763-YARN-6592.001.patch,
> YARN-7763-YARN-6592.002.patch, YARN-7763-YARN-6592.003.patch,
> YARN-7763-YARN-6592.004.patch
>
>
> As I mentioned on YARN-6599, we will add SchedulingRequest as part of the
> PlacementConstraintUtil method and both of processor/scheduler implementation
> will use the same logic. The logic looks like:
> {code:java}
> PlacementConstraint pc = schedulingRequest.getPlacementConstraint();
> If (pc == null) {
> pc =
> PlacementConstraintMgr.getPlacementConstraint(schedulingRequest.getAllocationTags());
> }
> // Do placement constraint match ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331969#comment-16331969
]
Konstantinos Karanasos commented on YARN-7774:
--
Thanks [~asuresh] for the patch. Looks good overall, some comments:
* Do we really need the CircularIterator? It seems to me that you can have a
normal iterator initialized outside the for loop and then each time
hasNext()=false, you can re-initialize it. But maybe I am missing something.
* For what [~cheersyang] mentioned about not being affinity friendly, you
could make a check whether the constraint has minCardinality>0 and scope=NODE,
and then keep the iterator at the same place in such a case. But if you feel it
is an over-optimization for the time being, I am fine tackling it in another
JIRA. Up to you.
* Do we clean up the black list for each tag? It seems that black-listing can
change based on the allocations that have been done so far, so we might need to
use it carefully.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7763) Allow Constraints specified in the SchedulingRequest to override application level constraints
[
https://issues.apache.org/jira/browse/YARN-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331917#comment-16331917
]
Konstantinos Karanasos commented on YARN-7763:
--
Thanks [~cheersyang].
{quote}I did not do that because I think we should have something better than
{{instanceof}} to tell which constraint we are dealing with. E.g would a
{{getType}} possible ?
{quote}
Without the instanceof, the cast will throw an exception though in case the
user adds a composite constraint.
We could add a getType later if we see that we have other than these two
constraint types to deal with.
{quote}We need to define the behavior how we merge constraints when there is
several ones, we can have more discussion in a followup JIRA.
{quote}
Agreed. I just file YARN-7778, so that we can do it later. Could you please add
a TODO when you deal with the different levels here, mentioning that Jira so
that we don't forget to perform the merging?
+1 otherwise.
> Allow Constraints specified in the SchedulingRequest to override application
> level constraints
> --
>
> Key: YARN-7763
> URL: https://issues.apache.org/jira/browse/YARN-7763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-7763-YARN-6592.001.patch,
> YARN-7763-YARN-6592.002.patch, YARN-7763-YARN-6592.003.patch
>
>
> As I mentioned on YARN-6599, we will add SchedulingRequest as part of the
> PlacementConstraintUtil method and both of processor/scheduler implementation
> will use the same logic. The logic looks like:
> {code:java}
> PlacementConstraint pc = schedulingRequest.getPlacementConstraint();
> If (pc == null) {
> pc =
> PlacementConstraintMgr.getPlacementConstraint(schedulingRequest.getAllocationTags());
> }
> // Do placement constraint match ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7778) Merging of constraints defined at different levels
Konstantinos Karanasos created YARN-7778: Summary: Merging of constraints defined at different levels Key: YARN-7778 URL: https://issues.apache.org/jira/browse/YARN-7778 Project: Hadoop YARN Issue Type: Sub-task Reporter: Konstantinos Karanasos When we have multiple constraints defined for a given set of allocation tags at different levels (i.e., at the cluster, the application or the scheduling request level), we need to merge those constraints. Defining constraint levels as cluster > application > scheduling request, constraints defined at lower levels should only be more restrictive than those of higher levels. Otherwise the allocation should fail. For example, if there is an application level constraint that allows no more than 5 HBase containers per rack, a scheduling request can further restrict that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7763) Allow Constraints specified in the SchedulingRequest to override application level constraints
[
https://issues.apache.org/jira/browse/YARN-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331832#comment-16331832
]
Konstantinos Karanasos commented on YARN-7763:
--
Just a small clarification to my above point about constraint priority.
Eventually we need to support constraint merging and checking constraint
conflicts.
If an app-level constraints says that there should not be more than 5 hbase
containers per rack, then a scheduling request-constraint can be more
restrictive (e.g., request no more than 3 hbase per rack), but it cannot be
less restrictive (e.g., request no more than 7 per rack), because that will
contradict the app-level constraint.
Until we get the constraint merging, the priority is not clear (I can see
reasons for both ways). In any case, for now it does not matter, because we
don't expect to have constraints in different levels.
> Allow Constraints specified in the SchedulingRequest to override application
> level constraints
> --
>
> Key: YARN-7763
> URL: https://issues.apache.org/jira/browse/YARN-7763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-7763-YARN-6592.001.patch,
> YARN-7763-YARN-6592.002.patch, YARN-7763-YARN-6592.003.patch
>
>
> As I mentioned on YARN-6599, we will add SchedulingRequest as part of the
> PlacementConstraintUtil method and both of processor/scheduler implementation
> will use the same logic. The logic looks like:
> {code:java}
> PlacementConstraint pc = schedulingRequest.getPlacementConstraint();
> If (pc == null) {
> pc =
> PlacementConstraintMgr.getPlacementConstraint(schedulingRequest.getAllocationTags());
> }
> // Do placement constraint match ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7763) Allow Constraints specified in the SchedulingRequest to override application level constraints
[
https://issues.apache.org/jira/browse/YARN-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331812#comment-16331812
]
Konstantinos Karanasos commented on YARN-7763:
--
A couple of things by a quick look in the patch:
* In the canSatisfyConstraints, probably we should do an instanceof
SingleConstraint before the cast (and we can explicitly say in the TODO that we
don't handle CompositeConstraints – we say generally other constraints)
* Not sure about the constraint priority. If I have a global constraint (by
the cluster operator), I want to override the app constraint and I want the app
constraint to override the specific scheduling request constraint. Eventually
we should merge such constraints if possible, but for the time being, I think
that should be the priority. Right now it is the other way around.
> Allow Constraints specified in the SchedulingRequest to override application
> level constraints
> --
>
> Key: YARN-7763
> URL: https://issues.apache.org/jira/browse/YARN-7763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-7763-YARN-6592.001.patch,
> YARN-7763-YARN-6592.002.patch, YARN-7763-YARN-6592.003.patch
>
>
> As I mentioned on YARN-6599, we will add SchedulingRequest as part of the
> PlacementConstraintUtil method and both of processor/scheduler implementation
> will use the same logic. The logic looks like:
> {code:java}
> PlacementConstraint pc = schedulingRequest.getPlacementConstraint();
> If (pc == null) {
> pc =
> PlacementConstraintMgr.getPlacementConstraint(schedulingRequest.getAllocationTags());
> }
> // Do placement constraint match ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous minor fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331286#comment-16331286
]
Konstantinos Karanasos commented on YARN-7774:
--
Thanks for the patch, [~asuresh].
Given that when we are doing an allocation with constraints we take into
account the state of the cluster before the allocation is done, I think that in
the {{PlacementConstraintsUtil}} we should check for {{<=}} to the max
cardinality and not for {{<}}.
For example, if we are placing a scheduling request with tag {{hb-m}} and the
constraint dictates that there should be no more than two {{hb-rs}} in a node,
then if there are two, placement should happen.
Similarly, if we are placing a scheduling request with tag {{hb}} and the
constraint dictates there should be no more than two other {{hb}} containers in
the node, then if there are two in a node, placement should happen (we are
looking at the state before the new allocation). This means that if a user does
not want to end up with three {{hb}} containers, (s)he should put as maximum
cardinality 1 and not 2.
> Miscellaneous minor fixes to the PlacementProcessor
> ---
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-7774-YARN-6592.001.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * The Placement Algorithm should shuffle the order it looks at nodes between
> two requests - to ensure a) more nodes are looked at and b) bias against
> placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
