[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451780#comment-15451780 ] stefanlee commented on YARN-1963: - [~sunilg] Thanks your jira, i have a doubt that "creating multiple queues and making users submit applications to higher priority and lower priority queues separately" in your doc,it means we create multiple queues, e.g. queue A and queue B,then label A higher priority queue ,label B lower priority queue, after that ,user can submit higher priority application to A ?but i look up your code ,it means user can submit different priority application to same queue and the queue has a default priority,the cluster has a max priority. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design _02.pdf, YARN Application Priorities Design.pdf, YARN > Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637512#comment-14637512 ] Sunil G commented on YARN-1963: --- Hi [~grey] Yes. I am updating the document now. I will upload a version of doc soon. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637303#comment-14637303 ] Lei Guo commented on YARN-1963: --- [~sunilg], YARN-2003 already committed, should we get the design document updated? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596481#comment-14596481 ] Jian He commented on YARN-1963: --- I think we need to move this forward.. Overall, I prefer using numeric priority to label-based priority because the former is simpler and more flexible if user wants to define a wide range of priorities. no extra configs. User does not need to be educated about the new mapping any time the mapping changes. Also, one problem is that if we refresh the priority mapping while some existing long-running jobs are already running on certain priority, how do we map the previous priority mapping range to the new priority mapping range? In addition, if everyone runs the application at “VERY_HIGH” priority, the “HIGH” priority, though named as “HIGH”, is not really the “HIGH” priority any more. It actually becomes the “LOWEST” priority. My point is that the importance of priority will make sense only when compared with its peers. In that sense, I think adding a utility to surface how applications are distributed across each priority so that user can reason about how to place the application on certain priority may be more useful than adding a static naming mapping to let people reason about the relative importance of priority by naming. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507501#comment-14507501 ] Sunil G commented on YARN-1963: --- Thank you [~grey] for sharing the thoughts. As per the design, integer will be used in schedulers all alone. Hence all comparisons and operations can be done on integer. However we can have a label mapping for the integer which can be used while application submission, and to view in UI etc. Labels can be added as only a mappings to integer. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507332#comment-14507332 ] Lei Guo commented on YARN-1963: --- Agree with [~jlowe], integer is the base of priority, and label should be just an alias during application submission. If we keep both label and integer in the system, it could be complicate when administrator changing the label/range mapping. It's true that we do not expect the user to assign many different priorities, but we may enhance scheduler to calculate priority dynamically based on certain criteria, for example, the pending time or at certain time frame. In this case, the priority could be any number. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499399#comment-14499399 ] Sunil G commented on YARN-1963: --- Yes. We could try support both Integer and Label (with mappings). We may open independent Jiras to handle this case (have both patches, will sync up as one) , and which should achieve the same goal w/o complexity. And we will look for simpler version for now, not complex re-mappings etc. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498986#comment-14498986 ] Wangda Tan commented on YARN-1963: -- Agree that we need to make basic functionality works, I suggest to keep the simple tight-packed alias for now, we should try to get both in, but label-based which shouldn't block int-based priority development. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366194#comment-14366194 ] Wangda Tan commented on YARN-1963: -- bq. I feel we can make such label config in a common place which can be accessible for any schedulers. Agree, this should be a part of YARN configuration. I put it as a part of queue config just for readability for the proposal :). > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365408#comment-14365408
]
Sunil G commented on YARN-1963:
---
HI [~vinodkv] , [~leftnoteasy] , [~eepayne] [~jlowe]
Using a priority as integer itself to scheduler's will be the first target, A
manager which can act as a single point of contact which can translate from
label to integer and vice versa. Yes, this will be an added complexity in RM,
but if it can be taken out of Scheduler it reduces much of manipulation logic.
{noformat}
0:VERY_LOW, 1:LOW, 2:NORMAL, 3:HIGH, 4:VERY_HIGH
{noformat}
I feel we can make such label config in a common place which can be accessible
for any schedulers.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: 0001-YARN-1963-prototype.patch, YARN Application
> Priorities Design.pdf, YARN Application Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363790#comment-14363790
]
Eric Payne commented on YARN-1963:
--
{quote}
I think label-based and integer-based priorities are just two different ways to
configure as well as API. No matter we choose to use label-based or
integer-based priority, we should use integer only to implement internal logic
(like in CapacityScheduler).
{quote}
I think that is true especially when passing priorities through proto buffers,
using integers is best.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: 0001-YARN-1963-prototype.patch, YARN Application
> Priorities Design.pdf, YARN Application Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363645#comment-14363645
]
Wangda Tan commented on YARN-1963:
--
I think label-based and integer-based priorities are just two different ways to
configure as well as API. No matter we choose to use label-based or
integer-based priority, we should use integer only to implement internal logic
(like in CapacityScheduler).
In addition, we can make "label-based" priority to be just an alias of
"integer-based" priority. For example, if we define queue's usable label to be
\[0-5\], we need to add one "label" alias for *all* priorities if we want to
support label-based priority (such as VERY_LOW, LOW, NORMAL, HIGH, VERY_HIGH).
When alias is setting, YARN can accept label for priority and can show them on
web UI.
With this, configuration/API and our internal logic could be consistent.
For example, a label-based configuration:
{code}
0:VERY_LOW, 1:LOW, 2:NORMAL, 3:HIGH, 4:VERY_HIGH
NORMAL
HIGH
HIGH
VERY_HIGH
...
{code}
Which means:
- Each integer has an alias
- For users from "u1", "u2", "g1", their default application priority is HIGH,
and they can manually set its priority up to VERY_HIGH
- For other users, their default priority is NORMAL, and they can set its
priority up to HIGH
This configuration is equal to number-based config.
{code}
2
3
3
4
...
{code}
bq. ...For example, take an app framework that runs multiple utility jobs
simultaneously and needs to set the relative priorities between them. If label
names themselves are in the API then that app framework doesn't work on some
clusters that don't have the expected labels configured...
I think this may not be a very big issue, no matter application uses label or
integer, additional configuration should be made. We should only edit
configuration instead of making changes of code.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: 0001-YARN-1963-prototype.patch, YARN Application
> Priorities Design.pdf, YARN Application Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363334#comment-14363334 ] Jason Lowe commented on YARN-1963: -- bq. Do we have a range? Otherwise, nothing stops users from setting their priority be INTEGER_MAX and everybody scratching their heads. With any range, users could just set to the range max. The problem of having all the users set their priority to the highest is an orthogonal problem to how priorities are represented. As similarly suggested for labels, I think there should be a portion of the numerical range for highest priority reserved for admins. bq. If we have a range, which side is up? is -20 >20 like unix (isn't intuitive at all to me) or -20 < 20 (intuitive)? I'd prefer higher priority values lead to higher priority in scheduling, but I don't care too much either way. bq. What does a negative priority means anything anyways? A priority in isolation is meaningless since it only derives semantic meaning when compared to another priority. This applies to labels as well. A job running at VERY_HIGH priority is not what you originally thought if everything else in the cluster is running at VERY_VERY_HIGH priority. So negative priorities are just a part of the numerical range, and it's straightforward to compare negative numbers as well as positive numbers and know their ordering. If it's still thought to be too confusing we can always limit priorities to >=0 without losing much. bq. Admin comes and says "I need a new super-high priority", now your ranges need to be dynamically size-able. Like UNIX, it would be easy to add limitations to the high priority range so users can't just arbitrarily set their priorities to the highest level. Then admins will always have the ability to make something higher priority than what users were able to set. I don't mind if we want to have label mappings to numerical priorities, but my biggest concern with labels is putting them in the API itself. For example, take an app framework that runs multiple utility jobs simultaneously and needs to set the relative priorities between them. If label names themselves are in the API then that app framework doesn't work on some clusters that don't have the expected labels configured. Or another concern: if one has to use labels to specify priority, what if they really need 40 different priorities? Can they come up with that many descriptive label names that users can reason their relative ordering just based on the name? Most likely they will end up using label names like PRIORITY_1, PRIORITY_2, ... PRIORITY_50, and the pain of having to configure all of that. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361409#comment-14361409 ] Vinod Kumar Vavilapalli commented on YARN-1963: --- Assuming integers are supported - Do we have a range? Otherwise, nothing stops users from setting their priority be INTEGER_MAX and everybody scratching their heads. - If we have a range, which side is up? is -20 >20 like unix (isn't intuitive at all to me) or -20 < 20 (intuitive)? - Either ways, it is an implicit decision that needs to be documented and told to users explicitly. Labels convey that without any of that. - What does a negative priority means anything anyways? - Admin comes and says "I need a new super-high priority", now your ranges need to be dynamically size-able. I don't see a difference between say 10 priorities and 10 labeled priorities, other than that labels are better in the following - They are more *human readable* on the UI and CLIs: "This app has priority 19" doesn't give much feedback as much as "This app has HIGH priority" - Even if we don't want them now, you can let admins create new priorities between two existing ones, create a new priority lower than the lowest easily etc. With integers, you start with 0-10, then adding one more lower than them all takes them into negative priorities' territory making it all confusing. - Specifying restrictions is very straight forward: for a root.enginnering queue, VERY_HIGH can be only be used by (u1,u2, g1), HIGH by (u3, u4) and everything else by everyone. The way I see it, we will provide a predefined set of labeled priorities that should work for 80% of the clusters, the remaining can define their own set. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357600#comment-14357600 ] Eric Payne commented on YARN-1963: -- Thanks, [~sunilg], for your work on in-queue priorities. Along with [~nroberts], I'm confused about why priority labels are needed. As a user, I just need to know that the higher the number, the higher the priority. Then, I just need a way to see what priority each application is using and a way to set the priority of applications. To me, it just seems like labels will get in the way. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357241#comment-14357241 ] Sunil G commented on YARN-1963: --- Thank you [~vinodkv] and [~nroberts] for the comments. Considering usability ways, labels will be handy. And scheduler must be agnostic of labels and should handle only integers like in linux. This will have a complexity on priority manager inside RM which will translate label -> integer an vice versa. But a call can be taken by seeing all possibilities and can be standardized the same so that a minimal working version can be pushed in by improvising on the patches submitted (working prototype was attached). Hoping [~leftnoteasy] and [~eepayne] to join the discussion. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design.pdf, YARN Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353677#comment-14353677
]
Nathan Roberts commented on YARN-1963:
--
{quote}
Without some sort of labels, it will be very hard for users to reason about the
definition and relative importance of priorities across queues and cluster. We
must support the notion of priority-labels to make this feature usable in
practice.
{quote}
Maybe I'm missing something... Isn't it relatively easy to reason about 2<4 and
therefore 2 is lower priority than 4? Unix/Linux hasn't had labels for
priorities and it seems to be working pretty well there. Even if I have labels,
I have to make sure that all queues and clusters define them precisely the same
way or I wind up just as confused, if not even more. Just my $0.02
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: 0001-YARN-1963-prototype.patch, YARN Application
> Priorities Design.pdf, YARN Application Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353390#comment-14353390
]
Vinod Kumar Vavilapalli commented on YARN-1963:
---
{quote}
As per discussion happened in YARN-2896 with Eric Payne and Wangda Tan, there
is proposal to use Integer alone as priority from client and as well as in
server. As per design doc, a priority label was used as wrapper for user and
internally server was using corresponding integer with same. We can continue
discussion on this here in parent JIRA. Looping Vinod Kumar Vavilapalli.
Current idea:
yarn.prority-labels = low:2, medium:4, high:6
Proposed:
yarn.application.priority = 2, 3 , 4
{quote}
Without some sort of labels, it will be very hard for users to reason about the
definition and relative importance of priorities across queues and cluster. We
must support the notion of priority-labels to make this feature usable in
practice.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: 0001-YARN-1963-prototype.patch, YARN Application
> Priorities Design.pdf, YARN Application Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329358#comment-14329358 ] Wangda Tan commented on YARN-1963: -- That's great! Thanks. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329357#comment-14329357 ] Wangda Tan commented on YARN-1963: -- That's great! Thanks. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329359#comment-14329359 ] Wangda Tan commented on YARN-1963: -- That's great! Thanks. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328450#comment-14328450 ] Sunil G commented on YARN-1963: --- Thank you Wangda and Jason for the input Yes, it's good to change the priority of an application at runtime. I had mentioned it in the design doc. I have created a user api jira already, and it's client part can be handled there. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328071#comment-14328071 ] Jason Lowe commented on YARN-1963: -- I'd like to see changing app priorities addressed as it is a common ask from users. In many cases jobs are submitted to the cluster via some workflow/pipeline, and they would like to change the priority of apps already submitted. Otherwise they have to update their workflow/pipeline to change the submit-time priority, kill the active jobs, and resubmit the apps for the priority to take effect. Then eventually they need to change it all back to normal priorities later. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327995#comment-14327995 ] Wangda Tan commented on YARN-1963: -- One more question: I didn't see there's an API proposed to update app priority, I think it may be very useful when a job ran for some time, and need get completed as soon as we can. Is this a valid use case that we need to do within YARN-1963 scope? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327243#comment-14327243 ] Sunil G commented on YARN-1963: --- Thank you [~devaraj.k] for input I have updated the subjiras and uploaded patch by considering integer rather than label names. As mentioned, we can have the enums supported from MR side (can try using enums). But a translation table is needed for same and its better keep the same YarnClient side. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327196#comment-14327196 ] Devaraj K commented on YARN-1963: - I would also agree for numbers rather than labels for not to make it more complex. If we are moving with numbers, I think we can just use the existing priority API from ApplicationSubmissionContext.setPriority(Priority priority) and not required any new API's to expose to clients. We may need to think for M/R Job priory case, M/R Job supports enums for priority (i.e. VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW) and we need to have some mechanism to map these enums to priority numbers. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319937#comment-14319937 ] Sunil G commented on YARN-1963: --- Sorry. Due some error in webpage, last message got updated many times. Kindly ignore. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319933#comment-14319933 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319931#comment-14319931 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319934#comment-14319934 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319927#comment-14319927 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319936#comment-14319936 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319926#comment-14319926 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319935#comment-14319935 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319932#comment-14319932 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319929#comment-14319929 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319924#comment-14319924 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319930#comment-14319930 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319928#comment-14319928 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319925#comment-14319925 ] Sunil G commented on YARN-1963: --- Pls discard this message. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293789#comment-14293789 ] Sunil G commented on YARN-1963: --- Yes [~leftnoteasy] Idea of having a range will clearly define the priority subset each queue can adhere. I will share a patch on same by keeping same subjiras. Hoping to have a look by [~vinodkv] on these latest changes. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290300#comment-14290300 ] Eric Payne commented on YARN-1963: -- +1 on using numbers and not labels. It seems that the use of labels adds more complexity in mapping, sending via PB, and converting back to numbers, and does not seem to add much clarity. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289832#comment-14289832 ] Wangda Tan commented on YARN-1963: -- Thanks for summary from [~sunilg], I think priority should be a range instead of a set of numbers, may be we can refer to how linux do it, the range \[-N, +N], and 0 is default priority. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289619#comment-14289619
]
Sunil G commented on YARN-1963:
---
As per discussion happened in YARN-2896 with [~eepayne] and [~leftnoteasy],
there is proposal to use Integer alone as priority from client and as well as
in server. As per design doc, a priority label was used as wrapper for user and
internally server was using corresponding integer with same. We can continue
discussion on this here in parent JIRA. Looping [~vinodkv].
Current idea:
{noformat}
yarn.prority-labels = low:2, medium:4, high:6
{noformat}
Proposed:
{noformat}
yarn.application.priority = 2, 3 , 4
{noformat}
Thank you for sharing your thoughts. I will now upload scheduler changes which
can be reviewed meantime.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: YARN Application Priorities Design.pdf, YARN Application
> Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223827#comment-14223827 ] Sunil G commented on YARN-1963: --- Thank you Wangda and [~eepayne] for the comments. ACL, if configured for a queue, will be considered before submitting the job. If there are no configuration, only queue ACL will be checked which is same as what is happening now. priority label level ACL is on top of queue level ACL which is extra and can be configured as needed by admin. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223369#comment-14223369
]
Eric Payne commented on YARN-1963:
--
Hi [~sunilg]. Thanks for the work you are doing on this issue.
bq. {{yarn.scheduler.capacity.root...acl}}
If this property doesn't exist, will queue admins still be able to change
priorities of jobs in the queue?
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: YARN Application Priorities Design.pdf, YARN Application
> Priorities Design_01.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223127#comment-14223127 ] Wangda Tan commented on YARN-1963: -- [~sunilg], I agree with your latest comment, Will get back to you once I read the new design doc. Thanks, > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf, YARN Application > Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219019#comment-14219019 ] Sunil G commented on YARN-1963: --- Thank you [~leftnoteasy] bq. I'd prefer highest + default priority. This configuration will make it easier for admins to config the same. Still I am not convinced with default acceptance coming from lower priorities by default. But I am not seeing any use case where this lower priorities are a problem also. Yes, we can have this as highest + default (this one i already have). Instead of labels per queue, it will be changed as highest per queue. I will update doc as per same, also my patch. bq. extra complexity both in implementation and configuration I agree about the more complicated config and implementation for this part. As you mentioned, if a preemption feature related to YARN-2069 runs in parallel, then the issue which I pointed out can be solved. So user-limit factor preemption if considers priority also, we can get the head room which is needed. User has to enable this preemption though. If this is workaround way is fine for resolving the issue mentioned, then I will file a jira to relate priority with user-limit preemption. Kindly share your thoughts. bq. I didn't see any related code in YarnClient Yes, this code is now in YarnRunner which is part of map reduce. I wanted to see it with YarnClient. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218381#comment-14218381 ] Wangda Tan commented on YARN-1963: -- [~sunilg], Thanks for reply, bq. All we need a YarnClient implementation for taking this config and setting to ApplicationSubmissionContext. ( Something similar to queue name which this app is submitted to ). Yes, that will be helpful, I wanna make sure they're not in YarnClient now? (including the queue). I didn't see any related code in YarnClient bq. The idea sounds good. The reason for specifying each label needed for a queue is because admin can specify the labels applicable for a queue. With high priority, we may always end up having default acceptance of lower priorities. How do you feel about having this as a range "low-high" Instead of having low-high range, I'd prefer highest + default priority. Admin can specify highest priority for queue/user, and default priority for queue/user bq. I have a use case scenario here. There are few applications running in a queue from 4 different users (sub... I understood the use case here, but I think maybe an easier way is not change the definition of user limit. Like having preemption mechanism to support higher priority applications take resource from lower priority applications, etc. Divide user limit by priority will add extra complexity both in implementation and configuration. bq. I suggest to add preemption within queue considering priority. ... +1. Already filed a subjira for this. The preemption I mentioned here is not YARN-2009, is to support the previous use case you mentioned, we can keep user-limit as-is, and enforce higher priority application can get resource, that should be possible :) Thanks, Wangda > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217672#comment-14217672
]
Sunil G commented on YARN-1963:
---
Hi Wangda,
Thank for sharing your comments.
bq. Does this means, any YARN application doesn't need change a line of their
code,
yarn.app.priority can be passed from client side. And if client can set the
priority value to ApplicationSubmissionContext which is received from this
config, then RM can get the same. All we need a YarnClient implementation for
taking this config and setting to ApplicationSubmissionContext. ( Something
similar to queue name which this app is submitted to ).
bq. Specify only highest priority for queue and user
The idea sounds good. The reason for specifying each label needed for a queue
is because admin can specify the labels applicable for a queue. With high
priority, we may always end up having default acceptance of lower priorities.
How do you feel about having this as a range "low-high"
{noformat}
cluster labels {very_high, high, medium, low}
yarn.scheduler.root..priority_label=low-high
yarn.scheduler.capacity.root..high.acl=user1,user2
yarn.scheduler.capacity.root..low.acl=user3,user4
{noformat}
This was the intention. Please share your thoughts [~vinodkv] [~gp.leftnoteasy]
bq. I think we shouldn't consider user limit within priority level
I have a use case scenario here. There are few applications running in a queue
from 4 different users (submitted to priority level low) and user-limit factor
is 20. 5th user has ACL for submitting high priority applications. Because of
user-limit, he can get only 20% maximum for his high priority apps. This high
priority apps submitted by user5 may need more resource which intern will be
rejected by user-limit check. How do you feel this use case?
bq. I suggest to add preemption within queue considering priority.
+1. Already filed a subjira for this.
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: YARN Application Priorities Design.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[
https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216674#comment-14216674
]
Wangda Tan commented on YARN-1963:
--
Thanks [~sunilg], [~vinodkv] for your great effort on this!
I've just read through the design doc, some comments:
1) yarn.app.priority
How this get to be implemented? Does this means, any YARN application doesn't
need change a line of their code, can specify priority when submit the app
using yarn CLI?
I think if this can be done, we should extend to other YARN parameters like
queue, node-label-expression, etc.
2) Specify only highest priority for queue and user
I found there are property like
{{yarn.scheduler.root..priority_label=high,low}} and
{{yarn.scheduler.capacity.root...acl=user1,user2}}.
I would perfer just specify only highest priority for queue and user. For
example, it doesn't make sense to me if priority = \{high,mid,low\}, and a
queue can access \{high,low\} only. Is there any benefit to specify individual
priorities instead of highest priority?
3) User limit and priority
I think we shouldn't consider user limit within priority level, because the
priority is not a specific kind of resource. Comparing to node label, you
cannot say, user-X of queue-A used 8G highest priority resource, but you can
say, user-X of queue-A used 8G resource in node with label=GPU. There's no
difference for a 2G resource allocated to highest/lowest priority.
If we want to implement this,
bq. it will not be fair to schedule resources in a uniform manner for all
application in a queue with respect to user limits.
I suggest to add preemption within queue considering priority. Upon YARN-2069,
we can considering user-limit and priority together -- while enforcing
user-limit, we always preempt from lower priority applications.
Any thoughts?
Thanks,
Wangda
> Support priorities across applications within the same queue
> -
>
> Key: YARN-1963
> URL: https://issues.apache.org/jira/browse/YARN-1963
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, resourcemanager
>Reporter: Arun C Murthy
>Assignee: Sunil G
> Attachments: YARN Application Priorities Design.pdf
>
>
> It will be very useful to support priorities among applications within the
> same queue, particularly in production scenarios. It allows for finer-grained
> controls without having to force admins to create a multitude of queues, plus
> allows existing applications to continue using existing queues which are
> usually part of institutional memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205445#comment-14205445 ] Eric Payne commented on YARN-1963: -- Thanks a lot [~sunilg] for taking the lead on this and putting together the design document. I have a question about per-priority ACLs. Can per-priority ACLs within a queue be inherited from queue-level ACLs if the per-priority ACLs aren't there? In a cluster that already has queues divided to be specific to business units, they will want to only specify the queue-level ACL list. In other words, in this use case, the queue-level users are already trusted enough to modify apps on that queue, regardless of priority, so they won't want the extra overhead of specifying additional priority-level ACLs. Is that part of the design? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14149696#comment-14149696 ] Sunil G commented on YARN-1963: --- Thank you [~maysamyabandeh] for providing us the use cases. 1. bq.use case seems to be mentioned in Item 3 of Section 1.5.3 Yes. By changing priority of an application at runtime, will help to over come the scenario mentioned by you. I will in-cooperate the same by providing more scenarios and impacts about it. 2. bq.priority can also be incorporated to the fair share calculation Application Priority will be supported by both schedulers. And there are sub jiras opened for same, however we can re allign the same w.r.t the same base design, and I will include changes from Fair also. As of now priority labels and internal implementation will be common, however separate ACL/per queue priority-label configurations will be required per scheduler level. In future, when both scheduler shares same config and common code, this can be pulled out as common code. For now, configurations and its specific implementation can be done separate for both schedulers. Sub jiras will be split ted accordingly > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148474#comment-14148474 ] Maysam Yabandeh commented on YARN-1963: --- Thanks [~sunilg] for the design doc. It might be useful if I share with you our use cases. Our most important use case is to let the admin change an app priority at runtime while it is running. The example is when a job gets unlucky taking much longer than usual due to some node failures or bugs. The user complains that the job is about to miss the deadline and admin needs a way to prioritize the user's job over the other jobs in the queue. This use case seems to be mentioned in Item 3 of Section 1.5.3 in the design doc but its "priority" seems not to be high. Another use case is to dynamically give a job higher priority based on the job status. For example, when mapper fails and there is no headroom in the queue, the job preempt its reducers to make space for its mappers. The freed space is however not necessarily offered back to the job in fair scheduling. Ideally job could increase its priority when its reducers are being stalled waiting for its mappers to be assigned. bq. Once all these requests of higher priority applications are served, then lower priority application requests will get served from Resource Manager. We are using fair scheduler and I assumed this jira is to also cover that since YARN-2098 created as a sub-task. The design doc however seems to be fairly centered around CapacityScheduler. In the case of fair scheduler, I guess the priority can also be incorporated to the fair share calculation, instead of the strict order of high priority first. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137564#comment-14137564 ] Sunil G commented on YARN-1963: --- HI [~maysamyabandeh], We are bringing up a design doc for this by capturing all details, will soon publish the same. [~vinodkv], could we discuss doc this offline and publish it. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137496#comment-14137496 ] Maysam Yabandeh commented on YARN-1963: --- I was wondering what is the long-term plan for this jira? It does not seem to have any activity in the past 4 months and I was wondering if we have any rough estimate that on which release we plan to have this feature added? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988528#comment-13988528 ] Ravi Prakash commented on YARN-1963: I wonder if it'd be a good idea to percolate the priorities onto the actual containers as well? (I'm thinking (re)nice-ing container processes) ? That way we can submit more jobs than can all fit into memory and take advantage of OS scheduling to pick up the ones with the highest priority? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987553#comment-13987553 ] Rohith commented on YARN-1963: -- Added to Sunil thoughts, priority of jobs can also be displayed at RM web UI.!! > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986163#comment-13986163 ] Arun C Murthy commented on YARN-1963: - [~sunilg] thanks for taking this up! As [~vinodkv] mentioned; a short writeup will help - look forward to helping get this in; thanks again! > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985900#comment-13985900 ] Vinod Kumar Vavilapalli commented on YARN-1963: --- [~sunilg], thanks for taking this up. This is a really useful feature! Before we jump into patches, we should consider writing up a small design doc that describes the requirements and the mechanism of implementation - not necessarily class-level design. There are few things to consider on the top of my head: - Values of priorities - static values like you described before or few known priorities backed by integers leaving gaps for more powerful interaction with priorities - ACLs on priorities - If we don't have some such mechanism, users will all be incentivized to submit apps all with the highest priority. - The classic priority inversion problem: MAPREDUCE-314 I am sure there are more things to consider once we start thinking through this. I can help write this down, let me know what you think. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985318#comment-13985318 ] Sunil G commented on YARN-1963: --- Thank you Sandy for the review. As you have mentioned, I will create these subtasks and will handle seperately. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985193#comment-13985193 ] Sandy Ryza commented on YARN-1963: -- Thanks for picking this up Sunil. Can we separate this into a couple JIRAs? One for the ResourceManager and protocol changes, one for the MapReduce changes, and one for the Capacity Scheduler changes. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985160#comment-13985160 ] Sunil G commented on YARN-1963: --- We have done few analysis and implemented support for application priority. I wish to share the thoughts here, kindly check the same. Design thoughts: 1. Configuration Part We planned to use some existing priority configuration as given below. These are used to set a Job priority. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration "mapreduce.job.priority". The values for priority can be VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW 2. Scheduler Side If the Capacity Scheduler queue has multiple applications(Jobs) to run with different priorities, CS will allocate containers for the highest priority application and then for next priority and so on. When multiple queues are configured with different capacities, this priority will work internal to the each queue. For this, we planned to add a priority comparison check in the below data structure. Comparator applicationComparator We added a priority check here in compare() of applicationComparator while selecting applications. Updated design here will be like, 1. Check for priority first. If there, return highest priority job. 2. Continue existing logic such as App ID comparison and TimeStamp comparison. With these changes, we can make highest priority job will get preference in a queue. NB: In addition to this, we added a preemption module also to get High priority jobs resources fast by preempting lower priority ones. I wish to upload a patch if this approach is fine. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.2#6252)
