[jira] [Commented] (YARN-10103) Capacity scheduler: add support for create=true/false per mapping rule
[
https://issues.apache.org/jira/browse/YARN-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229194#comment-17229194
]
Szilard Nemeth commented on YARN-10103:
---
[~pbacsko] +1 on closing this jira.
> Capacity scheduler: add support for create=true/false per mapping rule
> --
>
> Key: YARN-10103
> URL: https://issues.apache.org/jira/browse/YARN-10103
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> You can't ask Capacity Scheduler for a mapping to create a queue if it
> doesn't exist.
> For example, this mapping would use the first rule if the queue exist. If it
> doesn't, then it proceeds to the next rule:
> {{u:%user:%primary_group.%user:create=false;u:%user%:root.default}}
> Let's say user "alice" belongs to the "admins" group. It would first try to
> map {{root.admins.alice}}. But, if the queue doesn't exist, then it places
> the application into {{root.default}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10103) Capacity scheduler: add support for create=true/false per mapping rule
[
https://issues.apache.org/jira/browse/YARN-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229191#comment-17229191
]
Peter Bacsko commented on YARN-10103:
-
I'm about to close this Jira for the following reasons:
1) "create" will be introduced in the new placement engine (YARN-10370). Let's
leave the existing codebase and format alone which is already hard to read.
2) We will come up with a new approach to create dynamic queues anywhere,
pretty much like in FS. Not sure if there's an upstream ticket yet, but soon
we'll create one.
cc [~snemeth]
> Capacity scheduler: add support for create=true/false per mapping rule
> --
>
> Key: YARN-10103
> URL: https://issues.apache.org/jira/browse/YARN-10103
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> You can't ask Capacity Scheduler for a mapping to create a queue if it
> doesn't exist.
> For example, this mapping would use the first rule if the queue exist. If it
> doesn't, then it proceeds to the next rule:
> {{u:%user:%primary_group.%user:create=false;u:%user%:root.default}}
> Let's say user "alice" belongs to the "admins" group. It would first try to
> map {{root.admins.alice}}. But, if the queue doesn't exist, then it places
> the application into {{root.default}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10103) Capacity scheduler: add support for create=true/false per mapping rule
[
https://issues.apache.org/jira/browse/YARN-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026601#comment-17026601
]
Peter Bacsko commented on YARN-10103:
-
[~maniraj...@gmail.com] I'm afraid that's not enough.
Consider a nested Primary Group placement rule in FS with create=true. This
results in the following mapping in CS:
{{u:%user:%primary_group.%user}}
Now we know that this works. But what to do with create=true? This means that
we have to know all primary groups in advance and generate the
{{yarn.scheduler.capacity.root.[groupName].auto-create-child-queue.enabled=true}}
properties which doesn't seem all that feasible. Same goes for secondary
groups. Also, if someone uses LDAP group mapping (which is pretty common in an
enterprise environment), then it makes things even more complicated. Not to
mention that these properties should be updated whenever there is a change in
the groups.
So I think it's absolutely necessary to have this feature sooner or later.
> Capacity scheduler: add support for create=true/false per mapping rule
> --
>
> Key: YARN-10103
> URL: https://issues.apache.org/jira/browse/YARN-10103
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> You can't ask Capacity Scheduler for a mapping to create a queue if it
> doesn't exist.
> For example, this mapping would use the first rule if the queue exist. If it
> doesn't, then it proceeds to the next rule:
> {{u:%user:%primary_group.%user:create=false;u:%user%:root.default}}
> Let's say user "alice" belongs to the "admins" group. It would first try to
> map {{root.admins.alice}}. But, if the queue doesn't exist, then it places
> the application into {{root.default}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10103) Capacity scheduler: add support for create=true/false per mapping rule
[
https://issues.apache.org/jira/browse/YARN-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023050#comment-17023050
]
Manikandan R commented on YARN-10103:
-
[~pbacsko] I think AutoCreateEnabledParentQueue can be used to achieve leaf
queue creation under parent queue (for which auto has been enabled) in
conjunction with queue mapping rules. Please check
yarn.scheduler.capacity..auto-create-child-queue.enabled property.
> Capacity scheduler: add support for create=true/false per mapping rule
> --
>
> Key: YARN-10103
> URL: https://issues.apache.org/jira/browse/YARN-10103
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Priority: Major
>
> You can't ask Capacity Scheduler for a mapping to create a queue if it
> doesn't exist.
> For example, this mapping would use the first rule if the queue exist. If it
> doesn't, then it proceeds to the next rule:
> {{u:%user:%primary_group.%user:create=false;u:%user%:root.default}}
> Let's say user "alice" belongs to the "admins" group. It would first try to
> map {{root.admins.alice}}. But, if the queue doesn't exist, then it places
> the application into {{root.default}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
