[
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292620#comment-17292620
]
Wilfred Spiegelenburg commented on YARN-10652:
----------------------------------------------
I do not see the relation with placement rules or the FS for this fix at all.
The weight of a queue can be used with or without a placement rule. It is a
setting on a queue. Having this setting on a user based queue also does not
really make sense. It gives more resources to one user over others in the
queue. So I can see this being used on a parent queue or a group queue but not
on an individuals leaf queue. On top of that the queue path of the
configuration setting is not affected by this change. We're talking about this
setting:
{code:java}
yarn.scheduler.capacity.<queue-path>.user-settings.<user-name>.weight{code}
The weight is retrieved for a specific queue as defined in the _<queue-path>._
That part is already resolved and is not changed. The only resolution that is
changed is the _<user-name>_ part between the words _user-settings_. and
_.weight._ The queue path could be anything and is not in play here. It could
even a fixed configured queue or one mapped on a group name.
The administrator should know as minimal as possible, preferably nothing,
about the internals for storing users in the CS. If the queue mapping rule for
the user changes the dots to make it a single part of the queue path then that
is independent of this change. It still does not change the way the user is
stored in the CS. It changes the way you map a user to a queue in the placement
rules.
On the FS side we thought about standardising dot usage. We considered both
cases using and not _dot_ in the config files in user names. When I looked at
it I was not sure which was the correct solution. It could lead to strange
behaviour and extra administrative work. The admin forgets to remove the dot
and all of a sudden the config does not apply. That is why it never went
further than just the Jira YARN-5674.
With this change as proposed you will support weights for all users with a dot
except for the user called: something_.weights_ That will be the only user set
that breaks which is far less than breaking all users with a dot in the
username. I do not see any other <user-name> bound properties in the
configuration at the moment.
If you want to solve the generic dot issue for user based placement then that
is outside of this change.
> Capacity Scheduler fails to handle user weights for a user that has a "."
> (dot) in it
> -------------------------------------------------------------------------------------
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Affects Versions: 3.3.0
> Reporter: Siddharth Ahuja
> Assignee: Siddharth Ahuja
> Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format ->
> {{firstname.lastname}}. However, if you specify a username with this format
> against the Capacity Scheduler setting ->
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
> it fails to be applied and is instead assigned the default of 1.0f weight.
> This renders the user weight feature (being used as a means of setting user
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e.
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames
> with a "." (dot) in them.
> [1]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2]
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
