[
https://issues.apache.org/jira/browse/YARN-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667770#comment-15667770
]
Shane Kumpf commented on YARN-5736:
-----------------------------------
Parsing of container-executor.cfg appears to have been impacted by this patch.
If container-executor.cfg contains entries with empty values, parsing doesn't
work properly and all applications fail.
Here is the c-e output with additional debugging.
{code}
Failing this attempt.Diagnostics: Application application_1479220852002_0001
initialization failed (exitCode=255) with output: read_config :Conf file name
is : /usr/local/src/hadoop_install/hadoop/etc/hadoop/container-executor.cfg
read_config : Adding conf key : yarn.nodemanager.linux-container-executor.group
read_config : Adding conf value : yarn
read_config : Adding conf key : banned.users
read_config : Adding conf key : min.user.id
read_config : Adding conf value : 50
read_config : Adding conf key : allowed.system.users
read_config : Adding conf key : docker.binary
read_config : Adding conf value : /usr/bin/docker
read_config : Adding conf key : feature.docker.enabled
read_config : Adding conf value : 1
read_config : Adding conf key : feature.tc.enabled
read_config : Adding conf value : 1
Supplied key: yarn.nodemanager.linux-container-executor.group
Compare key: yarn.nodemanager.linux-container-executor.group
main : command provided 0
main : run as user is nobody
main : requested yarn user is root
get_value Supplied key: min.user.id
get_value Compare key: yarn.nodemanager.linux-container-executor.group
get_value Compare key: min.user.ids
get_value Compare key: docker.binarym.users
get_value Compare key: feature.docker.enabled
get_value Compare key: feature.tc.enabled
Requested user nobody is not whitelisted and has id 99,which is below the
minimum allowed 1000
{code}
What was observed is that the min.user.id value is not properly parsed, because
the key comparison in {{configuration.c#get_value}} will never succeed. This is
because the keys have extra characters appended (min.user.id vs min.user.ids
and docker.binary vs docker.binarym.users in the output above). Keys with an
"empty" value (represented by # in container-executor.cfg) are being
incorrectly combined with the key that follows in the config.
The relevant min.user.id parts below:
{code}
-snip-
get_value Supplied key: min.user.id
-snip-
get_value Compare key: min.user.ids
{code}
Note that banned.users and min.user.id are combined and docker.binary and
allowed.system.users are combined.
Here is an example container-executor.cfg that shows the issue:
{code}
yarn.nodemanager.linux-container-executor.group=yarn
banned.users=#
min.user.id=50
allowed.system.users=#
docker.binary=/usr/bin/docker
feature.docker.enabled=1
feature.tc.enabled=1
{code}
I haven't quite tracked down the root cause, but reverting this patch does
appear to resolve the issue. I'm going to reopen this issue for further
investigation.
> YARN container executor config does not handle white space
> ----------------------------------------------------------
>
> Key: YARN-5736
> URL: https://issues.apache.org/jira/browse/YARN-5736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Reporter: Miklos Szegedi
> Assignee: Miklos Szegedi
> Priority: Trivial
> Labels: oct16-medium
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5736.001.patch, YARN-5736.002.patch,
> YARN_5736.000.patch
>
>
> The container executor configuration reader does not handle white spaces or
> malformed key value pairs in the config file correctly or gracefully
> as an example the following key value line which is part of the configuration
> (note the << is used as a marker to show the extra trailing space):
> yarn.nodemanager.linux-container-executor.group=yarn <<
> is a valid line but when you run the check over the file:
> [root@test]#./container-executor --checksetup
> Can't get group information for yarn - Success.
> [root@test]#
> It fails to find the yarn group but it really tries to find the "yarn " group
> which fails. There is no trimming anywhere while processing the lines. If a
> space would be added in before or after the = sign a failure would also occur.
> Minor nit is the fact that a failure still is logged as a Success
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
