[
https://issues.apache.org/jira/browse/YARN-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497377#comment-16497377
]
Eric Yang commented on YARN-8384:
---------------------------------
[~vinodkv], there are 3 different paths to start docker container:
1. Someone who runs distributed shell that append:
{code}
1> <LOG_DIR>/stdout 2> LOG_DIR>/stderr
{code}
Filename is user defined, but the file permission on the file depending on
umask of the docker image. By default umask is 022, and anyone can read the
file using other bit. The file is owned by uid:gid of the submission user in
secure mode, or nobody:nobody in insecure mode. This is a bit leaky by
security standard. Hadoop 3.1 implementation does not break the backward
compatibility for this mode.
2. Yarn Native Service yarn mode
This mode initializes stdout.txt and stderr.txt to uid of submission user, and
gid of node manager. End user or viewing log file via node manager web
application is the only two possible users to look at the log. If end user
tries to add redirection of logs to other filename, the generated file
permission will end up as the docker container uid:gid and umask of the docker
container. However, the output will end up in stdout.txt and stderr.txt
because those redirection are appended last in the launch command.
3. Yarn Service docker mode (ENTRY_POINT)
When using ENTRY_POINT, the stdout and stderr are written to stdout.txt and
stderr.txt through dup2 redirection. It is not possible to use shell command
redirection because the execution is via execvp without shell expansion. User
can choose to write log to additional mount directories, but the custom logs
will not be aggregated by YARN framework.
Option 1 and 2 are kept around for backward compatibility reasons, but it is
possible for container to write file with permission that node manager can not
process. The setup of stdout.txt and stderr.txt to owned by launching user,
and readable by node manager in option 3 is safest and recommended for future
development.
> stdout, stderr logs of a Native Service container is coming with group as
> nobody
> --------------------------------------------------------------------------------
>
> Key: YARN-8384
> URL: https://issues.apache.org/jira/browse/YARN-8384
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
> Reporter: Sunil Govindan
> Assignee: Eric Yang
> Priority: Critical
> Labels: docker
> Attachments: YARN-8384.001.patch
>
>
> When {{yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users}}
> is set to true, and
> {{yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user}} is
> set to nobody.
> This will cause the docker to run as nobody:nobody in yarn mode.
> The log files will be initialized as nobody:nobody:
> {noformat}
> rw-rr- 1 nobody hadoop 354 May 31 17:33 container-localizer-syslog
> rw-rr- 1 nobody hadoop 1042 May 31 17:35 directory.info
> rw-r---- 1 nobody hadoop 4944 May 31 17:35 launch_container.sh
> rw-rr- 1 nobody hadoop 440 May 31 17:35 prelaunch.err
> rw-rr- 1 nobody hadoop 100 May 31 17:35 prelaunch.out
> rw-r---- 1 nobody nobody 18733 May 31 17:37 stderr.txt
> rw-r---- 1 nobody nobody 400 May 31 17:35 stdout.txt
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
