Haibo Chen created YARN-5001:
--------------------------------
Summary: Aggregated Logs root directory is created with wrong
owner and group if nonexistent
Key: YARN-5001
URL: https://issues.apache.org/jira/browse/YARN-5001
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Haibo Chen
Assignee: Haibo Chen
Usually, the owner and group for /tmp/logs, where the aggregated logs go, is:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt - mapred hadoop 0 2016-04-14 15:46 /tmp/logs
Under that dir, you then have
<username>/logs/<application_id>/<files_per_node>. The group should be hadoop
all the way down, while the user should be mapred at the top, and <username>
starting with the <username> dir and below.
If you delete the /tmp/logs dir (when you want to clean up all the logs):
[root@nightly57-1 ~]# sudo -u hdfs hadoop fs -rmr /tmp/logs
And then run an MR job:
[root@nightly57-1 ~]# hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 2 2
…
The directory will be created with a different owner and group:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt - yarn supergroup 0 2016-04-14 18:12 /tmp/logs
The owner being yarn might be okay, though this is inconsistent with the
original owner, mapred. However, the real problem is the group now being
supergroup instead of hadoop. The JHS runs as the mapred user, who is a member
of the hadoop group. With the new owner and group, the JHS doesn't have
permission to read the logs any more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
