[
https://issues.apache.org/jira/browse/YARN-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-5001:
-----------------------------
Description:
Usually, the group owner for /tmp/logs, where the aggregated logs go, is
"hadoop". Under that dir, you then have
<username>/logs/<application_id>/<files_per_node> with group being "hadoop" all
the way down.
If you delete the /tmp/logs dir (when you want to clean up all the logs), the
directory will be created with a different group "superuser". The JHS runs as
the mapred user, who is a member of the hadoop group. With the new group, the
JHS doesn't have permission to read the logs any more.
was:
Usually, the owner and group for /tmp/logs, where the aggregated logs go, is:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt - mapred hadoop 0 2016-04-14 15:46 /tmp/logs
Under that dir, you then have
<username>/logs/<application_id>/<files_per_node>. The group should be hadoop
all the way down, while the user should be mapred at the top, and <username>
starting with the <username> dir and below.
If you delete the /tmp/logs dir (when you want to clean up all the logs):
[root@nightly57-1 ~]# sudo -u hdfs hadoop fs -rmr /tmp/logs
And then run an MR job:
[root@nightly57-1 ~]# hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 2 2
…
The directory will be created with a different owner and group:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt - yarn supergroup 0 2016-04-14 18:12 /tmp/logs
The owner being yarn might be okay, though this is inconsistent with the
original owner, mapred. However, the real problem is the group now being
supergroup instead of hadoop. The JHS runs as the mapred user, who is a member
of the hadoop group. With the new owner and group, the JHS doesn't have
permission to read the logs any more.
> Aggregated Logs root directory is created with wrong owner and group if
> nonexistent
> ------------------------------------------------------------------------------------
>
> Key: YARN-5001
> URL: https://issues.apache.org/jira/browse/YARN-5001
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.0
> Reporter: Haibo Chen
> Assignee: Haibo Chen
>
> Usually, the group owner for /tmp/logs, where the aggregated logs go, is
> "hadoop". Under that dir, you then have
> <username>/logs/<application_id>/<files_per_node> with group being "hadoop"
> all the way down.
> If you delete the /tmp/logs dir (when you want to clean up all the logs), the
> directory will be created with a different group "superuser". The JHS runs as
> the mapred user, who is a member of the hadoop group. With the new group, the
> JHS doesn't have permission to read the logs any more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
