In core-site.xml I have:
<property>
<name>fs.default.name</name>
<value>hdfs://devubuntu05:9000</value>
<description>The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. </description>
</property>
In hdfs-site.xml I have
<property>
<name>hadoop.tmp.dir</name>
<value>/data/hadoop/tmp/hadoop-${user.name}</value>
<description>Hadoop temporary folder</description>
</property>
From: Arpit Gupta [mailto:[email protected]]
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: [email protected]
Subject: Re: Permission problem
Based on the logs your system dir is set to
<hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[email protected]>
wrote:
Thank you.
mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?
/tmp is already 777
kevin@devUbuntu05:~$ hadoop fs -ls /tmp
Found 1 items
drwxr-xr-x - hdfs supergroup 0 2013-04-29 15:45 /tmp/mapred
kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp
Found 1 items
drwxrwxrwt - hdfs supergroup 0 2013-04-29 15:45 /tmp
But notice that the mapred folder in the /tmp folder is 755.
So I changed it:
kevin@devUbuntu05 $ hadoop fs -ls -d /tmp
drwxrwxrwt - hdfs supergroup 0 2013-04-29 15:45 /tmp
kevin@devUbuntu05 $ hadoop fs -ls -R /tmp
drwxr-xr-x - mapred supergroup 0 2013-04-29 15:45 /tmp/mapred
drwxr-xr-x - mapred supergroup 0 2013-04-29 15:45
/tmp/mapred/system
I still get the errors in the log file:
2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir (
<hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
of permissions.
2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'
2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...
. . . . .
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
. . . . .
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x
2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
. . . . . .
From: Arpit Gupta [mailto:[email protected]]
Sent: Tuesday, April 30, 2013 9:25 AM
To: [email protected]
Subject: Re: Permission problem
what is your mapred.system.dir set to in mapred-site.xml?
By default it will write to /tmp on hdfs.
So you can do the following
create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
jobtracker and tasktrackers.
In case its set to /mapred/something then create /mapred and chown it to
user mapred.
--
Arpit Gupta
Hortonworks Inc.
<http://hortonworks.com/> http://hortonworks.com/
On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <
<mailto:[email protected]> [email protected]> wrote:
To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?
From: Kevin Burton [mailto:rkevinburton@ <http://charter.net> charter.net]
Sent: Tuesday, April 30, 2013 8:31 AM
To: <mailto:[email protected]> [email protected]
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem
That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.
From: Mohammad Tariq [ <mailto:[email protected]>
mailto:[email protected]]
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem
user?"ls" shows "hdfs" and the log says "mapred"..
Warm Regards,
Tariq
<https://mtariq.jux.com/> https://mtariq.jux.com/
<http://cloudfront.blogspot.com> cloudfront.blogspot.com
On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <
<mailto:[email protected]> [email protected]> wrote:
I have relaxed it even further so now it is 775
<mailto:kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$>
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /
Found 1 items
drwxrwxr-x - hdfs supergroup 0 2013-04-29 15:43 /
But I still get this error:
2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
From: Mohammad Tariq [mailto: <mailto:[email protected]>
[email protected]]
Sent: Monday, April 29, 2013 5:10 PM
To: <mailto:[email protected]> [email protected]
Subject: Re: Incompartible cluserIDS
make it 755.
Warm Regards,
Tariq
<https://mtariq.jux.com/> https://mtariq.jux.com/
<http://cloudfront.blogspot.com> cloudfront.blogspot.com
