Steve (Todd Lipcon helped here).
There are two users ( hdfs and mapred ) and one group (hadoop). All hdfs files are owned by hdfs and belong to the hadoop group. All mapred fles are owned by user mapred and belong to the hadoop group. For example 1.I have hadoop.tmp.dir as /hadoop/tmp and here are the permissions ( /hadoop 775 hdfs hadoop) (/hadoop/tmp - 1777 hdfs hadoop). 2. mapred.local.dir /hadoop/local . permisisons /hadoop/local is 775 hdfs hadoop 3. mapred.system.dir is /mapred/system 755 mapred system. 4. dfs.data.dir = /hadoop/dfs/data 755 hdfs hadoop 5. dfs.namedir = /hadoop/dfs/name 755 hdfs hadoop. Finally to get it to work you need to do - sudo -u hdfs hadoop fs -mkdir /mapred - sudo -u hdfs hadoop fs -chown mapred /mapred For regular M/R jobs the user needs to belong to the hadoop group. For fsadmin tasks ( formatting , fsck and such like you need to run them as hdfs user). Hope this works. Raj ________________________________ From: Steve Lewis <[email protected]> To: common-user <[email protected]> Sent: Tue, November 9, 2010 1:29:35 PM Subject: Permissions issue Using a copy of the Cloudera security-enabled CDH3b3, we installed vanilla hadoop in /home/www/hadoop Now when a try to run a job as me I get permission errors - I am not even sure if the error is in writing to local files or hdfs or where staging is but I need to set permissions to allow the job to work Any bright ideas 10/11/09 12:58:04 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=slewis, access=WRITE, inode="staging":www:supergroup:rwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:207) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:188) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:136) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4019) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:3993) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:1914) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:1882) at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:847) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344) -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
