Hi All,
We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to submit
map/reduce jobs. But we have noticed that these map/reduce job submission
failing time to time due to a permission issue occurring in Hadoop. When
this happens, we have to use the hadoop hdfs command and change the file
permission of mapreduce staging directory. Following is the error that is
getting logged.
[2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} - Job
Submission failed with exception 'java.io.IOException(The
ownership/permissions on the staging directory hdfs://
hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is not
as expected. It is owned by inosh and permissions are rwxr-xr-x. The
directory must be owned by the submitter inosh or by inosh and permissions
must be rwx------)'
java.io.IOException: The ownership/permissions on the staging directory
hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
directory must be owned by the submitter inosh or by inosh and permissions
must be rwx------
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
at
org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
...
When we get this error we go to hadoop name node and execute following
command to set proper permission to the folder mentioned in the exception.
./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
We would like know why this happens and how to fix this permanently.
Thanks and Regards,
Inosh
