Is mapreduce running as the hadoop user? If so it can’t erase the files in tmp. Which might be causing you some hilarity
:) J On 2011-05-02, at 7:43 PM, Jun Young Kim wrote: > Hi, > > To James. > > this is the permission of hadoop.tmp.dir. > > $> ls -al > drwxr-xr-x 6 juneng juneng 4096 5월 3 10:37 hadoop-juneng.2 > > > To Harsh. > yes, out cluster has 96 occupied reducer slots. > and my job is using 90 reduce tasks at one time to complete it. > > thanks for all. > > Junyoung Kim ([email protected]) > > > On 05/02/2011 08:32 PM, James Seigel wrote: >> What are your permissions on your hadoop.tmp.dir ? >> >> James >> >> Sent from my mobile. Please excuse the typos. >> >> On 2011-05-02, at 1:26 AM, Jun Young Kim<[email protected]> wrote: >> >>> hi, all. >>> >>> I got so many failures on a reducing step. >>> >>> see this error. >>> >>> java.io.IOException: Failed to delete earlier output of task: >>> attempt_201105021341_0021_r_000001_0 >>> at >>> org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:157) >>> at >>> org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:173) >>> at >>> org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:173) >>> at >>> org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:133) >>> at >>> org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:233) >>> at org.apache.hadoop.mapred.Task.commit(Task.java:962) >>> at org.apache.hadoop.mapred.Task.done(Task.java:824) >>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391) >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:396) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742) >>> at org.apache.hadoop.mapred.C >>> >>> >>> this error was happened after adopting MultipleTextOutputFormat class in my >>> job. >>> the job is producing thousands of different output files on a HDFS. >>> >>> anybody can guess reasons? >>> >>> thanks. >>> >>> -- >>> Junyoung Kim ([email protected]) >>>
