Hi, I am using multiple outputs in our job. So whenever any reduce task fails, all it's next task attempts are failing with file exist exception.
The output file name should also append the task attempt right? But it's only appending the task id. Is this the bug or Some thing wrong from my side? Where should look in src code? I went through code at FileOutputFormat$getTaskOutputPath(), but there it's only considering task id. Exception Trace: 13/12/29 09:13:00 INFO mapred.JobClient: Task Id : attempt_201312162255_60465_r_000008_0, Status : FAILED 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93% 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96% 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97% 13/12/29 09:19:34 INFO mapred.JobClient: Task Id : attempt_201312162255_60465_r_000008_1, Status : FAILED org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31 either because the filename is invalid or the file exists at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy7.create(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy7.create(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443) at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131) at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411) at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370) at com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254) at com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89% 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92% 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96% 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97% 13/12/29 09:23:48 INFO mapred.JobClient: Task Id : attempt_201312162255_60465_r_000008_2, Status : FAILED org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33 either because the filename is invalid or the file exists at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732) at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448) Thanks & Regards, B Anil Kumar.
