Do you notice anything related in the NameNode logs? One reason for this is that the NameNode may be in safe mode for some reason, but there are many other reasons so the NameNode's log would be the best place to look for exactly why the complete()-op fails.
On Wed, Aug 17, 2011 at 8:20 AM, Jianxin Wang <[email protected]> wrote: > hi, > my job runs once ervey day. but it failed sometimes. > i checked the log in job tracker. It seems a hdfs error? > thanks a lot! > > 2011-08-16 21:07:13,247 INFO org.apache.hadoop.mapred.TaskInProgress: > Error from attempt_201106021431_1719_r_000498_1: org.ap > ache.hadoop.ipc.RemoteException: java.io.IOException: Could not > complete write to file /xxx/yyy/110816_210016/_tem > porary/_attempt_201106021431_1719_r_000498_1/part-r-00498 by > DFSClient_attempt_201106021431_1719_r_000498_1 > at > org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.complete(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.complete(Unknown Source) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3264) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3188) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) > at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959) > at > org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1290) > at > org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:78) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > -- Harsh J
