Hi, I think the below issue was only due to HDFS architecture and not map-reduce. BUt just to make sure that's the case, I am cross-posting to this group as well. I have also attached the program used to get this error. My input file contains abt 2 million+ records.
Any help is greatly appreciated. Thanks Sudhan S On Fri, Nov 4, 2011 at 2:47 PM, Sudharsan Sampath <sudha...@gmail.com>wrote: > Hi, > > I have a simple map-reduce program [map only :) ]that reads the input and > emits the same to n outputs on a single node cluster with max map tasks set > to 10 on a 16 core processor machine. > > After a while the tasks begin to fail with the following exception log. > > 2011-01-01 03:17:52,149 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=temp,temp > ip=/x.x.x.x cmd=delete > > src=/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2 > dst=null perm=null > 2011-01-01 03:17:52,156 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* > NameSystem.addStoredBlock: addStoredBlock request received for > blk_7046642930904717718_23143 on x.x.x.x:<port> size 66148 But it does not > belong to any file. > 2011-01-01 03:17:52,156 WARN org.apache.hadoop.hdfs.StateChange: DIR* > NameSystem.completeFile: failed to complete > /TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000 > because dir.getFileBlocks() is null and pendingFile is null > 2011-01-01 03:17:52,156 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 12 on 9000, call > complete(/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000, > DFSClient_attempt_201101010256_0006_m_000000_2) from x.x.x.x:<port> error: > java.io.IOException: Could not complete write to file > /TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000 > by DFSClient_attempt_201101010256_0006_m_000000_2 > java.io.IOException: Could not complete write to file > /TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000 > by DFSClient_attempt_201101010256_0006_m_000000_2 > at > org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497) > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962) > > > Looks like there's a delete command issued by FsNameSystem.audit before > the it errors out stating it could not complete write to the file inside > that.. > > Any clue on what could have gone wrong? > > Thanks > Sudharsan S >
TestMultipleOutputs.java
Description: Binary data
