Hi,
Has anyone seen hadoop getting stuck on reduces?
I'm using a compiled version of hadoop from cloudera:
Hadoop 0.18.3-14.cloudera.CH0_3
Subversion -r HEAD
Compiled by root on Mon Jul 6 15:02:31 EDT 2009
I've a map reduce job and hadoop gets stuck at 96.49% with no progress since
the last 1 hour. I've tried to look into the logs and there isn't anything
interesting there. Here's the log from the last 1 hour:
>>>>
2009-09-17 14:51:47,033 ERROR org.apache.hadoop.dfs.DataNode:
DatanodeRegistration(10.42.1.1:50010,
storageID=DS-1052225239-129.170.192.42-50010-1252112893659, infoPort=50075,
ipcPort=50020):DataXceiver: java.net.SocketTimeoutException: 480000 millis
timeout while waiting for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/10.42.1.1:50010
remote=/10.42.255.247:41915]
at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at org.apache.hadoop.dfs.DataNode$BlockSender.sendChunks(DataNode.java:1938)
at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2032)
at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1159)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1087)
at java.lang.Thread.run(Thread.java:619)
2009-09-17 14:52:29,916 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40
blocks got processed in 7 msecs
2009-09-17 15:15:06,586 INFO org.apache.hadoop.dfs.DataBlockScanner:
Verification succeeded for blk_4472685249728744796_2145
2009-09-17 15:52:28,686 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40
blocks got processed in 8 msecs
2009-09-17 16:52:30,397 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40
blocks got processed in 8 msecs
>>>>
while the hadoop reduce is stuck at 96.49 for the last 1 hr:
09/09/17 15:49:49 INFO mapred.JobClient: map 100% reduce 96%
This is the second time I've tried to run this code and both of the times I've
seen it hitting a barrier on reduce. My reduce step is just aggregating all of
the keys and dumping them into files using MultipleTextOutputFormat. I've run a
problem of similar size before where the input to reduce was of the same size
as this one.
Any help would be greatly appreciated. I can't seem to find any reason why this
is happening.
Thanks
Himanshu
Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.
