Hello,
I am running specific Map/Reduce task on Hadoop cluster 0.19.2, the job was split on 509 maps, 507 maps run quickly enough, 1-2 minutes each; cluster capacity: 9 maps, 3 reduces. The problem is with last 2 maps. I can't monitor it, but it is doing something at the end. I don't think task is hanged, but I was forced to increase timeout to even 10 hours; default is 600000 milliseconds. It runs already 2 hours, I am still waiting. I can see this in data node logs, and I believe last two maps do something (sorting?)... "top" command shows constantly 100% for the process (3-core CPU); some I/O wait time, etc. 2009-12-07 17:55:26,257 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-5446742124105873401_5013 2009-12-07 17:58:26,976 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_943490007545023289_5013 2009-12-07 18:11:55,926 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_1600535934489014079_5013 2009-12-07 18:16:57,888 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-5513951734130352427_5013 2009-12-07 18:26:24,374 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-8095951827886168570_5013 2009-12-07 18:27:28,976 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_8799624630405158602_5013 2009-12-07 18:38:30,432 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_7949352562734117454_5013 2009-12-07 18:43:47,935 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 394 blocks got processed in 18 msecs 2009-12-07 18:45:05,563 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-7431375870171222932_5013 2009-12-07 18:56:21,270 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-2572350130286370167_5013 Any idea? Thanks, Fuad Efendi +1 416-993-2060 http://www.tokenizer.ca Data Mining, Vertical Search
