Here are few hadoop exceptions that I am getting while running mapred job on 700MB of data on a 3 node cluster on Windows platform (using cygwin):
1. 2009-01-08 17:54:10,597 INFO org.apache.hadoop.dfs.DataNode: writeBlock blk_-4309088198093040326_1001 received exception java.io.IOException: Block blk_-4309088198093040326_1001 is valid, and cannot be written to. 2009-01-08 17:54:10,597 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.120.12.91:50010, storageID=DS-70805886-10.120.12.91-50010-1231381442699, infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException: Block blk_-4309088198093040326_1001 is valid, and cannot be written to. at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:921) at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:2364) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1218) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1076) at java.lang.Thread.run(Thread.java:619) 2. This particular job succeeded. Is it possible that this task was a speculative execution and was killed before it could be started. Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2195) 3. 2009-01-15 21:27:13,547 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200901152118_0001_r_000000_0 Merge of the inmemory files threw an exception: java.io.IOException: Expecting a line not the end of stream at org.apache.hadoop.fs.DF.parseExecResult(DF.java:109) at org.apache.hadoop.util.Shell.runCommand(Shell.java:179) at org.apache.hadoop.util.Shell.run(Shell.java:134) at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:296) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2105) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2078) 4. 2009-01-15 21:27:13,547 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 47 files left. 2009-01-15 21:27:13,579 WARN org.apache.hadoop.mapred.TaskTracker: Error running child java.io.IOException: attempt_200901152118_0001_r_000000_0The reduce copier failed at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) 5. Caused by: java.io.IOException: An established connection was aborted by the software in your host machine ... 12 more Can anyone help me in giving some pointers to what could be the issue. Thanks, Sandeep -- View this message in context: http://www.nabble.com/Hadoop-Exceptions-tp21548261p21548261.html Sent from the Hadoop core-user mailing list archive at Nabble.com.