Hi Abhay, Ideally the error line - "Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out" suggests you either do not have permissions for output folder or disk is full.
Also 5 is not a big number on thread spawning, (infact, default on parallelcopies) to recommend reducing it, but a lower value might work.only long-term indications are for your system to under-go node maintenance. Thanks Rekha From: Abhay Ratnaparkhi <[email protected]<mailto:[email protected]>> Reply-To: <[email protected]<mailto:[email protected]>> Date: Tue, 28 Aug 2012 14:52:27 +0530 To: <[email protected]<mailto:[email protected]>> Subject: error in shuffle in InMemoryMerger Hello, I am getting following error when reduce task is running. "mapreduce.reduce.shuffle.parallelcopies" property is set to 5. org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_119.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF Regards, Abhay
