I would post the configuration files -- easier for someone to spot something wrong than to imagine what configuration would get you to that stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out would suggest you might not have hadoop.tmp.dir set (?) On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie < [email protected]> wrote: > Hi, > > Is this a coding or a setup issue ? > > I¹m using Hadoop 2.41 > My program is doing a concordance on 500,000 sequences of 400 chars. > My cluster set is 32 data nodes and two masters. > > The exact error is: > Error: java.io.IOException: Spill failed > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas > k.java:1535) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692) > at > org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput > OutputContextImpl.java:89) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp > er.java:112) > at > par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: > 96) > at > par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: > 1) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j > ava:1556) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) > Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could > not find any valid local directory for > attempt_1405523201400_0006_m_000000_0_spill_8.out > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW > rite(LocalDirAllocator.java:402) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat > or.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat > or.java:131) > at > org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil > es.java:159) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java: > 1566) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85 > 3) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja > va:1505) > > Regards, > > Chris > > >
