[ http://issues.apache.org/jira/browse/NUTCH-170?page=comments#action_12362482 ]
Doug Cutting commented on NUTCH-170: ------------------------------------ I have sucessfully used mapred.local.dir with multiple values on many occasions. Can you please try to distill this to an easy to reproduce test-case? Thanks. > Crash with multiple temp directories > ------------------------------------ > > Key: NUTCH-170 > URL: http://issues.apache.org/jira/browse/NUTCH-170 > Project: Nutch > Type: Bug > Reporter: Rod Taylor > Priority: Critical > > A brief read of the code indicated it may be possible to use multiple local > directories using something like the below: > <property> > <name>mapred.local.dir</name> > <value>/local,/local1,/local2</value> > <description>The local directory where MapReduce stores intermediate > data files. > </description> > </property> > This failed with the below exception during either the generate or update > phase (not entirely sure which). > java.lang.ArrayIndexOutOfBoundsException > at java.util.zip.CRC32.update(CRC32.java:51) > at > org.apache.nutch.fs.NFSDataInputStream$Checker.read(NFSDataInputStream.java:92) > at > org.apache.nutch.fs.NFSDataInputStream$PositionCache.read(NFSDataInputStream.java:156) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > at java.io.BufferedInputStream.read(BufferedInputStream.java:313) > at java.io.DataInputStream.readFully(DataInputStream.java:176) > at > org.apache.nutch.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55) > at > org.apache.nutch.io.DataOutputBuffer.write(DataOutputBuffer.java:89) > at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:378) > at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:301) > at org.apache.nutch.io.SequenceFile$Reader.next(SequenceFile.java:323) > at > org.apache.nutch.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:60) > at > org.apache.nutch.segment.SegmentReader$InputFormat$1.next(SegmentReader.java:80) > at org.apache.nutch.mapred.MapTask$2.next(MapTask.java:106) > at org.apache.nutch.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.nutch.mapred.MapTask.run(MapTask.java:116) > at > org.apache.nutch.mapred.TaskTracker$Child.main(TaskTracker.java:604) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
