Hi, Just to provide more info. By setting the "mapred.job.tracker" to local which make the program run locally, everything works fine. but turn to fully cluster the exception comes.
2009/3/2 Nick Cen <[email protected]> > Hi, > > I have set the seperator value, but the same exception is thrown. > As i take the first part of the whole key as the key, so i think even i > didn't set the correct seperator value, it will not thrown and array index > out of bound exception. > > > 2009/3/2 jason hadoop <[email protected]> > >> Did you by change set the separator character to ','? >> map.output.key.field.separator is the key, the default is TAB. >> >> On Sun, Mar 1, 2009 at 6:13 PM, Nick Cen <[email protected]> wrote: >> >> > Hi, >> > >> > my key has the format "key1,key2,key3“,and >> > conf.setKeyFieldPartitionerOptions("-k 1,1"). When i limit the input >> size, >> > it works fine, i think this because i limit the total number of the >> > possible >> > "key1,key2,key3" compositions. but when i increate the input size, this >> > exception was thrown. >> > >> > 2009/3/2 jason hadoop <[email protected]> >> > >> > > The way you are specifying the section of your key to compare is >> reaching >> > > beyond the end of the last part of the key. >> > > >> > > Your key specification is not terminating explicitly on the last >> > character >> > > of the final field of the key. >> > > >> > > if your key splits in to N parts, and you are comparing on the Nth >> part, >> > > -kN,N will work while -kN will throw the exception. >> > > >> > > The way the comparator picks up a piece, is it takes the piece and the >> > > trailing separator by default. For the last piece there is no trailing >> > > separator and you get the array out of bounds exception.. >> > > >> > > >> > > >> > > On Sun, Mar 1, 2009 at 5:38 PM, Nick Cen <[email protected]> wrote: >> > > >> > > > java.lang.ArrayIndexOutOfBoundsException: 4096 >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.io.WritableComparator.compareBytes(WritableComparator.java:129) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.mapred.lib.KeyFieldBasedComparator.compareByteSequence(KeyFieldBasedComparator.java:109) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.mapred.lib.KeyFieldBasedComparator.compare(KeyFieldBasedComparator.java:85) >> > > > at >> > > > org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:308) >> > > > at >> > > > >> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:139) >> > > > at >> > > > >> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:270) >> > > > at >> > > org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:285) >> > > > at >> > > > >> org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:870) >> > > > at >> > > org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:829) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) >> > > > at ufida.ReduceTask.reduce(ReduceTask.java:39) >> > > > at ufida.ReduceTask.reduce(ReduceTask.java:1) >> > > > at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> > > > at org.apache.hadoop.mapred.Child.main(Child.java:155) >> > > > >> > > > my hadoop version is 0.19.0, if i limit the input file number, the >> > > > exception >> > > > wil not be thrown. >> > > > -- >> > > > http://daily.appspot.com/food/ >> > > > >> > > >> > >> > >> > >> > -- >> > http://daily.appspot.com/food/ >> > >> > > > > -- > http://daily.appspot.com/food/ > -- http://daily.appspot.com/food/
