Hi,
We have an avro map/reduce job used to be working with avro 1.4, but broken
with avro 1.5 when the reducer tried to do de-serilization. By looking at the
trace, it looked like the reducer was broken when trying to resolve 'union' of
a 'map' definition in our avdl schema. We have three fields in our schema
relating to this. These are:
union {map <union {int,long,float,string}>, null} evpl;
union {map <union {int,long,float,string}>, null} plst;
union {map <union {int,long,float,string}>, null} change;
Can anybody let me know if this is a 1.5 bug? The stack trace was as follows:
java.lang.ArrayIndexOutOfBoundsException: -1576799025 at
org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) at
org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) at
org.apache.avro.io.parsing.Parser.advance(Parser.java:88) at
org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) at
org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141) at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86)
at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68)
at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136) at
org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076) at
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
at
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242)
at
org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47)
at
com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46) at
com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1) at
org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60) at
org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30) at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) at
org.apache.hadoop.mapred.Child$4.run(Child.java:240) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234)
Thanks.
Ey-Chih Chow
