Scott, thanks for the information. Do you have any suggestion about my latest findings? I don't think the there is any wrong with the AVRO record itself, but when multi records together in the MR intermediate data, in this case, somehow trigger this exception. I have this 900M avor file, and I know when this exception happened for my MR job, the reducer is trying to deserialize a value with key = '3a410d00656911e3ac0bd4ae52986b44', which is the list id in my schema.Now if I change my mapper, only omit the three key value pairs: one of them is the above key, and 2 more randomly pick up keys. To my surprise, I didn't get this exception in the reducer of MR job.If I remove the this filter, I will get this exception in MR job.So with more key/value pairs in the intermediate result, I will get this exception consistently for this key. But if I explicit pick up this avro record, and 2 other records, I cannot reproduce this problem.I can confirm the list id is unique in this AVRO file, and I also changed the ReducerContextImpl of Hadoop 2.2.0 codebase, to dump the key when the value deserialize failed like following:https://github.com/apache/hadoop/blob/release-2.2.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.javaline 145 ==> value = valueDeserializer.deserialize(value);to ==> try{ value = valueDeserializer.deserialize(value); }catch (RuntimeException re){ System.out.println("key = " + key); throw re }So right now, I am stuck to simplify the data, any idea how should I debug this issue? Thanks
> Date: Mon, 25 Jan 2016 12:08:44 -0800 > Subject: Re: Question related to org.apache.avro.io.DirectBinaryDecoder > From: [email protected] > To: [email protected] > > Looking at the ticket (sorry, can¹t update in JIRA at the moment): > > The value 101 in the raw data is the integer -51. > > Therefore the cause is either: > > * Corrupt data > * Improper schema used to read (was written with a different schema than > the reader is configured to use for its Œwriter¹ schema) > > You might want to Œlook around¹ the bytes near that 101 and see if the > data looks like it is from a different schema than expected. > > Given that some other tools can read it, it is likely the latter case ‹ > the other tools are reading it with a different schema. > > Note, a reader requires _two_ schemas: The schema that the reader wants > to interpret the data as, and the schema that was used when the data was > written. If the latter is wrong, this sort of thing can happen as Avro > tries to read a variable length item from data that is in the wrong > position. > > You could also see if BinaryDecoder behaves any differently from > DirectBinaryDecoder. The issue is most likely above those ‹in the code > that uses these (the resolver and/or DatumReader). > > -Scott > > On 1/23/16, 10:59 AM, "Yong Zhang" <[email protected]> wrote: > > >Hi, Avro Developers: > > > >Is anyone familiar the code logic related to > >org.apache.avro.io.DirectBinaryDecoder? > > > >I am asking this question related to AVRO-1786, which I believe I am > >facing a bug related to this class. > > > >A valid Avro record sent from Mapper to the Reducer, but Reducer cannot > >read it due to IndexOutOfBoundException, because the readInt() method of > >this class return "-51". > > > >I even can dump the local variables of the method in this exception case, > >and described in the comments area of Jira ticket. > > > >I don't understand the internal logic of this class, and how the > >readInt() method implemented. But an inputstream read 101 bytes out will > >cause this method return a negative number, and causes following method > >IndexOutofBoundException looks like a bug to me. > > > >Can anyone understand this class's logic confirm this is a bug or not? If > >it is a bug, what is the best way to fix it? > > > >I can consistently reproduce this bug on our production cluster, which > >mean I can verify any code fix works or not for this case. > > > >Let me know any question related to this JIRA. > > > >Thanks > > > >Yong > > > >
