thanks Scott, this is actually due to the bug that I filed. once that is solved , it should be fine.
I created my own because I want to swap the encoder easily , right now DataFile uses binaryEncoder only. I do keep a separate marker which is persisted *kind of* synchronously as I read out records. On Fri, Sep 2, 2011 at 5:58 PM, Scott Carey <[email protected]> wrote: > AvroDataFile deals with this for some cases. Is it an acceptable API for > your use case? You can configure the block size to be very small and/or > flush() regularly. > > If you do this on your own, you will need to track the position that you > start to read a record at, and if there is a failure, rewind and reset the > reader to that position. > > -Scott > > > On 8/25/11 7:17 PM, "Yang" <[email protected]> wrote: > > >I'm trying to implement an on-disk queue, which contains avro records, > >SpecificRecord > > > >my queue implementation basically contains a > >SpecificDatumWriter, and a SpecificDatumReader pointing to the same file > >. > > > >the problem is, that when the reader reaches the EOF, I can no longer > >use it again, > >even after I append more records to the file, if I call the same > >SpecificDatumReader.read() again, > >it gave me exceptions: > > > > > >-------------------------------------------------------------------------- > >----- > >Test set: blah.MyTest > >-------------------------------------------------------------------------- > >----- > >Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257 > >sec <<< FAILURE! > >testBasic(blah.MyTest) Time elapsed: 0.24 sec <<< ERROR! > >java.lang.ArrayIndexOutOfBoundsException > > at java.lang.System.arraycopy(Native Method) > > at > >org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j > >ava:670) > > at > >org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453) > > at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120) > > at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405) > > at > >org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > > at > >org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > > at > >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14 > >2) > > at > >org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j > >ava:166) > > at > >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13 > >8) > > at > >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12 > >9) > > at blah.DiskEventsQueue.dequeue2(MyTest.java:55) > > at blah.MyTest.testBasic(MyTest.java:85) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > >57) > > at > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm > >pl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:616) > > > > > > > > > >Thanks > >Yang > > >
