AvroDataFile deals with this for some cases. Is it an acceptable API for your use case? You can configure the block size to be very small and/or flush() regularly.
If you do this on your own, you will need to track the position that you start to read a record at, and if there is a failure, rewind and reset the reader to that position. -Scott On 8/25/11 7:17 PM, "Yang" <[email protected]> wrote: >I'm trying to implement an on-disk queue, which contains avro records, >SpecificRecord > >my queue implementation basically contains a >SpecificDatumWriter, and a SpecificDatumReader pointing to the same file >. > >the problem is, that when the reader reaches the EOF, I can no longer >use it again, >even after I append more records to the file, if I call the same >SpecificDatumReader.read() again, >it gave me exceptions: > > >-------------------------------------------------------------------------- >----- >Test set: blah.MyTest >-------------------------------------------------------------------------- >----- >Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257 >sec <<< FAILURE! >testBasic(blah.MyTest) Time elapsed: 0.24 sec <<< ERROR! >java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method) > at >org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j >ava:670) > at >org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453) > at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120) > at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405) > at >org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at >org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14 >2) > at >org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j >ava:166) > at >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13 >8) > at >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12 >9) > at blah.DiskEventsQueue.dequeue2(MyTest.java:55) > at blah.MyTest.testBasic(MyTest.java:85) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >57) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >pl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > > > > >Thanks >Yang
