It looks like there's a bug where `Serialize.read()` will throw that exception if it reads a segment count greater than 2**31. It is supposed to throw this exception <https://github.com/capnproto/capnproto-java/blob/633b1f619939dae9089a45f208523d58d159a3f5/runtime/src/main/java/org/capnproto/Serialize.java#L67-L69> in that case, but that can fail when there is an overflow.
The segment count should not ever be that large, so there's probably also a bug somewhere on the writing side. On Mon, Jul 10, 2017 at 12:40 PM, Kenton Varda <[email protected]> wrote: > Hi Farid, > > Does the problem happen around 2^31 bytes in? My guess is that the library > is using an `int` somewhere where it should be using a `long`. Or, perhaps > it's trying to map more than 2^31 bytes in a single ByteBuffer, which won't > work since ByteBuffer seems to use `int`s for indexes. > > David, any thoughts? > > -Kenton > > On Fri, Jul 7, 2017 at 11:55 AM, Farid Zakaria <[email protected]> > wrote: > >> Hi everyone, >> >> I'm looking for some guidance on what I may be doing wrong. >> I'm serializing (unpacked) multiple MessageBuilder to FileChannel via a >> BufferedOutputStreamWrapper >> >> Here is a snippet of the code in Kotlin >> >> val partitionFileInputStream = FileOutputStream(filename, true).channel >> val buffered = BufferedOutputStreamWrapper(partitionFileInputStream) >> >> val recordIterator = RecordIterator(dataSource) >> recordIterator.asSequence().map { row -> >> converter(row) >> }.forEach { message -> >> Serialize.write(buffered, message) >> } >> >> buffered.flush() >> buffered.close() >> >> >> >> I'm writing millions of records to a file which is several GB in size at the >> end. >> >> I then try to read the file: >> >> >> val fileChannel = RandomAccessFile(filePath.toFile(), "r").getChannel() >> >> for(message in SerializedIterator(fileChannel)) { >> val record = message.getRoot(SomeClass.Object.factory) >> } >> >> >> here is the iterator implementation: >> >> >> class SerializedIterator(readChan: ReadableByteChannel) : >> AbstractIterator<MessageReader>(), AutoCloseable { >> >> val buffer = BufferedInputStreamWrapper(readChan) >> >> override fun close() { >> buffer.close() >> } >> >> override fun computeNext() { >> try { >> setNext(Serialize.read(buffer)) >> } catch (e : Error) { >> close() >> done() >> } >> } >> >> } >> >> >> >> It seems to go fine for several million records and then I get hit with: >> >> >> java.lang.NegativeArraySizeException: null >> at org.capnproto.Serialize.read(Serialize.java:91) >> ~[runtime-0.1.1.jar:0.1.1] >> at org.capnproto.Serialize.read(Serialize.java:51) >> ~[runtime-0.1.1.jar:0.1.1] >> at SerializedIterator.computeNext(SerializedIterator.kt:18) >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> Visit this group at https://groups.google.com/group/capnproto. >> > > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
