I'll mention too: the way I discovered this crash was when an HTTP client connected to a Cap'n Proto RPC port. The "GET " string now triggers the segment limit.
On Mon, Jul 10, 2017, 6:18 PM Ross Light <[email protected]> wrote: > Yeah, I remember hitting that bug in the Go implementation. I feel like > the security implications docs should also mention limits on number of > segments, since I can't imagine a case where you would use the full 2^32 > space. > > On Mon, Jul 10, 2017, 6:15 PM David Renshaw <[email protected]> wrote: > >> It looks like there's a bug where `Serialize.read()` will throw that >> exception if it reads a segment count greater than 2**31. It is supposed to >> throw this exception >> <https://github.com/capnproto/capnproto-java/blob/633b1f619939dae9089a45f208523d58d159a3f5/runtime/src/main/java/org/capnproto/Serialize.java#L67-L69> >> in that case, but that can fail when there is an overflow. >> >> The segment count should not ever be that large, so there's probably also >> a bug somewhere on the writing side. >> >> On Mon, Jul 10, 2017 at 12:40 PM, Kenton Varda <[email protected]> >> wrote: >> >>> Hi Farid, >>> >>> Does the problem happen around 2^31 bytes in? My guess is that the >>> library is using an `int` somewhere where it should be using a `long`. Or, >>> perhaps it's trying to map more than 2^31 bytes in a single ByteBuffer, >>> which won't work since ByteBuffer seems to use `int`s for indexes. >>> >>> David, any thoughts? >>> >>> -Kenton >>> >>> On Fri, Jul 7, 2017 at 11:55 AM, Farid Zakaria < >>> [email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> I'm looking for some guidance on what I may be doing wrong. >>>> I'm serializing (unpacked) multiple MessageBuilder to FileChannel via >>>> a BufferedOutputStreamWrapper >>>> >>>> Here is a snippet of the code in Kotlin >>>> >>>> val partitionFileInputStream = FileOutputStream(filename, true).channel >>>> val buffered = BufferedOutputStreamWrapper(partitionFileInputStream) >>>> >>>> val recordIterator = RecordIterator(dataSource) >>>> recordIterator.asSequence().map { row -> >>>> converter(row) >>>> }.forEach { message -> >>>> Serialize.write(buffered, message) >>>> } >>>> >>>> buffered.flush() >>>> buffered.close() >>>> >>>> >>>> >>>> I'm writing millions of records to a file which is several GB in size at >>>> the end. >>>> >>>> I then try to read the file: >>>> >>>> >>>> val fileChannel = RandomAccessFile(filePath.toFile(), "r").getChannel() >>>> >>>> for(message in SerializedIterator(fileChannel)) { >>>> val record = message.getRoot(SomeClass.Object.factory) >>>> } >>>> >>>> >>>> here is the iterator implementation: >>>> >>>> >>>> class SerializedIterator(readChan: ReadableByteChannel) : >>>> AbstractIterator<MessageReader>(), AutoCloseable { >>>> >>>> val buffer = BufferedInputStreamWrapper(readChan) >>>> >>>> override fun close() { >>>> buffer.close() >>>> } >>>> >>>> override fun computeNext() { >>>> try { >>>> setNext(Serialize.read(buffer)) >>>> } catch (e : Error) { >>>> close() >>>> done() >>>> } >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> It seems to go fine for several million records and then I get hit with: >>>> >>>> >>>> java.lang.NegativeArraySizeException: null >>>> at org.capnproto.Serialize.read(Serialize.java:91) >>>> ~[runtime-0.1.1.jar:0.1.1] >>>> at org.capnproto.Serialize.read(Serialize.java:51) >>>> ~[runtime-0.1.1.jar:0.1.1] >>>> at SerializedIterator.computeNext(SerializedIterator.kt:18) >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Cap'n Proto" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> Visit this group at https://groups.google.com/group/capnproto. >>>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> Visit this group at https://groups.google.com/group/capnproto. >> > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
