Hi everyone,
I'm looking for some guidance on what I may be doing wrong.
I'm serializing (unpacked) multiple MessageBuilder to FileChannel via a
BufferedOutputStreamWrapper
Here is a snippet of the code in Kotlin
val partitionFileInputStream = FileOutputStream(filename, true).channel
val buffered = BufferedOutputStreamWrapper(partitionFileInputStream)
val recordIterator = RecordIterator(dataSource)
recordIterator.asSequence().map { row ->
converter(row)
}.forEach { message ->
Serialize.write(buffered, message)
}
buffered.flush()
buffered.close()
I'm writing millions of records to a file which is several GB in size at the
end.
I then try to read the file:
val fileChannel = RandomAccessFile(filePath.toFile(), "r").getChannel()
for(message in SerializedIterator(fileChannel)) {
val record = message.getRoot(SomeClass.Object.factory)
}
here is the iterator implementation:
class SerializedIterator(readChan: ReadableByteChannel) :
AbstractIterator<MessageReader>(), AutoCloseable {
val buffer = BufferedInputStreamWrapper(readChan)
override fun close() {
buffer.close()
}
override fun computeNext() {
try {
setNext(Serialize.read(buffer))
} catch (e : Error) {
close()
done()
}
}
}
It seems to go fine for several million records and then I get hit with:
java.lang.NegativeArraySizeException: null
at org.capnproto.Serialize.read(Serialize.java:91)
~[runtime-0.1.1.jar:0.1.1]
at org.capnproto.Serialize.read(Serialize.java:51)
~[runtime-0.1.1.jar:0.1.1]
at SerializedIterator.computeNext(SerializedIterator.kt:18)
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
