Hi Tao, You can get random access to files on disk by memory mapping the file. In Java, you would use FileChannel.map() to get a MappedByteBuffer. You can then pass that ByteBuffer off to Cap'n Proto and use it like any other ByteBuffer. The operating system will not actually read in the data from disk until your program attempts to access the corresponding part of the MappedByteBuffer, which Cap'n Proto will only do when you invoke the accessor for a field located there. So, somewhat magically, you get random access.
Unfortunately, you cannot get random access to compressed data this way, unless the compression is implemented inside the OS / filesystem. (And most compression methods are not random-access-friendly anyhow.) -Kenton On Tue, Dec 5, 2017 at 10:38 AM, <[email protected]> wrote: > Hi, > > I am working on a project which is using protobuf to encode/decode > messages. I am evaluating if it is worth to migrate to Cap'N proto. I am > using the Java implementation of Cap'N. https://github.com/capn > proto/capnproto-java > > From the documentation, https://capnproto.org/index.html, Random access > is mentioned as a key feature. But I am not able to find any piece of code > example to demonstrate this feature. Am I misunderstanding it? Does "random > access" simply means we can access any field without "deserializing" the > whole message (it actually not serialized at all if not packed)? > > What I thought about "random access" is Cap'N is able to read any field > back from disk without loading the whole bunch of message data to memory. > But from the java API implementation (the source code), it seems that it > always read the whole message back to byte buffer, getRoot and then access > any field. So, I guess my understanding is wrong, isn't it? > > Our scenario: > Our current protobuf message schema has many fields (~100) with embedded > other messages. The serialized message size varies from hundreds bytes to > tens of kilobytes and a few large messages may over 1 megabytes. We store > the messages in term of compressed byte array to underlying KV store and > read back from KV store, uncompress and then parse to protobuf object. > > In this case, do you think it is worth to migrate from protobuf to cap'N ? > If so, how can I benefit from "random access" feature? > > > Thanks, > Tao > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
