Is MMAP the only way to randomly seek to an offset in the file?
I can't seem to find a way to do that with kj::FdInputStream ?
I'm trying to create an index of the elements in the file.
On Thursday, July 20, 2017 at 2:42:44 PM UTC-7, Kenton Varda wrote:
>
> Hi Farid,
>
> Try using mmap() (disclaimer: haven't tried compiling this):
>
> struct stat stats;
> KJ_SYSCALL(fstat(fd, &stats));
> size_t size = stats.st_size;
> const void* data = mmap(nullptr, size, PROT_READ, MAP_PRIVATE, fd, 0);
>
> if (data == MAP_FAILED) {
> KJ_FAIL_SYSCALL("mmap");
> }
>
> KJ_DEFER(KJ_SYSCALL(munmap(data, size)) { break; });
>
> KJ_SYSCALL(madvise(data, size, MADV_SEQUENTIAL));
>
>
> kj::ArrayPtr<capnp::word> words(
> reinterpret_cast<const capnp::word*>(data),
> size / sizeof(capnp::word));
>
> while (words.size() > 0) {
> capnp::FlatArrayMessageReader message(words);
> Message::Reader chunk = message.getRoot<Message>();
> count++;
> words = kj::arrayPtr(message.getEnd(), words.end());
> }
>
>
> (On Windows you'll need to use CreateFileMapping() and MapViewOfFile() or
> whatever.)
>
> -Kenton
>
> On Thu, Jul 20, 2017 at 2:11 PM, Farid Zakaria <[email protected]
> <javascript:>> wrote:
>
>> Looking for some guidance on possibly the fastest way to read multiple
>> messages in a file that has multiple MessageRoots
>>
>> So far I have this written:
>>
>> capnp::MallocMessageBuilder messageBuilder;
>> //What is a good size for our words? As long as its smaller?
>> capnp::word scratch[1024];
>> kj::ArrayPtr<capnp::word> scratchSpace(scratch);
>> kj::FdInputStream stream(fd);
>> kj::BufferedInputStreamWrapper buff(stream);
>>
>> unsigned long count = 0;
>> while (buff.tryGetReadBuffer().size() != 0) {
>> capnp::InputStreamMessageReader message(buff, capnp::ReaderOptions(),
>> scratchSpace);
>> Message::Reader chunk = message.getRoot<Message>();
>> count++;
>> }
>>
>>
>> I tried the helper methods first, but they seem too slow I think without the
>> BufferedInputStreamWrapper.
>>
>>
>> I don't do much C++ so I appreciate any help :)
>>
>> Btw, Is there an IRC chat or something ?
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> Visit this group at https://groups.google.com/group/capnproto.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
