On Tue, Mar 15, 2011 at 06:05, ksamdev <samvel.khalat...@gmail.com> wrote:
> I like the interest in the topic.
> I've put 1GB to emphasize that the use case is safe. In fact, I save
> messages in file in next way:
> XYXYXYXYXY.....
> where X is the size of the message and Y is the message itself. Each message
> is read in the loop and overwritten. Clearly, I do not read the whole file
> (N GB's) into memory at once.
> Now, with this technique, I can generate files with size larger than 2^31 (~
> 2GB). The file is successfully written. Consider the case with 5 GB file.
> Unfortunately, whenever I start reading this 5 GB's file, ProtoBuf fails
> after 2^31 bytes are read. Of course, I have to push the limit of read bytes
> with:
>
> CodedInputStream::SetTotalBytesLimit(int, int)
> Pay attention at the arguments type: int . I suppose ProtoBuf uses bytes
> read counter or some internal file read position pointer that is also int
> and therefore fails whenever reading procedure passes the 2^31 threshold.

You should just create a new CodedInputStream on the stack for each
message, that way you don't run into this limit and can read files as
large as you want.
(CodedInputStream is cheap to create, so it shouldn't influence your
performance numbers).

> Thanks for the link to perftools. Like you mentioned, I reuse the message in
> my code. Therefore there is no overhead.
> I guess, the problem was in the way I measured execution time. My command
> looked like:
> time executable args && echo "-----" && time executable args
> So, I've cut it into 3 pieces and time, that is shown on the screen, start
> make sense:
> time executable args
> echo ------
> time executable args
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to