On Tue, Mar 15, 2011 at 10:09, Samvel Khalatyan
<samvel.khalat...@gmail.com> wrote:
> Thanks for the clarification. Does it mean that my code should look like the
> example below?
> fstream in("10_GB_file.pb", ios::in | ios::binary);
> bool continue_reading = true
> while(continue_reading)
> {
>   ::google::protobuf::io::IstreamInputStream raw_in(&in);

The ::google::protobuf::io::IstreamInputStream raw_in can probably be
out of the loop. I haven't looked at the code closely, but I think it
is not limited (except an int64 offset, but that is right now big
enough...).

Just try it.

>   ::google::protobuf::io::CodedInputStream coded_in(&raw_in);

... and this one should be in the loop, yes.

>   // Read message and use it if read was successfull
>   if (! /* are there more messages left? */)
>     continue_reading = false;
> }
> On Tue, Mar 15, 2011 at 11:48 AM, Henner Zeller
> <henner.zel...@googlemail.com> wrote:
>>
>> On Tue, Mar 15, 2011 at 06:05, ksamdev <samvel.khalat...@gmail.com> wrote:
>> > I like the interest in the topic.
>> > I've put 1GB to emphasize that the use case is safe. In fact, I save
>> > messages in file in next way:
>> > XYXYXYXYXY.....
>> > where X is the size of the message and Y is the message itself. Each
>> > message
>> > is read in the loop and overwritten. Clearly, I do not read the whole
>> > file
>> > (N GB's) into memory at once.
>> > Now, with this technique, I can generate files with size larger than
>> > 2^31 (~
>> > 2GB). The file is successfully written. Consider the case with 5 GB
>> > file.
>> > Unfortunately, whenever I start reading this 5 GB's file, ProtoBuf fails
>> > after 2^31 bytes are read. Of course, I have to push the limit of read
>> > bytes
>> > with:
>> >
>> > CodedInputStream::SetTotalBytesLimit(int, int)
>> > Pay attention at the arguments type: int . I suppose ProtoBuf uses bytes
>> > read counter or some internal file read position pointer that is also
>> > int
>> > and therefore fails whenever reading procedure passes the 2^31
>> > threshold.
>>
>> You should just create a new CodedInputStream on the stack for each
>> message, that way you don't run into this limit and can read files as
>> large as you want.
>> (CodedInputStream is cheap to create, so it shouldn't influence your
>> performance numbers).
>>
>> > Thanks for the link to perftools. Like you mentioned, I reuse the
>> > message in
>> > my code. Therefore there is no overhead.
>> > I guess, the problem was in the way I measured execution time. My
>> > command
>> > looked like:
>> > time executable args && echo "-----" && time executable args
>> > So, I've cut it into 3 pieces and time, that is shown on the screen,
>> > start
>> > make sense:
>> > time executable args
>> > echo ------
>> > time executable args
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To post to this group, send email to protobuf@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > protobuf+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/protobuf?hl=en.
>> >
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to