Thanks for the clarification. Does it mean that my code should look like the
example below?
fstream in("10_GB_file.pb", ios::in | ios::binary);
bool continue_reading = true
while(continue_reading)
{
::google::protobuf::io::IstreamInputStream raw_in(&in);
::google::protobuf::io::CodedInputStream coded_in(&raw_in);
// Read message and use it if read was successfull
if (! /* are there more messages left? */)
continue_reading = false;
}
On Tue, Mar 15, 2011 at 11:48 AM, Henner Zeller <
henner.zel...@googlemail.com> wrote:
> On Tue, Mar 15, 2011 at 06:05, ksamdev <samvel.khalat...@gmail.com> wrote:
> > I like the interest in the topic.
> > I've put 1GB to emphasize that the use case is safe. In fact, I save
> > messages in file in next way:
> > XYXYXYXYXY.....
> > where X is the size of the message and Y is the message itself. Each
> message
> > is read in the loop and overwritten. Clearly, I do not read the whole
> file
> > (N GB's) into memory at once.
> > Now, with this technique, I can generate files with size larger than 2^31
> (~
> > 2GB). The file is successfully written. Consider the case with 5 GB file.
> > Unfortunately, whenever I start reading this 5 GB's file, ProtoBuf fails
> > after 2^31 bytes are read. Of course, I have to push the limit of read
> bytes
> > with:
> >
> > CodedInputStream::SetTotalBytesLimit(int, int)
> > Pay attention at the arguments type: int . I suppose ProtoBuf uses bytes
> > read counter or some internal file read position pointer that is also int
> > and therefore fails whenever reading procedure passes the 2^31 threshold.
>
> You should just create a new CodedInputStream on the stack for each
> message, that way you don't run into this limit and can read files as
> large as you want.
> (CodedInputStream is cheap to create, so it shouldn't influence your
> performance numbers).
>
> > Thanks for the link to perftools. Like you mentioned, I reuse the message
> in
> > my code. Therefore there is no overhead.
> > I guess, the problem was in the way I measured execution time. My command
> > looked like:
> > time executable args && echo "-----" && time executable args
> > So, I've cut it into 3 pieces and time, that is shown on the screen,
> start
> > make sense:
> > time executable args
> > echo ------
> > time executable args
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to protobuf@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/protobuf?hl=en.
> >
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
