Hi Kenton,

For the problem I observed, the Refresh() call hang on a message of 2
bytes. I'll try to reproduce a small example and get back to you.


On Fri, Sep 18, 2009 at 1:53 PM, Kenton Varda <ken...@google.com> wrote:
> That Refresh() call should only block if there are no bytes at all available
> on the stream.  But if you're about to read a message, then you expect there
> to be some bytes, right?  Or is it that you're actually receiving a message
> of zero size?  In that case, you could check if the message size is zero
> before calling ParseFromBoundedZeroCopyStream() and skip the call if so.
>  Arguably, ParseFromBoundedZeroCopyStream() should itself check for
> zero-size messages and return immediately in this case, without creating a
> CodedInputStream -- I would accept a patch that makes this change.
> On Fri, Sep 18, 2009 at 9:56 AM, Martin Bravenboer
> <martin.bravenb...@gmail.com> wrote:
>> Hi all,
>> In case you enjoy success stories: at LogicBlox we have recently
>> started using Protocol Buffers for a protocol between our Datalog
>> database server (written in C++) and our Datalog compiler (written in
>> Java). We use protobuf in two different configurations: over sockets
>> between separate processes, but also for communication via JNI inside
>> the same process. Amusingly, the use of protobuf was a performance
>> improvement over our earlier JNI-based implementation, where we
>> created Java objects directly from C++ using JNI calls. Apparently,
>> the serialization overhead of Protocol Buffers was lower than the
>> overhead of the JNI calls! This probably does not apply in general,
>> but at least for us it did. Thanks to all the developers of protobuf!
>> We encountered one issue that might be interesting to others.
>> We send sequences of protobuf messages over a socket. Some of these
>> messages might be very small. The size of the message is indicated by
>> a simple header that precedes the serialized protobuf message.
>> The problem we had was that there is no way to read a limited number
>> of bytes from the socket input stream on the C++ side. We tried
>> several alternatives. The most attractive one would be:
>> --------------------------------
>>  google::protobuf::io::IstreamInputStream zistream(&io);
>>  if(!msg.ParseFromBoundedZeroCopyStream(&zistream, size))
>> ---------------------------------
>> ParseFromBounded is implemented as:
>> -----------------------------------
>>  io::CodedInputStream decoder(input);
>>  decoder.PushLimit(size);
>> -----------------------------------
>> Unfortunately, this hangs in the constructor of CodedInputStream
>> because insufficient bytes are available to make the call to Refresh
>> in the constructor of CodedInputStream terminate:
>> -----------------------------------
>>  // Eagerly Refresh() so buffer space is immediately available.
>>  Refresh();
>> -----------------------------------
>> So, as far as I know there is currently no way to read a limited
>> number of bytes from a stream when the remaining bytes on that stream
>> are not yet available. We resorted to reading the message in a uint8*
>> buffer separately, and parse the message from that buffer.
>> Two emails in the mail archive of this mailing list suggest that using
>> PushLimit should work, so it seems this issue is not widely known.
>> http://markmail.org/message/fvmubiw5ihwge7wt
>> http://markmail.org/message/sdjovyr5ng6tjgpm
>> Thanks again for all the work!
>> --
>> Martin Bravenboer
>> LogicBlox
>> >>

Martin Bravenboer

You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
For more options, visit this group at 

Reply via email to