Once I learned that Messages are not self-delimiting (thanks, Kenton!), I
started working with Hadoop's source to stop the trailing bits from being
included in the InputStream. I've since fixed this issue, kind of at least
Perhaps a good general solution is to allow a user to put an option in a
.proto file or a Message declaration that makes Messages self-delimiting.
That way users who want speed don't need to us it, and users who want
convenience can use it. The implementation of this would probably be
tricky, I'm sure.
Thanks for the follow up, Chris. For now I'm good to go! Let me know if I
can provide any other feedback.
On Thu, Sep 11, 2008 at 4:18 PM, Chris <[EMAIL PROTECTED]> wrote:
> Hi Alex,
> Kenton Varda wrote:
> On Mon, Sep 8, 2008 at 9:11 PM, Alex Loddengaard <
>> [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>> I have a follow-up question:
>> Will using
>> work for a stream that contains trailing binary information?
>> No, it won't work. Protocol buffers are not self-delimiting. They assume
>> that the input you provide is supposed to be one complete message, not a
>> message possibly followed by other stuff.
>> You will need to somehow communicate the size of the message and make sure
>> to limit the input to that size.
> Aha. This <binary>message<binary> case is one of the heretofore
> hypothetical use cases I am discussing in the adjacent thread on this
> mailing list / group. The thread is online at
> and was spawned from
> This is mainly myself, Jon, and Kenton slowly forming a consensus on the
> right API for delimited messages. I had proposed simply adding the length
> (varint) before the message, and Kenton demonstrated c++ code for this. Jon
> proposed adding a field number / wiretype tag before the length and message,
> which makes it look much more like a protocol-buffer field on the wire.
> What do you need Alex?
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to email@example.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at