Thanks very much Jon (see below).  You make good points and I like the
approach the you describe.  I am still thinking, however, that there is
power in the ability for message instances to write and parse themselves
from a stream.

A message instance could be passed a stream object which chains back to
the network connection from which bytes are being received.  A stop flag
based parsing mechanism could be passed this buffer object, and would
handle reading the stream and initializing its properties, exiting when
the serialization of that message instance stopped.  At this point, a
new message instance could be created, and the process repeated.  

The type of message doing the parsing could vary from message to
message, even with the serializations being sent and received back to
back.  This mechanism would work regardless of field-types being
streamed.  A message type consisting solely of varint fields, whose
length is determined while reading the varint's value, would support
streaming no differently than any other message type.

The solution also seems to support every requirement supported by the
original buffer type.  Messages serialized to a buffer, could just as
easily be initialized from that buffer as they could from the string
contained by the buffer.

m1 = Message()
buffer = Buffer()
[...] (initialize instance vars)

m2 = Message()

Produces same result as: 

m2 = Message()
bytes = m1.SerializeToString()

The string-based parse would ignore the stop bit, parsing the entire
string.  The buffer-based parsing would stop parsing when the stop bit,
producing the same result.

Handling of concatenated serializations is supported through repeated
calls to parse from buffer:

m1 = Message()
[...] (initialize instance vars)
m2 = Message()
[...] (initialize instance vars)

buffer = Buffer()

m3 = Message()

Would produce same result as:

m3 = Message()
m3.ParseFromString(m1.SerializeToString() + m2.SerializeToString())

As long as an unused, and never to be used, field number is used to
generate the stop bit's key, then I don't believe there are any
incompatibilities between buffer-based message marshalling and the
existing string-based code.  A very easy usage:

# Sending side
for message in messages:

# Receiving side
for msgtype in types:
  message = msgtype()

Unless I've overlooked something, it seems like the stream based
marshalling and unmarshalling is powerful, simple, and completely
compatible with all existing code.  But there is a very real chance I've
overlooked something...

- Shane

-------- Forwarded Message --------
> From: Jon Skeet <[EMAIL PROTECTED]>
> To: Shane Green <[EMAIL PROTECTED]>
> Subject: Re: Streaming
> Date: Fri, 5 Dec 2008 08:19:41 +0000
> 2008/12/5 Shane Green <[EMAIL PROTECTED]>
>         Thanks Jon.  Those are good points.  I rather liked the
>         self-delimiting
>         nature of fields, and thought this method would bring that
>         feature up to
>         the message level, without breaking any of the existing
>         capabilities.
>         So my goal was a message which could truly be streamed;
>         perhaps even
>         sent without knowing its own size up front.  Perhaps I
>         overlooked
>         something?
> Currently the PB format requires that you know the size of each
> submessage before you send it. You don't need to know the size of the
> whole message, as it's assumed to be the entire size of the
> datastream. It's unfortunate that you do need to provide the whole
> message to the output stream though, unless you want to manually
> serialize the individual fields.
> My goal was slightly different - I wanted to be able to stream a
> sequence of messages. The most obvious use case (in my view) is a log.
> Write out a massive log file as a sequence of entries, and you can
> read it back in one at a time. It's not designed to help to stream a
> single huge message though.
>         Would you mind if I resent my questions to the group?  I lack
>         confidence and wanted to make sure I wasn't overlooking
>         something
>         ridiculous, but am thinking that the exchange would be
>         informative.
> Absolutely. Feel free to quote anything I've written if you think it
> helps.
>         Also, how are you serializing and parsing messages as if they
>         are
>         repeated fields of a container message?  Is there a fair bit
>         of parsing
>         or work being done outside the standard protocol-buffer APIs?
> There's not a lot of work, to be honest. On the parsing side the main
> difficulty is getting a type-safe delegate to read a message from the
> stream. The writing side is trivial. Have a look at the code:
> There may have been some trivial changes to CodedInputStream - I can't
> remember offhand.
> Hope this helps,
> Jon

You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at

Reply via email to