(I have this on the back burner as I'm kind of swamped, but I do want to get
this submitted at some point, hopefully within a week.)

On Tue, Jan 5, 2010 at 3:57 AM, Graham Cox <cox.gra...@gmail.com> wrote:

> I was saying the user *could* do that, and that it's currently what I'm
> doing in my server-side code. The reason being, as you said, if you naively
> read from a stream and the message isn't all present then you need to block
> until it is with the way that the Java code works at present. If you are
> using it for client-side code then likely this is not an issue in the
> slightest, but a server that needs to be able to handle many clients at once
> just can not block on one of them...
> As to your other alternative, (a), I would suggest that this leaves too
> much of the underlying network protocol bare to the caller. This will make
> it very difficult to change the way that delimiting messages happens in the
> future should such a thing be required. If - for example - it is decided to
> go from having the length prefixed to having a special delimiting sequence
> after the message then it will cause all current calling code to need to be
> changed. It might be that this is considered a low enough level library that
> this is acceptable, but that would be a Google decision...
> One more alternative would be how the asn1c library works for parsing ASN.1
> streams into objects, which is to be resumable. The decoder reads all the
> data it is given, and tries to build the object from this. If it doesn't
> have enough data yet then it does what it can, remembers where it got to and
> returns back to the user who can then supply more data when it becomes
> available. If the entire message does parse from the data provided then
> return back to the user the amount of data consumed so that they can discard
> this (reading from the stream directly makes this slightly cleaner still).
> At present, the Protobuf libraries (any of them) can not support this method
> of decoding an object, and it is not a trivial change to make it possible to
> do, but it does - IMO - give a much cleaner and easier to use method of use.
> --
> Graham Cox
> On Tue, Jan 5, 2010 at 1:32 AM, Kenton Varda <ken...@google.com> wrote:
>> Make sure to "reply all" so that the group is CC'd.
>> So you are saying that the user should read whatever data is on the
>> socket, then attempt to parse it, and if it fails, assume that it's because
>> there is more data to read?  Seems rather wasteful.  I think what we ideally
>> want is either:
>> (a) Provide a way for the caller to read the size independently, so that
>> they can then make sure to read that many bytes from the input before
>> parsing.
>> (b) Provide a method that reads from a stream, so that the protobuf
>> library can automatically take care of reading all necessary bytes.
>> Option (b) is obviously cleaner but has a few problems:
>> - We have to choose a particular stream interface to support.  While the
>> Python "file-like" interface is pretty common I'm not sure if it's universal
>> for this kind of task.
>> - If not all bytes of the message are available yet, we'd have to block.
>>  This might be fine most of the time, but would be unacceptable for some
>> uses.
>> Thoughts?
>> On Mon, Jan 4, 2010 at 3:09 PM, Graham Cox <cox.gra...@gmail.com> wrote:
>>> I'm using it for reading/writing to sockets in my functional tests -
>>> works well enough there...
>>> In my Java-side server code, I read from the socket into a byte buffer,
>>> then deserialize the byte buffer into Protobuf objects, throwing away the
>>> data that has been deserialized. The python "MergeDelimitedFromString"
>>> function also returns the number of bytes that were processed to build up
>>> the Protobuf object, so the user could easily do the same - read the socket
>>> onto the end of a buffer, and then while the buffer is successfully
>>> deserializing into objects throw away the first x bytes as appropriate...
>>> Just a thought :)
>>> On Mon, Jan 4, 2010 at 9:57 PM, Kenton Varda <ken...@google.com> wrote:
>>>> Hmm, it occurs to me that this currently is not useful for reading from
>>>> a socket or similar stream since the caller has to make sure to read an
>>>> entire message before trying to parse it, but the caller doesn't actually
>>>> know how long the message is (because the code that determines this is
>>>> encapsulated).  Any thoughts on this?
>>>> On Mon, Jan 4, 2010 at 12:11 PM, Kenton Varda <ken...@google.com>wrote:
>>>>> Mostly looks good.  There are some style issues (e.g. lines over 80
>>>>> chars) but I can clean those up myself.
>>>>> You'll need to sign the contributor license agreement:
>>>>> http://code.google.com/legal/individual-cla-v1.0.html -- If you own
>>>>> copyright on this change.
>>>>> http://code.google.com/legal/corporate-cla-v1.0.html -- If your
>>>>> employer does.
>>>>> Please let me know after you've done this and then I can submit these.
>>>>> On Fri, Jan 1, 2010 at 12:53 PM, Graham <cox.gra...@gmail.com> wrote:
>>>>>> On Jan 1, 7:32 am, Kenton Varda <ken...@google.com> wrote:
>>>>>> > I don't think an equivalent has been added to the Python API.  Want
>>>>>> to write
>>>>>> > up a patch?
>>>>>> Well - if you insist... Here's a first run, which seems to work but
>>>>>> I'm a very long way from a competent python programmers so feel free
>>>>>> to fix it up some :)
>>>>>> I can't see how to attach files using the google groups interface, so
>>>>>> I've stuck them on my webspace for now:
>>>>>> http://grahamcox.co.uk/patches/protobuf/
>>>>>> There's two patches - one for serializing in a delimited form, and one
>>>>>> for deserializing from a delimited form.
>>>>>> --
>>>>>> Graham Cox
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Protocol Buffers" group.
>>>>>> To post to this group, send email to proto...@googlegroups.com.
>>>>>> To unsubscribe from this group, send email to
>>>>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>>>>> .
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/protobuf?hl=en.
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Reply via email to