[protobuf] Re: Streaming Serialization - Suggestion

[Yoav H]

Any comment on this?
Will you consider this for proto3?

On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote:
>
> Hi,
>
> I have a suggestion fr improving the protobuf encoding.
> Is proto3 final?
>
> I like the simplicity of the encoding of protobuf.
> But I think it has one issue with serialization, using streams.
> The problem is with length delimited fields and the fact that they require 
> knowing the length ahead of time.
> If we have a very long string, we need to encode the entire string before 
> we know its length, so we basically duplicate the data in memory.
> Same is true for embedded messages, where we need to encode the entire 
> embedded message before we can append it to the stream.
>
> I think there is a simple solution for both issues.
>
> For strings and byte arrays, a simple solution is to use "chunked 
> encoding".
> Which means that the byte array is split into chunks and every chunk 
> starts with the chunk length. End of array is indicated by length zero.
>
> For embedded messages, the solution is to have an "start embedding" tag 
> and an "end embedding tag".
> Everything in between is the embedded message.
>
> By adding these two new features, serialization can be fully streamable 
> and there is no need to pre-serialize big chunks in memory before writing 
> them to the stream.
>
> Hope you'll find this suggestion useful and incorporate it into the 
> protocol.
>
> Thanks,
> Yoav.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Ilia Mirkin]

Encoding is identical... just the API is different. In proto2, you
have (in C++) FooMessage->has_field() which will tell you whether a
field was present in the encoded version (or has been set prior if
you're building a new message). The Java API has something rather
similar... hasField() I think?

On Sat, Mar 26, 2016 at 4:00 PM, Yoav H  wrote:
> Thanks all,
>
> Do you know where I can find the proto2 encoding guide?
> The proto site has only the proto3 encoding described.
>
> On Saturday, March 26, 2016 at 12:21:39 PM UTC-7, Tim Kientzle wrote:
>>
>>
>> > On Mar 26, 2016, at 11:43 AM, Yoav H  wrote:
>> >
>> > Hi,
>> >
>> > I wanted ask regarding the decision to populate fields with default
>> > values, even if they do not appear in the encoded message.
>> > If I want to send a "patch" message, where I want to update just the
>> > provided fields, how can I do that with protobuf (without adding IsXXXSet
>> > for every field)?
>> >
>> > Why not add another type, representing a default value?
>> > So the schematics would be, if the field is missing, it is null, and if
>> > the field exists, but with this "missing value" type, it will get the
>> > default value?
>>
>> As Ilia pointed out, proto2 still exists, is still supported, and can be
>> used for
>> cases where you require these particular semantics.
>>
>> For proto3, you might look at google.protobuf.FieldMask, which is a new
>> standard message (one of the "well-known types") specifically designed
>> to store a set of field names.  You might be able to achieve what you want
>> by
>> providing a FieldMask with your data listing the specific fields to
>> be updated.
>>
>> Tim
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Yoav H]

Thanks all,

Do you know where I can find the proto2 encoding guide?
The proto site has only the proto3 encoding described.

On Saturday, March 26, 2016 at 12:21:39 PM UTC-7, Tim Kientzle wrote:
>
>
> > On Mar 26, 2016, at 11:43 AM, Yoav H  
> wrote: 
> > 
> > Hi, 
> > 
> > I wanted ask regarding the decision to populate fields with default 
> values, even if they do not appear in the encoded message. 
> > If I want to send a "patch" message, where I want to update just the 
> provided fields, how can I do that with protobuf (without adding IsXXXSet 
> for every field)? 
> > 
> > Why not add another type, representing a default value? 
> > So the schematics would be, if the field is missing, it is null, and if 
> the field exists, but with this "missing value" type, it will get the 
> default value? 
>
> As Ilia pointed out, proto2 still exists, is still supported, and can be 
> used for 
> cases where you require these particular semantics. 
>
> For proto3, you might look at google.protobuf.FieldMask, which is a new 
> standard message (one of the "well-known types") specifically designed 
> to store a set of field names.  You might be able to achieve what you want 
> by 
> providing a FieldMask with your data listing the specific fields to 
> be updated. 
>
> Tim 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Tim Kientzle]

> On Mar 26, 2016, at 11:43 AM, Yoav H  wrote:
> 
> Hi,
> 
> I wanted ask regarding the decision to populate fields with default values, 
> even if they do not appear in the encoded message.
> If I want to send a "patch" message, where I want to update just the provided 
> fields, how can I do that with protobuf (without adding IsXXXSet for every 
> field)?
> 
> Why not add another type, representing a default value? 
> So the schematics would be, if the field is missing, it is null, and if the 
> field exists, but with this "missing value" type, it will get the default 
> value?

As Ilia pointed out, proto2 still exists, is still supported, and can be used 
for
cases where you require these particular semantics.

For proto3, you might look at google.protobuf.FieldMask, which is a new
standard message (one of the "well-known types") specifically designed
to store a set of field names.  You might be able to achieve what you want by
providing a FieldMask with your data listing the specific fields to
be updated.

Tim


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Ilia Mirkin]

Use proto2, which has the has_* checks per field. (Using get_* you
still get the default value, of course.) It's extremely unfortunate
that this functionality was removed in proto3, I see that making
proto3 unattractive for all but the simplest uses of protos. I know in
almost every protobuf use-case I've had, the presence accessors were
imperative to proper operation.

On Sat, Mar 26, 2016 at 2:43 PM, Yoav H  wrote:
> Hi,
>
> I wanted ask regarding the decision to populate fields with default values,
> even if they do not appear in the encoded message.
> If I want to send a "patch" message, where I want to update just the
> provided fields, how can I do that with protobuf (without adding IsXXXSet
> for every field)?
>
> Why not add another type, representing a default value?
> So the schematics would be, if the field is missing, it is null, and if the
> field exists, but with this "missing value" type, it will get the default
> value?
>
> Thanks,
> Yoav.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Default Values vs Missing Values

[Yoav H]

Hi,

I wanted ask regarding the decision to populate fields with default values, 
even if they do not appear in the encoded message.
If I want to send a "patch" message, where I want to update just the 
provided fields, how can I do that with protobuf (without adding IsXXXSet 
for every field)?

Why not add another type, representing a default value? 
So the schematics would be, if the field is missing, it is null, and if the 
field exists, but with this "missing value" type, it will get the default 
value?

Thanks,
Yoav. 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.