Re: [protobuf] can protobuf3 be used with protobuf2?

[Jeremy Ong]

Why does adding JSON require dropping unknown fields? So long as fields are
keyed to field number, I don't see why the JSON encoding requires special
treatment with respect to the binary one.

I can understand how transitioning between major versions may require
breaks in compatibility. However, proto3 makes breaks in compatibility with
the underlying data (proto2 encoded), which is where I find myself in
disagreement. Why not preserve data compatibility so that overtime, proto2
users can migrate? Unknown field handling (or lack thereof) is honestly the
one I find most egregious.

On Wed, May 18, 2016 at 9:51 PM, Tim Kientzle  wrote:

> After studying proto3 pretty carefully, I’ve come around quite a bit on
> these changes:
>
> I believe adding JSON requires dropping unknown fields.  You simply cannot
> preserve unknown fields and properly support multiple encodings.
>
> I’m less sure about replacing extension support with Any.  Extensions have
> some ugly problems, but I feel the current spec for Any also has some real
> drawbacks.
>
> Removing field presence is a subtle issue, but I’m starting to suspect it
> was actually a very good change.  It reduces the generated code and the
> workaround of using a single-element oneof is cleaner than it might sound.
> In essence, a single-element oneof is just a way to explicitly declare that
> you want to track presence for that field.  And oneof is supported by
> proto2 now, so you can use that technique there as well.
>
> Finally, remember that proto2 is not going away:   If proto2 assumptions
> are deeply baked into your systems, you can keep using it.  protoc will
> continue to support it for a very long time.
>
> Cheers,
>
> Tim
>
>
>
> > On May 18, 2016, at 1:33 PM, Jeremy Ong  wrote:
> >
> > Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1
> because I love the removal of required fields, hate the removal of field
> presence. All the changes I dislike are significant losses in functionality
> and break compatibility with existing users of proto2 and I'd be interested
> to understand why "ease of implementation" is good justification for this
> break in compatibility and what I perceive to be a loss in functionality.
> >
> > On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers <
> protobuf@googlegroups.com> wrote:
> >
> >
> > On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov 
> wrote:
> > +1
> > Yes, a checklist would be extremely helpful.
> >
> >
> > On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi
> wrote:
> > It would be nice to have a migration guide (checklist) somewhere, like:
> >
> > 1. All fields should be optional.
> > 2. Do not use custom defailt values.
> > 3. All enums should have first element with tag = 0.
> > 4. Do not use extension for anything except custom options.
> >
> > Something else?
> > In 3.0.0-alpha-1 release note there is a list of main proto3 changes:
> > The following are the main new features in language version 3:
> >
> >   • Removal of field presence logic for primitive value fields,
> removal of required fields, and removal of default values. This makes
> proto3 significantly easier to implement with open struct representations,
> as in languages like Android Java, Objective C, or Go.
> >   • Removal of unknown fields.
> >   • Removal of extensions, which are instead replaced by a new
> standard type called Any.
> >   • Fix semantics for unknown enum values.
> >   • Addition of maps.
> >   • Addition of a small set of standard types for representation of
> time, dynamic data, etc.
> >   • A well-defined encoding in JSON as an alternative to binary
> proto encoding.
> >
> >
> >
> >
> > On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
> >
> >
> > On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
> > suppose server side is updating into protobuf3, but client side still
> use protobuf2, can then communicate will?
> > Yes, as long as you only use proto3 features, they are wire compatible.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> > To post to this group, send email to prot...@googlegroups.com.
> > Visit this group at https://groups.google.com/group/protobuf.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+unsubscr...@googlegroups.com.
> > To post to this group, send email to protobuf@googlegroups.com.
> > Visit this group at https://groups.google.com/group/protobuf.
> > For more options, visit https://groups.google.com/d/optout.
> 

Re: [protobuf] can protobuf3 be used with protobuf2?

[Tim Kientzle]

After studying proto3 pretty carefully, I’ve come around quite a bit on these 
changes:

I believe adding JSON requires dropping unknown fields.  You simply cannot 
preserve unknown fields and properly support multiple encodings.

I’m less sure about replacing extension support with Any.  Extensions have some 
ugly problems, but I feel the current spec for Any also has some real drawbacks.

Removing field presence is a subtle issue, but I’m starting to suspect it was 
actually a very good change.  It reduces the generated code and the workaround 
of using a single-element oneof is cleaner than it might sound.  In essence, a 
single-element oneof is just a way to explicitly declare that you want to track 
presence for that field.  And oneof is supported by proto2 now, so you can use 
that technique there as well.

Finally, remember that proto2 is not going away:   If proto2 assumptions are 
deeply baked into your systems, you can keep using it.  protoc will continue to 
support it for a very long time.

Cheers,

Tim



> On May 18, 2016, at 1:33 PM, Jeremy Ong  wrote:
> 
> Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1 because I 
> love the removal of required fields, hate the removal of field presence. All 
> the changes I dislike are significant losses in functionality and break 
> compatibility with existing users of proto2 and I'd be interested to 
> understand why "ease of implementation" is good justification for this break 
> in compatibility and what I perceive to be a loss in functionality.
> 
> On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers 
>  wrote:
> 
> 
> On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov  wrote:
> +1
> Yes, a checklist would be extremely helpful. 
> 
> 
> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi 
> wrote:
> It would be nice to have a migration guide (checklist) somewhere, like:
> 
> 1. All fields should be optional.
> 2. Do not use custom defailt values.
> 3. All enums should have first element with tag = 0.
> 4. Do not use extension for anything except custom options.
> 
> Something else?
> In 3.0.0-alpha-1 release note there is a list of main proto3 changes:
> The following are the main new features in language version 3:
> 
>   • Removal of field presence logic for primitive value fields, removal 
> of required fields, and removal of default values. This makes proto3 
> significantly easier to implement with open struct representations, as in 
> languages like Android Java, Objective C, or Go.
>   • Removal of unknown fields.
>   • Removal of extensions, which are instead replaced by a new standard 
> type called Any.
>   • Fix semantics for unknown enum values.
>   • Addition of maps.
>   • Addition of a small set of standard types for representation of time, 
> dynamic data, etc.
>   • A well-defined encoding in JSON as an alternative to binary proto 
> encoding.
> 
> 
>  
> 
> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
> 
> 
> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
> suppose server side is updating into protobuf3, but client side still use 
> protobuf2, can then communicate will?
> Yes, as long as you only use proto3 features, they are wire compatible.
>  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> -- 
> Jeremy Ong
> PlexChat CTO
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to 

Re: [protobuf] can protobuf3 be used with protobuf2?

[Jeremy Ong]

Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1 because
I love the removal of required fields, hate the removal of field presence.
All the changes I dislike are significant losses in functionality and break
compatibility with existing users of proto2 and I'd be interested to
understand why "ease of implementation" is good justification for this
break in compatibility and what I perceive to be a loss in functionality.

On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

>
>
> On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov  wrote:
>
>> +1
>> Yes, a checklist would be extremely helpful.
>>
>>
>> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi
>> wrote:
>>>
>>> It would be nice to have a migration guide (checklist) somewhere, like:
>>>
>>> 1. All fields should be optional.
>>> 2. Do not use custom defailt values.
>>> 3. All enums should have first element with tag = 0.
>>> 4. Do not use extension for anything except custom options.
>>>
>>> Something else?
>>>
>> In 3.0.0-alpha-1 release note
>  there is
> a list of main proto3 changes:
>
> The following are the main new features in language version 3:
>
>1. Removal of field presence logic for primitive value fields, removal
>of required fields, and removal of default values. This makes proto3
>significantly easier to implement with open struct representations, as in
>languages like Android Java, Objective C, or Go.
>2. Removal of unknown fields.
>3. Removal of extensions, which are instead replaced by a new standard
>type called Any.
>4. Fix semantics for unknown enum values.
>5. Addition of maps.
>6. Addition of a small set of standard types for representation of
>time, dynamic data, etc.
>7. A well-defined encoding in JSON as an alternative to binary proto
>encoding.
>
>
>
>
>
>>
>>> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:



 On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:

> suppose server side is updating into protobuf3, but client side still
> use protobuf2, can then communicate will?
>
 Yes, as long as you only use proto3 features, they are wire compatible.


> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

 --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Jeremy Ong
PlexChat CTO

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Teddy Zhang]

On Wednesday, May 18, 2016 at 11:32:03 AM UTC-7, Feng Xiao wrote:
>
>
>
> On Tue, May 17, 2016 at 7:53 PM, Teddy Zhang  > wrote:
>
>> I'm really not happy to see that proto3 removed the ability in generate 
>> code for check whether a field exits or not.
>>
>> For a message like this:
>> message Test1 {
>>   required int32 a = 1;
>> }
>> If field a is present, the encoded message will have field with id 1 and 
>> its value. If the field is not set, the encoded message will not have field 
>> id 1.
>> In proto2 generated code, it provides a has method to check whether the 
>> field exists or not.
>> In proto3, this is no such thing. During deserialization, if the field is 
>> not exists, default value is set. So you can't tell whether the field does 
>> not exist or have a default value. That doesn't match the underline 
>> encoding anymore.
>>
>> This is a breaking change and will portentially impact a lot of people. 
>> Basically we're losing nullable support.
>> For our project, we heavily depends on that. There are workarounds (add 
>> a Boolean field) but it is ugly. I think that will stop us from moving from 
>> proto2 to proto3 (may need find alternatives).
>>
> There are two workarounds to get back the field presence info in proto3.
> 1. Use a wrapper message, such as google.protobuf.Int32Value 
> .
>  
> In proto3, message fields still have has-bits.
>
Wrapper field consumes more space. Also, the wire format is not compatible 
when move from proto2 to proto3 (given the schema needs to change).
 

> 2. Use an oneof. For example:
> message Test1 {
>   oneof a_oneof {
> int32 a = 1;
>   }
> }
> then you can check test.getAOneofCase().
>
Same issue as above.  

>
>
>
>> Can we add the functionality back?
>>
> It's very unlikely to happen as proto3 features are already finalized and 
> implemented in many languages.
>
Is it possible to add a option on message to control this?
I know proto3 is probably in last beta and try to avoid big changes. 
However, remove support for this creates a lot of pain in a big system 
which already leverage this feature, and may move many people away. 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Teddy Zhang]

On Wednesday, May 18, 2016 at 11:32:03 AM UTC-7, Feng Xiao wrote:
>
>
>
> On Tue, May 17, 2016 at 7:53 PM, Teddy Zhang  > wrote:
>
>> I'm really not happy to see that proto3 removed the ability in generate 
>> code for check whether a field exits or not.
>>
>> For a message like this:
>> message Test1 {
>>   required int32 a = 1;
>> }
>> If field a is present, the encoded message will have field with id 1 and 
>> its value. If the field is not set, the encoded message will not have field 
>> id 1.
>> In proto2 generated code, it provides a has method to check whether the 
>> field exists or not.
>> In proto3, this is no such thing. During deserialization, if the field is 
>> not exists, default value is set. So you can't tell whether the field does 
>> not exist or have a default value. That doesn't match the underline 
>> encoding anymore.
>>
>> This is a breaking change and will portentially impact a lot of people. 
>> Basically we're losing nullable support.
>> For our project, we heavily depends on that. There are workarounds (add 
>> a Boolean field) but it is ugly. I think that will stop us from moving from 
>> proto2 to proto3 (may need find alternatives).
>>
> There are two workarounds to get back the field presence info in proto3.
> 1. Use a wrapper message, such as google.protobuf.Int32Value 
> .
>  
> In proto3, message fields still have has-bits.
>
Wrapper field consumes more space. Also, the wire format is not compatible 
when move from proto2 to proto3 (given the schema needs to change).
 

> 2. Use an oneof. For example:
> message Test1 {
>   oneof a_oneof {
> int32 a = 1;
>   }
> }
> then you can check test.getAOneofCase().
>
Same issue as above. 


>
>> Can we add the functionality back?
>>
> It's very unlikely to happen as proto3 features are already finalized and 
> implemented in many languages.
>
Is it possible to add a option on message to control this?
I know proto3 is probably in last beta and try to avoid big changes. 
However, remove support for this creates a lot of pain in a big 
system which already leverage this feature, and may move many people away.

>  
>
>>
>>
>> On Saturday, March 26, 2016 at 11:47:08 AM UTC-7, Ilia Mirkin wrote:
>>
>>> Use proto2, which has the has_* checks per field. (Using get_* you 
>>> still get the default value, of course.) It's extremely unfortunate 
>>> that this functionality was removed in proto3, I see that making 
>>> proto3 unattractive for all but the simplest uses of protos. I know in 
>>> almost every protobuf use-case I've had, the presence accessors were 
>>> imperative to proper operation. 
>>>
>>> On Sat, Mar 26, 2016 at 2:43 PM, Yoav H  wrote: 
>>> > Hi, 
>>> > 
>>> > I wanted ask regarding the decision to populate fields with default 
>>> values, 
>>> > even if they do not appear in the encoded message. 
>>> > If I want to send a "patch" message, where I want to update just the 
>>> > provided fields, how can I do that with protobuf (without adding 
>>> IsXXXSet 
>>> > for every field)? 
>>> > 
>>> > Why not add another type, representing a default value? 
>>> > So the schematics would be, if the field is missing, it is null, and 
>>> if the 
>>> > field exists, but with this "missing value" type, it will get the 
>>> default 
>>> > value? 
>>> > 
>>> > Thanks, 
>>> > Yoav. 
>>> > 
>>> > -- 
>>> > You received this message because you are subscribed to the Google 
>>> Groups 
>>> > "Protocol Buffers" group. 
>>> > To unsubscribe from this group and stop receiving emails from it, send 
>>> an 
>>> > email to protobuf+u...@googlegroups.com. 
>>> > To post to this group, send email to prot...@googlegroups.com. 
>>> > Visit this group at https://groups.google.com/group/protobuf. 
>>> > For more options, visit https://groups.google.com/d/optout. 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

['Feng Xiao' via Protocol Buffers]

On Tue, May 17, 2016 at 7:53 PM, Teddy Zhang  wrote:

> I'm really not happy to see that proto3 removed the ability in generate
> code for check whether a field exits or not.
>
> For a message like this:
> message Test1 {
>   required int32 a = 1;
> }
> If field a is present, the encoded message will have field with id 1 and
> its value. If the field is not set, the encoded message will not have field
> id 1.
> In proto2 generated code, it provides a has method to check whether the
> field exists or not.
> In proto3, this is no such thing. During deserialization, if the field is
> not exists, default value is set. So you can't tell whether the field does
> not exist or have a default value. That doesn't match the underline
> encoding anymore.
>
> This is a breaking change and will portentially impact a lot of people.
> Basically we're losing nullable support.
> For our project, we heavily depends on that. There are workarounds (add
> a Boolean field) but it is ugly. I think that will stop us from moving from
> proto2 to proto3 (may need find alternatives).
>
There are two workarounds to get back the field presence info in proto3.
1. Use a wrapper message, such as google.protobuf.Int32Value
.
In proto3, message fields still have has-bits.
2. Use an oneof. For example:
message Test1 {
  oneof a_oneof {
int32 a = 1;
  }
}
then you can check test.getAOneofCase().



> Can we add the functionality back?
>
It's very unlikely to happen as proto3 features are already finalized and
implemented in many languages.


>
>
> On Saturday, March 26, 2016 at 11:47:08 AM UTC-7, Ilia Mirkin wrote:
>
>> Use proto2, which has the has_* checks per field. (Using get_* you
>> still get the default value, of course.) It's extremely unfortunate
>> that this functionality was removed in proto3, I see that making
>> proto3 unattractive for all but the simplest uses of protos. I know in
>> almost every protobuf use-case I've had, the presence accessors were
>> imperative to proper operation.
>>
>> On Sat, Mar 26, 2016 at 2:43 PM, Yoav H  wrote:
>> > Hi,
>> >
>> > I wanted ask regarding the decision to populate fields with default
>> values,
>> > even if they do not appear in the encoded message.
>> > If I want to send a "patch" message, where I want to update just the
>> > provided fields, how can I do that with protobuf (without adding
>> IsXXXSet
>> > for every field)?
>> >
>> > Why not add another type, representing a default value?
>> > So the schematics would be, if the field is missing, it is null, and if
>> the
>> > field exists, but with this "missing value" type, it will get the
>> default
>> > value?
>> >
>> > Thanks,
>> > Yoav.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf3 be used with protobuf2?

['Feng Xiao' via Protocol Buffers]

On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov  wrote:

> +1
> Yes, a checklist would be extremely helpful.
>
>
> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi
> wrote:
>>
>> It would be nice to have a migration guide (checklist) somewhere, like:
>>
>> 1. All fields should be optional.
>> 2. Do not use custom defailt values.
>> 3. All enums should have first element with tag = 0.
>> 4. Do not use extension for anything except custom options.
>>
>> Something else?
>>
> In 3.0.0-alpha-1 release note
 there is a
list of main proto3 changes:

The following are the main new features in language version 3:

   1. Removal of field presence logic for primitive value fields, removal
   of required fields, and removal of default values. This makes proto3
   significantly easier to implement with open struct representations, as in
   languages like Android Java, Objective C, or Go.
   2. Removal of unknown fields.
   3. Removal of extensions, which are instead replaced by a new standard
   type called Any.
   4. Fix semantics for unknown enum values.
   5. Addition of maps.
   6. Addition of a small set of standard types for representation of time,
   dynamic data, etc.
   7. A well-defined encoding in JSON as an alternative to binary proto
   encoding.





>
>> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
>>>
>>>
>>>
>>> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
>>>
 suppose server side is updating into protobuf3, but client side still
 use protobuf2, can then communicate will?

>>> Yes, as long as you only use proto3 features, they are wire compatible.
>>>
>>>
 --
 You received this message because you are subscribed to the Google
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at https://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf3 be used with protobuf2?

[osuciu]

Protobuf schema evolution rules:
https://developers.google.com/protocol-buffers/docs/proto#updating


On Wednesday, May 18, 2016 at 11:09:15 AM UTC-7, Artem Kazakov wrote:
>
> +1
> Yes, a checklist would be extremely helpful. 
>
> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi 
> wrote:
>>
>> It would be nice to have a migration guide (checklist) somewhere, like:
>>
>> 1. All fields should be optional.
>> 2. Do not use custom defailt values.
>> 3. All enums should have first element with tag = 0.
>> 4. Do not use extension for anything except custom options.
>>
>> Something else?
>>
>> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
>>>
>>>
>>>
>>> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
>>>
 suppose server side is updating into protobuf3, but client side still 
 use protobuf2, can then communicate will?

>>> Yes, as long as you only use proto3 features, they are wire compatible.
>>>  
>>>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at https://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf3 be used with protobuf2?

[Artem Kazakov]

+1
Yes, a checklist would be extremely helpful. 

On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi 
wrote:
>
> It would be nice to have a migration guide (checklist) somewhere, like:
>
> 1. All fields should be optional.
> 2. Do not use custom defailt values.
> 3. All enums should have first element with tag = 0.
> 4. Do not use extension for anything except custom options.
>
> Something else?
>
> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
>>
>>
>>
>> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
>>
>>> suppose server side is updating into protobuf3, but client side still 
>>> use protobuf2, can then communicate will?
>>>
>> Yes, as long as you only use proto3 features, they are wire compatible.
>>  
>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] protobuf-java-shaded-[241|250|261]

[osuciu]

Hi all,

If you ever had runtime compatibility issues with different versions of 
protobuf-java used in the same application, 
protobuf-java-shaded-[241|250|261] may be of interest:

https://github.com/os72/protobuf-java-shaded-241
https://github.com/os72/protobuf-java-shaded-250
https://github.com/os72/protobuf-java-shaded-261

These are simply pre-shaded clones of protobuf-java with renamed namespace 
(com.google.protobuf) that includes the protobuf version explicitly:

com.github.os72.protobuf241
com.github.os72.protobuf250
com.github.os72.protobuf261

protoc-jar/protoc-jar-maven-plugin can help with the namespace rewriting 
for code generated by the protoc compiler:

https://github.com/os72/protoc-jar
https://github.com/os72/protoc-jar-maven-plugin

All of this is similar with what can be accomplished with 
maven-shade-plugin, but it's arguably simpler, with explicit versioning, 
and available at compile time (no bytecode manipulation after compilation)

Cheers
--Oliver

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Protobuf : How to generate the descriptor file dynamically from .proto

[Benjamin Krämer]

I had a similar discussion some months ago. What you want is to generate 
the TypeRegistry at Runtime. I did this by using the following approach:

https://groups.google.com/d/msg/protobuf/fnhsS9tVhyc/9pkapOl-AQAJ

Therefore you don't have to gereate the file descriptors but just use the 
ones you already have at runtime. This worked, but needed to also generate 
partial classes for the Reflection classes which implement just the 
IReflection interface (just containing the Descriptor property). This would 
work as general solution, but right now I'm not using it since I found it 
better to just have a method in my code to return the FileDescriptors they 
are needing to have registered. Since I know all modules I don't have to 
rely on reflection.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to erase particular field with MergeFrom()

[Benjamin Krämer]

And instead of encapsulating the field mask into a new message, you should 
be alright to just use google.protobuf.FieldMask as it is already a message 
by itself (in case that you don't add other data to RepAndOptMask of 
course).

Am Dienstag, 17. Mai 2016 10:51:04 UTC+2 schrieb Denis Bakhvalov:
>
>
> message RepAndOptMask
> {
> google.protobuf.FieldMask field_mask = 1;
> }
>
> I think there is nothing wrong if I will send them separately, right?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any vs. oneof in message wrappers

['David Koch' via Protocol Buffers]

Hi Feng,

Thank you for the answer. Good point about redundant type information in 
the header message.

Regards,

/David

On Monday, May 16, 2016 at 11:01:47 PM UTC+2, Feng Xiao wrote:
>
>
>
> On Mon, May 16, 2016 at 11:30 AM, 'David Koch' via Protocol Buffers <
> prot...@googlegroups.com > wrote:
>
>> Our platform generates/processes different types of proto3 messages with 
>> Kafka as buffer. We currently store this type as part of the Kafka message 
>> key so message consumers know which class to use for de-serialization.
>>
>> I want to add the type to the serialized message in order to not rely on 
>> the existence of a separate key. I thought about creating a wrapper message 
>> like this:
>>
>> message Header {
>>   enum PayloadType {
>> TYPE_ONE = 1;
>> TYPE_TWO = 2;
>>   }
>>
>>   int64 tmst = 1;
>>   string message_id = 2;
>>   string producer_id = 3;
>>   PayloadType payload_type = 4;
>> }
>>
>>
>> // Any payload.
>> message MessageWrapper {
>>   Header header = 1;
>>   Any payload = 2;
>> }
>>
>>
>> or alternatively:
>>
>> // List different payload options.
>> message MessageWrapper {
>>   Header header = 1;
>>   oneof payload {
>> PayloadTypeOne payload_type_one = 2;
>> PayloadTypeTwo payload_type_two = 3;
>>   }
>> }
>>
>>
>> My question: Is one of the two alternatives to be preferred?
>>
> I don't think you need the header in either case. If you are using an Any 
> message as payload, the payload already has the message type encoded in the 
> type URL. If you are using an oneof, the receiver can also query which 
> payload is set by using the oneof case API.
>
> Comparing the two (using Any and using oneof), using Any would give your 
> users more flexibility to use their own message types, while using oneof 
> will require all users to define their message types in a single place (all 
> imported by the MessageWrapper type). The API for oneof is straightforward 
> then Any though. If you need the flexibility to allow different components 
> to define different payload types, Any is probably better. Otherwise I 
> would favor the simpler oenof API.
>  
>
>>
>> Using Any leads to more complex setting/getting in the application code 
>> but it is more generic from the message definition point of view. Dropping 
>> the Header.PayloadType field in the definition using Any would allow me 
>> to not modify the MessageWrapper definition at all when adding new 
>> payload types.
>>
>> Thanks.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Protobuf : How to generate the descriptor file dynamically from .proto

[osuciu]

This works in Java, perhaps you can adapt it to C# (?)
https://github.com/os72/protobuf-dynamic

Is there DynamicMessage in the C# API?

On Monday, May 16, 2016 at 10:30:18 AM UTC-7, jigmesh...@gmail.com wrote:
>
> Hi,
>
> I wanted to generate FileDescriptor dynamically from .proto without 
> referencing, as in the given example we have reference Student.Interop.DTO 
> so that we can get the StudentDTO.Descriptor and EmployeeDTO.Descriptor, so 
> i wanted to avoid this reference.
>
> *Example :*
> namespace Student.WebApi
> {
> using System.Web.Http;
> using Framework.WebApi.Formatters;
> using Google.Protobuf.Reflection;
> using Repository;  
> using Student.Interop.DTO;  
>
> public class WebApiApplication : System.Web.HttpApplication
> {
> protected void Application_Start()
> {
> GlobalConfiguration.Configure(WebApiConfig.Register);
> var json = 
> GlobalConfiguration.Configuration.Formatters.JsonFormatter;
> json.SerializerSettings.Converters.Add(new 
> AnyJsonConverter(TypeRegistry.FromMessages(StudentDTO.Descriptor, 
> EmployeeDTO.Descriptor)));
> 
> }  
> }
> }
>
> So to achieve this following few steps i tried :
>
> *Step 1* : Sample Proto i.e Student.proto which consist of message type 
> i.e StudentDTO and EmployeeDTO
>
> syntax = "proto3";
> package AnalysisConsoleApplication;
> import "google/protobuf/any.proto";
> option csharp_namespace = "AnalysisConsoleApplication";
>
> message StudentDTO
> { google.protobuf.Any Any = 1;
> string USN = 2; 
> string Name = 3;
> string University = 4;
> }
> message EmployeeDTO
> { 
> string Name = 1;
> string Gender = 2;
> }
>
> Step 2 : I generated Student.pb by using below command i.e 
>
> .\protoc.exe --proto_path=D:\Proto Proto\Student.proto --include_imports 
> --descriptor_set_out=D:\AnalysisConsoleApplication\Student.pb
>
> Step 3 : If i reference Google.Protobuf.3.0.0-beta2 in my 
> SampleConsoleApplication and try to use FileDescriptor class then it will 
> give error as recently in updated version FileDescriptor  is of Internal 
> sealed class. so i clone the Google.Protobuf repository which was of 
> Google.Protobuf.3.0.0-beta2 release time, at that time FileDescriptor class 
> was public sealed class. Now my SampleConsoleApplication  solution will 
> have another project which is of Google.Protobuf.
>
> Note: SampleConsoleApplication zip file is been attached for detail 
> understanding.
>
> This code is from Actual generated C# file(Student.cs) from the given 
> proto, but now next step we will create this below descriptorData dynamically 
> from Student.pb
> byte[] descriptorData = global::System.Convert.FromBase64String(
>   string.Concat(
> "Cg1TdHVkZW50LnByb3RvEhpBbmFseXNpc0NvbnNvbGVBcHBsaWNhdGlvbhoZ",
> "Z29vZ2xlL3Byb3RvYnVmL2FueS5wcm90byJeCgpTdHVkZW50RFRPEiEKA0Fu",
> "eRgBIAEoCzIULmdvb2dsZS5wcm90b2J1Zi5BbnkSCwoDVVNOGAIgASgJEgwK",
> "BE5hbWUYAyABKAkSEgoKVW5pdmVyc2l0eRgEIAEoCSIrCgtFbXBsb3llZURU",
> "TxIMCgROYW1lGAEgASgJEg4KBkdlbmRlchgCIAEoCUIdqgIaQW5hbHlzaXND",
> "b25zb2xlQXBwbGljYXRpb25iBnByb3RvMw=="));
>   descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
>   new pbr::FileDescriptor[] { 
> global::Google.Protobuf.WellKnownTypes.AnyReflection.Descriptor, },
>   new pbr::GeneratedCodeInfo(null, new pbr::GeneratedCodeInfo[] {
> new 
> pbr::GeneratedCodeInfo(typeof(global::AnalysisConsoleApplication.StudentDTO), 
> global::AnalysisConsoleApplication.StudentDTO.Parser, new[]{ "Any", "USN", 
> "Name", "University" }, null, null, null),
> new 
> pbr::GeneratedCodeInfo(typeof(global::AnalysisConsoleApplication.EmployeeDTO),
>  
> global::AnalysisConsoleApplication.EmployeeDTO.Parser, new[]{ "Name", 
> "Gender" }, null, null, null)
>   }));
>
> Step 4: Now to generate descriptorData  dynamically from Student.pb, i 
> have written the following code which gives the descriptorData in byte[]
>
> //get the path of Student.pb
> var path = string.Format("{0}/Student.pb", Path.GetDirectoryName(new 
> Uri(Assembly.GetExecutingAssembly().CodeBase).LocalPath)); 
> //create messageParser
> MessageParser messageParser = new 
> MessageParser(() => new FileDescriptorSet());
> // open and read it
> CodedInputStream stream = new CodedInputStream(new FileStream(path, 
> FileMode.Open));
> //Parses a message from the given coded input stream.
> var data = messageParser .ParseFrom(stream);
> //Covert to ByteString
> var descriptorData  = data.File[0].ToByteString();   
>
> Step 5: Last step to generate filedescriptor by calling 
> FileDescriptor.FromGeneratedCode() 
> which accepts 3 parameters
> Parameter 1:  byte[] descriptorData -> this parameter we are able to 
> generate dynamically as done in step 4 i.e   descriptorData.ToByteArray()  
>  
> Parameter 2:  FileDescriptor[] dependencies-> this 

Re: [protobuf] Dynamic/run-time decoding

[osuciu]

At least in Java, you can do everything fully dynamically at runtime. The 
native API can get quite confusing (between all the 
protos/descriptors/files...). I distilled following wrapper to make it 
easier:
https://github.com/os72/protobuf-dynamic


On Friday, April 1, 2016 at 4:32:29 PM UTC-7, Josh Haberman wrote:
>
> Hi Mike,
>
> Python indeed has descriptor classes that you can use to build proto 
> definitions at runtime. However it does not have a parser for .proto files 
> built in. So if you want to load .proto files at runtime, you'll need to 
> use protoc to build a descriptor (for example, you could shell out to 
> protoc).
>
> Josh
>
> On Monday, March 21, 2016 at 10:43:24 AM UTC-7, Mike Trienis wrote:
>>
>> Hi,
>>
>> Do you know if it's possible do dynamic decoding at run-time using the 
>> Python API? 
>>
>>- 
>>https://developers.google.com/protocol-buffers/docs/reference/python/
>>
>> Indeed there is a descriptor.FileDescriptor, however I'm not sure what 
>> the other classes correspond to?
>>
>> Thanks, Mike. 
>>
>> On Tuesday, August 12, 2014 at 10:36:13 AM UTC-7, Feng Xiao wrote:
>>>
>>> Protobuf supports creating message types dynamically at runtime and use 
>>> them for parsing/serialization/etc.
>>>
>>> First you need to build up a DescriptorPool 
>>> 
>>>  
>>> that contains all types that you may want to use. There are two approaches 
>>> to construct this pool. One is to call DescriptorPool::BuildFile() directly 
>>> with parsed proto files. For example:
>>>   // Convert .proto files into parsed FileDescriptorProto
>>>   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
>>> FileInputStream stream(filename);
>>> google::protobuf::io::Tokenizer tokenizer();
>>> google::protobuf::compiler::Parser parser;
>>> return parser.Parse(, result);
>>>   }
>>>   // Build the descriptor pool
>>>   DescriptorPool pool;
>>>   for (string filename : proto_files) {
>>> FileDescriptorProto proto;
>>> ParseProtoFile(filename, );
>>> pool.BuildFile(proto);
>>>   }
>>>
>>> After you have the pool, you can query for a type by its name. For 
>>> example, DescriptorPool::FindMessageTypeByName().
>>>
>>> Then to actually parse/serialize/use message types in the pool, you need 
>>> to construct message objects around them. DynamicMessage 
>>> 
>>>  
>>> is used for that:
>>>   // Suppose you want to parse a message type with a specific type name.
>>>   Descriptor* descriptor = 
>>> pool.FindMessageTypeByName(message_type_to_parse);
>>>   DynamicMessageFactory factory;
>>>   unique_ptr message = factory.GetPrototype(descriptor)->New();
>>>   // Use the message object for parsing/etc.
>>>   message->ParseFromString(input_data);
>>>   // Access a specific field in the message
>>>   FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);
>>>   switch (field->type()) {
>>> case TYPE_INT32: message->GetReflection()->GetInt32(*message, 
>>> field); break;
>>> ...
>>>   }
>>>
>>> On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský  
>>> wrote:
>>>
 Hi,

 I have very specific problem. I have data and proto file available and 
 my application should take both and based on external configuration 
 determine how to interpret data (many different types/messages in proto). 
 Yet that can be determine only during run. My question is if there is any 
 support for that, I mean that I will be able to parse proto and decode 
 data 
 based on content of interpret intermediate structures.

 I have been trying to analyze this possibility directly from codes but 
 not with much success. I would be glad for any guidance.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Checking compatibility

[osuciu]

Check out the protobuf schema evolution rules, you may find it's actually 
simpler with protobuf and don't have to keep track of writer schemas like 
in Avro:
https://developers.google.com/protocol-buffers/docs/proto#updating

Corresponding to Avro GenericRecord you would use protobuf DynamicMessage

It's not documented very prominently, but you can use protobuf dynamically 
(no codegen). Schemas can be constructed at runtime, although the API can 
get confusing. And from schemas you can construct DynamicMessages. I wrote 
a wrapper to help make that a bit easier:
https://github.com/os72/protobuf-dynamic


On Friday, May 6, 2016 at 5:00:26 PM UTC-7, Vinicius Carvalho wrote:
>
> Hi there, does protobuf has an equivalent of Avro's 
> checkReaderWriterCompatibility? Can I check if two .proto files would allow 
> match between versions?
>
> Also, Avro has GenericRecord as a container object to read any data in a 
> map like style, does protobuf has something similar?
>
> Regards
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.