Re: [protobuf] Status of protobuf-java 2.x / 3.x compatibility

['Feng Xiao' via Protocol Buffers]

On Tue, Dec 5, 2017 at 1:33 PM, Chris Thunes  wrote:

> Thanks Feng. It seems like the GeneratedMessage / GeneratedMessageV3 split
> introduced between 3.0.0-beta-4 and the 3.0.0 final release caused the
> java/compatability_tests suite to start failing when run against 2.5.0 /
> 2.6.1 ("java/compatibility_tests/v2.5.0/test.sh 2.5.0" passes at git tag
> v3.0.0-beta-4 but fails at v3.0.0).
>
This is expected. Only 3.0.0-beta-4 is intended to be compatible with 2.5.0/
2.6.1. The final 3.0.0 release is not compatible with 2.5.0/2.6.1. 3.0.0
release is compatible with 3.0.0-beta-4 though. So a migration path could
be:
1. Move individual components to depend on 3.0.0-beta-4 instead of 2.5.0/
2.6.1. Since 3.0.0-beta-4 is compatible with 2.5.0/2.6.1 you don't need to
upgrade everything at once but can do it incrementally.
2. Once every component is on 3.0.0-beta-4, upgrade the dependency to 3.x.x.

This is how we do the migration inside google, but I guess it's hard to do
the same in the opensource world because you don't have control over your
dependencies.


> In 3.0.0 and newer, what is the role of the non-V3 variants of
> GeneratedMessage, SingleFieldBuilder, etc? Are these classes simply
> vestigial at this point or do they still provide some benefit to
> compatibility (even if not 100%).
>
They are still needed to be compatible with older beta releases like
3.0.0-beta-3 etc.


>
> Thanks again,
> Chris
>
> On Mon, Dec 4, 2017 at 8:24 PM, Feng Xiao  wrote:
>
>>
>>
>> On Mon, Dec 4, 2017 at 9:00 AM, Chris Thunes  wrote:
>>
>>>
>>> I'm looking at options for moving some applications that currently
>>> depend on protobuf-java 2.5.0 to a more recent version. This is made
>>> complicated by the fact that we have a mixure of internal and external
>>> dependencies (Hadoop & HBase) which depend on protobuf-java. My
>>> understanding is that this will require these dependencies to move to a 3.x
>>> release sychronously (i.e. regenerate using a 3.x protoc and update
>>> protobuf-java to a corresponding release).
>>>
>>> However, looking through release notes and protobuf source code it seems
>>> like some attempts have been made to address the source and binary
>>> compatibility issues between 2.5/2.6.1 and 3.x. Specifically,
>>>
>>>- The 3.0.0-beta-4 release notes
>>>
>>>mention runtime updates "to be compatible with 2.5.0/2.6.1 generated
>>>protos".
>>>- A number of classes have "V3" variants where the non-V3 variants
>>>appear to exist solely in an attempt to maintain binary compatibility 
>>> with
>>>pre-3.x generated code.
>>>
>>> Running the compatibility tests in java/compatibility_tests/v2.5.0 it
>>> appears that source and binary incompatibilities still exist.
>>>
>>>
>>> I'm curious if anyone can shed some light on this effort and its status
>>> or provide suggestions for migrating to a recent protobuf release under
>>> these circumstances.
>>>
>> Protobuf 2.5.0/2.6.1 should be compatible with 3.0.0-beta-4 if you only
>> uses protobuf public APIs. That's unfortunately not the case with Hadoop &
>> HBase though. They introduced a class into com.google.protobuf package
>> with the sole purpose to access protobuf package private classes
>> .
>>  As
>> such there is no way you can upgrade to protobuf 3.x if the version of
>> Hadoop & HBase you use still depends on protobuf 2.5/2.6 private symbols.
>>
>>
>>> Thanks,
>>>
>>> Chris
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Status of protobuf-java 2.x / 3.x compatibility

[Chris Thunes]

Thanks Feng. It seems like the GeneratedMessage / GeneratedMessageV3 split
introduced between 3.0.0-beta-4 and the 3.0.0 final release caused the
java/compatability_tests suite to start failing when run against 2.5.0 /
2.6.1 ("java/compatibility_tests/v2.5.0/test.sh 2.5.0" passes at git tag
v3.0.0-beta-4 but fails at v3.0.0). In 3.0.0 and newer, what is the role of
the non-V3 variants of GeneratedMessage, SingleFieldBuilder, etc? Are these
classes simply vestigial at this point or do they still provide some
benefit to compatibility (even if not 100%).

Thanks again,
Chris

On Mon, Dec 4, 2017 at 8:24 PM, Feng Xiao  wrote:

>
>
> On Mon, Dec 4, 2017 at 9:00 AM, Chris Thunes  wrote:
>
>>
>> I'm looking at options for moving some applications that currently depend
>> on protobuf-java 2.5.0 to a more recent version. This is made complicated
>> by the fact that we have a mixure of internal and external dependencies
>> (Hadoop & HBase) which depend on protobuf-java. My understanding is that
>> this will require these dependencies to move to a 3.x release sychronously
>> (i.e. regenerate using a 3.x protoc and update protobuf-java to a
>> corresponding release).
>>
>> However, looking through release notes and protobuf source code it seems
>> like some attempts have been made to address the source and binary
>> compatibility issues between 2.5/2.6.1 and 3.x. Specifically,
>>
>>- The 3.0.0-beta-4 release notes
>>
>>mention runtime updates "to be compatible with 2.5.0/2.6.1 generated
>>protos".
>>- A number of classes have "V3" variants where the non-V3 variants
>>appear to exist solely in an attempt to maintain binary compatibility with
>>pre-3.x generated code.
>>
>> Running the compatibility tests in java/compatibility_tests/v2.5.0 it
>> appears that source and binary incompatibilities still exist.
>>
>>
>> I'm curious if anyone can shed some light on this effort and its status
>> or provide suggestions for migrating to a recent protobuf release under
>> these circumstances.
>>
> Protobuf 2.5.0/2.6.1 should be compatible with 3.0.0-beta-4 if you only
> uses protobuf public APIs. That's unfortunately not the case with Hadoop &
> HBase though. They introduced a class into com.google.protobuf package
> with the sole purpose to access protobuf package private classes
> .
>  As
> such there is no way you can upgrade to protobuf 3.x if the version of
> Hadoop & HBase you use still depends on protobuf 2.5/2.6 private symbols.
>
>
>> Thanks,
>>
>> Chris
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Benchmark test dataset updated.

[yilunchong via Protocol Buffers]

Hi all,

We've just added some more confirming testing datasets of google_message3 
and google_mssage4 for benchmark. 
You may download the big testing data by run the shell download_data.sh 


Feel free to make comments and suggestion on benchmarks!

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Migrating groups to nested fields

['Adam Cozzette' via Protocol Buffers]

No, unfortunately groups and nested messages have incompatible wire
formats, so if you want to convert to proto3 then you will have to first
migrate away from the group to a new submessage field.

On Mon, Dec 4, 2017 at 1:51 AM,  wrote:

> Hi,
>
> My goal is to convert existing proto2 to proto3. Unfortunately, there are
> group fields. It is not hard to represent them as nested messages. But is
> wire format compatible for them?
> Will it be possible to achieve backward compatibility?
>
> Regards,
> Gor.
> Image already added
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.