Re: [protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

[Behrang Saeedzadeh]

Unfortunately I don't own the actual protocol buffer models and can't 
change their definitions.

What I actually want to do is this: I have a Protocol buffer model in 
byte[] format. I want to deserialize it back to Java objects but then 
replace all the ID fields in the model, including its extensions. 

For example, I have a "template" Employee model in byte[] format, and I 
want to create new Java objects using that template. After deserializing 
the byte array to a Java object, I want to change the UserHeader.userID and 
ID of all the skills in it.

Is there a fluent API or a concise way available to do this?



On Thursday, December 21, 2017 at 10:56:10 AM UTC+11, Feng Xiao wrote:
>
> JsonFormat doesn't support extensions. You can replace extensions with 
> google.protobuf.Any if you want to use the proto with JsonFormat.
>
> On Wed, Dec 20, 2017 at 3:39 PM, Behrang Saeedzadeh  > wrote:
>
>>
>>
>> *down vote**favorite* 
>> 
>>
>> *Cross-post from 
>> StackOverflow: 
>> https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
>>  
>> *
>>
>> I have 2 Protobuf models:
>> User:
>>
>> package demo;
>>
>> option java_package = "com.stackoverflow.question";
>> option java_outer_classname = "UserModel";
>>
>> message User {
>>
>> message UserHeader {
>> required int64 userId = 1;
>> }
>>
>> required UserHeader header = 1;
>>
>> extensions 100 to 200;}
>>
>> Employee:
>>
>> import "person.proto";
>> package demo;
>>
>> option java_package = "com.stackoverflow.question";
>> option java_outer_classname = "EmployeeModel";
>>
>> extend demo.User {
>> optional EmployeeDetails details = 101;}
>>
>> message EmployeeDetails {
>> required string department = 1;
>> repeated Skill skills = 2;}
>>
>> message Skill {
>> required int64 id = 1;
>> required string name = 2;}
>>
>> I can create a model and serialize it to JSON using 
>> JsonFormat.printer().print(...):
>>
>> ExtensionRegistry registry = 
>> ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
>> UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
>> userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
>> EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
>> EmployeeModel.EmployeeDetails.newBuilder();
>> employeeBuilder.setDepartment("Department 1")
>>.addSkills(EmployeeModel.Skill.newBuilder()
>>  .setId(10_000)
>>  .setName("Skill 10_")
>>  .build())
>>.addSkills(EmployeeModel.Skill.newBuilder()
>>  .setId(11_000)
>>  .setName("Skill 11_")
>>  .build());
>>
>> userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
>> final String json = JsonFormat.printer().print(userBuilder.build());
>>
>> However deserializing the generated JSON back to Java objects fails with 
>> com.google.protobuf.InvalidProtocolBufferException: 
>> Cannot find field: details in message demo.User:
>>
>> UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
>> JsonFormat.parser().merge(json, userBuilder2);
>>
>> And there doesn't seem to be a way to pass an ExtensionRegistry to 
>> JsonFormat.parser()either.
>>
>> Is there a way to make this *Protobuf → JSON → Protobuf* 
>> serialization/deserialization 
>> chain work?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

['Feng Xiao' via Protocol Buffers]

JsonFormat doesn't support extensions. You can replace extensions with
google.protobuf.Any if you want to use the proto with JsonFormat.

On Wed, Dec 20, 2017 at 3:39 PM, Behrang Saeedzadeh 
wrote:

>
>
> *down vote**favorite*
> 
>
> *Cross-post from
> StackOverflow: 
> https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
> *
>
> I have 2 Protobuf models:
> User:
>
> package demo;
>
> option java_package = "com.stackoverflow.question";
> option java_outer_classname = "UserModel";
>
> message User {
>
> message UserHeader {
> required int64 userId = 1;
> }
>
> required UserHeader header = 1;
>
> extensions 100 to 200;}
>
> Employee:
>
> import "person.proto";
> package demo;
>
> option java_package = "com.stackoverflow.question";
> option java_outer_classname = "EmployeeModel";
>
> extend demo.User {
> optional EmployeeDetails details = 101;}
>
> message EmployeeDetails {
> required string department = 1;
> repeated Skill skills = 2;}
>
> message Skill {
> required int64 id = 1;
> required string name = 2;}
>
> I can create a model and serialize it to JSON using JsonFormat.printer().
> print(...):
>
> ExtensionRegistry registry = 
> ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
> UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
> userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
> EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
> EmployeeModel.EmployeeDetails.newBuilder();
> employeeBuilder.setDepartment("Department 1")
>.addSkills(EmployeeModel.Skill.newBuilder()
>  .setId(10_000)
>  .setName("Skill 10_")
>  .build())
>.addSkills(EmployeeModel.Skill.newBuilder()
>  .setId(11_000)
>  .setName("Skill 11_")
>  .build());
>
> userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
> final String json = JsonFormat.printer().print(userBuilder.build());
>
> However deserializing the generated JSON back to Java objects fails with
> com.google.protobuf.InvalidProtocolBufferException: Cannot find field:
> details in message demo.User:
>
> UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
> JsonFormat.parser().merge(json, userBuilder2);
>
> And there doesn't seem to be a way to pass an ExtensionRegistry to JsonFo
> rmat.parser()either.
>
> Is there a way to make this *Protobuf → JSON → Protobuf* 
> serialization/deserialization
> chain work?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

[Behrang Saeedzadeh]

*down vote**favorite* 


*Cross-post from 
StackOverflow: 
https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
 
*

I have 2 Protobuf models:
User:

package demo;

option java_package = "com.stackoverflow.question";
option java_outer_classname = "UserModel";

message User {

message UserHeader {
required int64 userId = 1;
}

required UserHeader header = 1;

extensions 100 to 200;}

Employee:

import "person.proto";
package demo;

option java_package = "com.stackoverflow.question";
option java_outer_classname = "EmployeeModel";

extend demo.User {
optional EmployeeDetails details = 101;}

message EmployeeDetails {
required string department = 1;
repeated Skill skills = 2;}

message Skill {
required int64 id = 1;
required string name = 2;}

I can create a model and serialize it to JSON using 
JsonFormat.printer().print(...):

ExtensionRegistry registry = 
ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
EmployeeModel.EmployeeDetails.newBuilder();
employeeBuilder.setDepartment("Department 1")
   .addSkills(EmployeeModel.Skill.newBuilder()
 .setId(10_000)
 .setName("Skill 10_")
 .build())
   .addSkills(EmployeeModel.Skill.newBuilder()
 .setId(11_000)
 .setName("Skill 11_")
 .build());

userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
final String json = JsonFormat.printer().print(userBuilder.build());

However deserializing the generated JSON back to Java objects fails with 
com.google.protobuf.InvalidProtocolBufferException: 
Cannot find field: details in message demo.User:

UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
JsonFormat.parser().merge(json, userBuilder2);

And there doesn't seem to be a way to pass an ExtensionRegistry to 
JsonFormat.parser()either.

Is there a way to make this *Protobuf → JSON → Protobuf* 
serialization/deserialization 
chain work?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Feng Xiao' via Protocol Buffers]

On Wed, Dec 20, 2017 at 11:25 AM, Josh Haberman  wrote:

> Thanks Feng and sorry for the confusion everyone!
>
> Feng quoted my original email, but I'll repeat it here for clarity.
> Hopefully everyone will be able to reply to messages properly this time. :)
>
> --
>
> Hi all,
>
> I wanted to kick off a conversation about the future of Protobuf and
> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
> Protobuf package, and has also expressed interest in packaging Abseil.
>
> Debian currently distributes a binary shared library for protobuf
> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
> time there is an ABI-incompatible change to the library:
>
> https://www.debian.org/doc/debian-policy/#shared-libraries
>
> Since protobuf does not guarantee any kind of binary compatibility, we
> have traditionally updated our SONAME for every minor release of protobuf.
> We are currently at major version 15:
>
> https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab95c773192b
> 5d908c5a61/src/Makefile.am#L186
>
> This has worked reasonably well so far. But we want to start depending on
> Abseil (https://abseil.io) which does not do releases at all (so doesn't
> even have a major version number to track). How would this work for Debian?
>
> Abseil's philosophy is that users will build from source. But Debian is
> inherently a binary distribution. We have to figure out how to reconcile
> these two philosophies. I don't know the answer, but I'm going to throw out
> a few straw man ideas as a starting point for discussion.
>
> 1. Distribute libprotobuf and libabsl as static libraries only.
>
If protobuf is static only, could other .so libraries still link against
protobuf? If multiple .so libraries has linked-in protobuf statically,
would it cause symbol conflicts when they are used together? It seems to me
if protobuf goes static only, every other library that depends on protobuf
must be static only as well.


>
> Debian policy says that libraries can be distributed in static form only
> if that is what is intended by upstream:
>
> https://www.debian.org/doc/debian-policy/#static-libraries
>
> Since both libprotobuf and libabsl are disavowing any binary compatibility
> guarantees, that implies that they should not be distributed as .so's. Any
> Debian packages or user programs that want to use protobuf have to link it
> statically.
>
> The tricky part is that libprotobuf.a and libabsl.a would need to match
> each other exactly. They would need to be compiled from exactly the same
> ABSL headers. This implies that libprotobuf-dev would need to pin itself to
> one specific libabsl-dev version -- the one it was compiled against. Other
> libraries that depend on ABSL (like TensorFlow) would also need to be
> pinned to exactly the same libabsl-dev version, otherwise it would be
> impossible to install them all at the same time. So realistically all
> packages that depend on ABSL would need to be updated at the same time.
>
> 2. Distribute libprotobuf and libabsl as shared libraries.
>
> This is against the philosophy of ABSL. But if Debian updated its ABSL
> infrequently enough, it could just treat each ABSL update as a separate
> major version. Sort of like we do with protobuf now.
>
> The tricky part is if two different libraries pass ABSL types across
> library boundaries. For example, imagine that a user program was developing
> against libabsl.so.2 but also using libprotobuf.so.15 that was linked
> against libabsl.so.1. If a user got an absl::string_view from protobuf,
> this could be unsafe for the user to pass directly to ABSL functions. The
> representation of absl::string_view could have changed between the two
> versions of libabsl.
>
I'm not so worried about this because this problem is not new. With or
without absl, there are libraries depending on different versions of
protobuf.so and there must already exists an established solution to deal
with this. Adding absl to the picture won't make it any trickier than it is
before.


>
> The all-static option (1) seems to solve this problem by pinning all of
> the ABSL dependencies so that they all rev together. But this solution
> could become unwieldy if too many packages depend on ABSL. Also any library
> that exposes ABSL types in its API would have to be part of this lock-step
> ABSL upgrade.
>
> 3. Remove protobuf from Debian completely, ask users to build from source.
>
It seems very unlikely we can take down all the protobuf.so distributed on
the internet... We can make it harder but people will get a way around it.


>
> Since the philosophy of ABSL is that users will always build from source,
> we could ask that users do this if they want to develop against protobuf.
> Debian packages that depend on protobuf could build protobuf (and ABSL) and
> statically link against it to build runnable binaries. It would not be
> possible to have library packages in Debian that depend on protobuf.
>

> 

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Josh Haberman' via Protocol Buffers]

Thanks Feng and sorry for the confusion everyone!

Feng quoted my original email, but I'll repeat it here for clarity.
Hopefully everyone will be able to reply to messages properly this time. :)

--

Hi all,

I wanted to kick off a conversation about the future of Protobuf and
Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
Protobuf package, and has also expressed interest in packaging Abseil.

Debian currently distributes a binary shared library for protobuf
(libprotobuf.so). Debian policy requires that the SONAME is bumped every
time there is an ABI-incompatible change to the library:

https://www.debian.org/doc/debian-policy/#shared-libraries

Since protobuf does not guarantee any kind of binary compatibility, we have
traditionally updated our SONAME for every minor release of protobuf. We
are currently at major version 15:

https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab95c773192b5d908c5a61/src/Makefile.am#L186

This has worked reasonably well so far. But we want to start depending on
Abseil (https://abseil.io) which does not do releases at all (so doesn't
even have a major version number to track). How would this work for Debian?

Abseil's philosophy is that users will build from source. But Debian is
inherently a binary distribution. We have to figure out how to reconcile
these two philosophies. I don't know the answer, but I'm going to throw out
a few straw man ideas as a starting point for discussion.

1. Distribute libprotobuf and libabsl as static libraries only.

Debian policy says that libraries can be distributed in static form only if
that is what is intended by upstream:

https://www.debian.org/doc/debian-policy/#static-libraries

Since both libprotobuf and libabsl are disavowing any binary compatibility
guarantees, that implies that they should not be distributed as .so's. Any
Debian packages or user programs that want to use protobuf have to link it
statically.

The tricky part is that libprotobuf.a and libabsl.a would need to match
each other exactly. They would need to be compiled from exactly the same
ABSL headers. This implies that libprotobuf-dev would need to pin itself to
one specific libabsl-dev version -- the one it was compiled against. Other
libraries that depend on ABSL (like TensorFlow) would also need to be
pinned to exactly the same libabsl-dev version, otherwise it would be
impossible to install them all at the same time. So realistically all
packages that depend on ABSL would need to be updated at the same time.

2. Distribute libprotobuf and libabsl as shared libraries.

This is against the philosophy of ABSL. But if Debian updated its ABSL
infrequently enough, it could just treat each ABSL update as a separate
major version. Sort of like we do with protobuf now.

The tricky part is if two different libraries pass ABSL types across
library boundaries. For example, imagine that a user program was developing
against libabsl.so.2 but also using libprotobuf.so.15 that was linked
against libabsl.so.1. If a user got an absl::string_view from protobuf,
this could be unsafe for the user to pass directly to ABSL functions. The
representation of absl::string_view could have changed between the two
versions of libabsl.

The all-static option (1) seems to solve this problem by pinning all of the
ABSL dependencies so that they all rev together. But this solution could
become unwieldy if too many packages depend on ABSL. Also any library that
exposes ABSL types in its API would have to be part of this lock-step ABSL
upgrade.

3. Remove protobuf from Debian completely, ask users to build from source.

Since the philosophy of ABSL is that users will always build from source,
we could ask that users do this if they want to develop against protobuf.
Debian packages that depend on protobuf could build protobuf (and ABSL) and
statically link against it to build runnable binaries. It would not be
possible to have library packages in Debian that depend on protobuf.

Thoughts?
Josh

On Wed, Dec 20, 2017 at 10:58 AM, Feng Xiao  wrote:

> +abseil...@googlegroups.com, +proto...@gogolegroups.com
>
> CCed public google groups so László can reply to the group. BCCed internal
> mailing lists.
>
> On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman 
> wrote:
>
>> Hi all,
>>
>> I wanted to kick off a conversation about the future of Protobuf and
>> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
>> Protobuf package, and has also expressed interest in packaging Abseil.
>>
>> Debian currently distributes a binary shared library for protobuf
>> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
>> time there is an ABI-incompatible change to the library:
>>
>> https://www.debian.org/doc/debian-policy/#shared-libraries
>>
>> Since protobuf does not guarantee any kind of binary compatibility, we
>> have traditionally updated our SONAME for every minor release of protobuf.
>> We are currently at 

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Jisi Liu' via Protocol Buffers]

There's also header dependency issue, i.e. libprotobuf-dev may distribute
headers which may transitively include headers from libabseil-dev
(currently we duplicate atomicops, platform macros, etc). This means the
protobuf dev package needs to pin with a specific libabseil-dev, even if we
go with the shared library approach. One way to solve the problem is avoid
transitive include of abseil headers in protobuf headers.

On Wed, Dec 20, 2017 at 10:58 AM Feng Xiao  wrote:

> +abseil...@googlegroups.com, +proto...@gogolegroups.com
>
> CCed public google groups so László can reply to the group. BCCed internal
> mailing lists.
>
> On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman 
> wrote:
>
>> Hi all,
>>
>> I wanted to kick off a conversation about the future of Protobuf and
>> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
>> Protobuf package, and has also expressed interest in packaging Abseil.
>>
>> Debian currently distributes a binary shared library for protobuf
>> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
>> time there is an ABI-incompatible change to the library:
>>
>> https://www.debian.org/doc/debian-policy/#shared-libraries
>>
>> Since protobuf does not guarantee any kind of binary compatibility, we
>> have traditionally updated our SONAME for every minor release of protobuf.
>> We are currently at major version 15:
>>
>>
>> https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab95c773192b5d908c5a61/src/Makefile.am#L186
>>
>> This has worked reasonably well so far. But we want to start depending on
>> Abseil (https://abseil.io) which does not do releases at all (so doesn't
>> even have a major version number to track). How would this work for Debian?
>>
>> Abseil's philosophy is that users will build from source. But Debian is
>> inherently a binary distribution. We have to figure out how to reconcile
>> these two philosophies. I don't know the answer, but I'm going to throw out
>> a few straw man ideas as a starting point for discussion.
>>
>> 1. Distribute libprotobuf and libabsl as static libraries only.
>>
>> Debian policy says that libraries can be distributed in static form only
>> if that is what is intended by upstream:
>>
>> https://www.debian.org/doc/debian-policy/#static-libraries
>>
>> Since both libprotobuf and libabsl are disavowing any binary
>> compatibility guarantees, that implies that they should not be distributed
>> as .so's. Any Debian packages or user programs that want to use protobuf
>> have to link it statically.
>>
>> The tricky part is that libprotobuf.a and libabsl.a would need to match
>> each other exactly. They would need to be compiled from exactly the same
>> ABSL headers. This implies that libprotobuf-dev would need to pin itself to
>> one specific libabsl-dev version -- the one it was compiled against. Other
>> libraries that depend on ABSL (like TensorFlow) would also need to be
>> pinned to exactly the same libabsl-dev version, otherwise it would be
>> impossible to install them all at the same time. So realistically all
>> packages that depend on ABSL would need to be updated at the same time.
>>
>> 2. Distribute libprotobuf and libabsl as shared libraries.
>>
>> This is against the philosophy of ABSL. But if Debian updated its ABSL
>> infrequently enough, it could just treat each ABSL update as a separate
>> major version. Sort of like we do with protobuf now.
>>
>> The tricky part is if two different libraries pass ABSL types across
>> library boundaries. For example, imagine that a user program was developing
>> against libabsl.so.2 but also using libprotobuf.so.15 that was linked
>> against libabsl.so.1. If a user got an absl::string_view from protobuf,
>> this could be unsafe for the user to pass directly to ABSL functions. The
>> representation of absl::string_view could have changed between the two
>> versions of libabsl.
>>
>> The all-static option (1) seems to solve this problem by pinning all of
>> the ABSL dependencies so that they all rev together. But this solution
>> could become unwieldy if too many packages depend on ABSL. Also any library
>> that exposes ABSL types in its API would have to be part of this lock-step
>> ABSL upgrade.
>>
>> 3. Remove protobuf from Debian completely, ask users to build from source.
>>
>> Since the philosophy of ABSL is that users will always build from source,
>> we could ask that users do this if they want to develop against protobuf.
>> Debian packages that depend on protobuf could build protobuf (and ABSL) and
>> statically link against it to build runnable binaries. It would not be
>> possible to have library packages in Debian that depend on protobuf.
>>
>> Thoughts?
>> Josh
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to 

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Feng Xiao' via Protocol Buffers]

+abseil...@googlegroups.com, +proto...@gogolegroups.com

CCed public google groups so László can reply to the group. BCCed internal
mailing lists.

On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman  wrote:

> Hi all,
>
> I wanted to kick off a conversation about the future of Protobuf and
> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
> Protobuf package, and has also expressed interest in packaging Abseil.
>
> Debian currently distributes a binary shared library for protobuf
> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
> time there is an ABI-incompatible change to the library:
>
> https://www.debian.org/doc/debian-policy/#shared-libraries
>
> Since protobuf does not guarantee any kind of binary compatibility, we
> have traditionally updated our SONAME for every minor release of protobuf.
> We are currently at major version 15:
>
> https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab95c773192b
> 5d908c5a61/src/Makefile.am#L186
>
> This has worked reasonably well so far. But we want to start depending on
> Abseil (https://abseil.io) which does not do releases at all (so doesn't
> even have a major version number to track). How would this work for Debian?
>
> Abseil's philosophy is that users will build from source. But Debian is
> inherently a binary distribution. We have to figure out how to reconcile
> these two philosophies. I don't know the answer, but I'm going to throw out
> a few straw man ideas as a starting point for discussion.
>
> 1. Distribute libprotobuf and libabsl as static libraries only.
>
> Debian policy says that libraries can be distributed in static form only
> if that is what is intended by upstream:
>
> https://www.debian.org/doc/debian-policy/#static-libraries
>
> Since both libprotobuf and libabsl are disavowing any binary compatibility
> guarantees, that implies that they should not be distributed as .so's. Any
> Debian packages or user programs that want to use protobuf have to link it
> statically.
>
> The tricky part is that libprotobuf.a and libabsl.a would need to match
> each other exactly. They would need to be compiled from exactly the same
> ABSL headers. This implies that libprotobuf-dev would need to pin itself to
> one specific libabsl-dev version -- the one it was compiled against. Other
> libraries that depend on ABSL (like TensorFlow) would also need to be
> pinned to exactly the same libabsl-dev version, otherwise it would be
> impossible to install them all at the same time. So realistically all
> packages that depend on ABSL would need to be updated at the same time.
>
> 2. Distribute libprotobuf and libabsl as shared libraries.
>
> This is against the philosophy of ABSL. But if Debian updated its ABSL
> infrequently enough, it could just treat each ABSL update as a separate
> major version. Sort of like we do with protobuf now.
>
> The tricky part is if two different libraries pass ABSL types across
> library boundaries. For example, imagine that a user program was developing
> against libabsl.so.2 but also using libprotobuf.so.15 that was linked
> against libabsl.so.1. If a user got an absl::string_view from protobuf,
> this could be unsafe for the user to pass directly to ABSL functions. The
> representation of absl::string_view could have changed between the two
> versions of libabsl.
>
> The all-static option (1) seems to solve this problem by pinning all of
> the ABSL dependencies so that they all rev together. But this solution
> could become unwieldy if too many packages depend on ABSL. Also any library
> that exposes ABSL types in its API would have to be part of this lock-step
> ABSL upgrade.
>
> 3. Remove protobuf from Debian completely, ask users to build from source.
>
> Since the philosophy of ABSL is that users will always build from source,
> we could ask that users do this if they want to develop against protobuf.
> Debian packages that depend on protobuf could build protobuf (and ABSL) and
> statically link against it to build runnable binaries. It would not be
> possible to have library packages in Debian that depend on protobuf.
>
> Thoughts?
> Josh
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] How to speedup serialization and deserialization process?

[Ravi]

I have defined the Protocol Buffers message file as follows:

syntax = "proto3";
package Tutorial;
import "google/protobuf/timestamp.proto";
option csharp_namespace = "Tutorial";

message PointCloud {
  int32 width  = 1;
  int32 height = 2;

  message Point {
float x = 1;
float y = 2;
float z = 3;
fixed32 rgb = 4;
  }
  repeated Point points = 3;
  google.protobuf.Timestamp timestamp = 4;
}

This is how I am preparing the data and serializing it in C#:
using Google.Protobuf;
using Tutorial;
using ZeroMQ;

PointCloud pointCloud = new PointCloud();
pointCloud.Height = Height
pointCloud.Width = Width;
pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);

for (var index = 0; index < POINTS3D_COUNT; index++) {
  PointCloud.Types.Point point = new PointCloud.Types.Point {
X = points3D[index].X,
Y = points3D[index].Y,
Z = points3D[index].Z,
Rgb = (uint)((red << 16) | (green << 8) | blue)
  };

  pointCloud.Points.Add(point);
}

zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));

This is how I deserialize the data in C++:
while (receive) {
  zmq::message_t msg;
  int rc = zmq_socket.recv();
  if (rc) {
Tutorial::PointCloud point_cloud;
point_cloud.ParseFromArray(msg.data(), msg.size());
  }point_cloud.ParseFromArray(msg.data(), msg.size())
}

I am able to get the data back properly. However, the serialization and 
deserialization process seems slow. 

   - I used *System.Diagnostics.Stopwatch *in C# and noticed that 
   *pointCloud.ToByteArray()* is taking 100ms approximately.
   - Similarly I used *std::chrono::steady_clock::now()* in C++ and noticed 
   that *point_cloud.ParseFromArray(msg.data(), msg.size())* is taking 96ms 
   approximately.
   - Just for information, the length of the byte array is roughly 
   3,784,000.
   

*I want to know how to speed up serialization and deserialization process?*

-
Thanks
Ravi

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.