[protobuf] How to integrate c++ protobuf (3.4) with Android NDK?

[Ilya Tarakanov]

How to compile static library for Android in Ubuntu 16.04? I want to use it 
with Cocos2d. 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

['Feng Xiao' via Protocol Buffers]

On Wed, Dec 20, 2017 at 4:18 PM, Behrang Saeedzadeh 
wrote:

> Unfortunately I don't own the actual protocol buffer models and can't
> change their definitions.
>
> What I actually want to do is this: I have a Protocol buffer model in
> byte[] format. I want to deserialize it back to Java objects but then
> replace all the ID fields in the model, including its extensions.
>
> For example, I have a "template" Employee model in byte[] format, and I
> want to create new Java objects using that template. After deserializing
> the byte array to a Java object, I want to change the UserHeader.userID and
> ID of all the skills in it.
>
> Is there a fluent API or a concise way available to do this?
>
Why not just use protobuf binary format? For example:

byte[] data = userBuilder.build().toByteArray();

UserModel.User.Builder userBuilder2 =
UserModel.User.newBuilder().mergeFrom(data);


>
>
>
> On Thursday, December 21, 2017 at 10:56:10 AM UTC+11, Feng Xiao wrote:
>>
>> JsonFormat doesn't support extensions. You can replace extensions with
>> google.protobuf.Any if you want to use the proto with JsonFormat.
>>
>> On Wed, Dec 20, 2017 at 3:39 PM, Behrang Saeedzadeh 
>> wrote:
>>
>>>
>>>
>>> *down vote**favorite*
>>> 
>>>
>>> *Cross-post from
>>> StackOverflow: 
>>> https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
>>> *
>>>
>>> I have 2 Protobuf models:
>>> User:
>>>
>>> package demo;
>>>
>>> option java_package = "com.stackoverflow.question";
>>> option java_outer_classname = "UserModel";
>>>
>>> message User {
>>>
>>> message UserHeader {
>>> required int64 userId = 1;
>>> }
>>>
>>> required UserHeader header = 1;
>>>
>>> extensions 100 to 200;}
>>>
>>> Employee:
>>>
>>> import "person.proto";
>>> package demo;
>>>
>>> option java_package = "com.stackoverflow.question";
>>> option java_outer_classname = "EmployeeModel";
>>>
>>> extend demo.User {
>>> optional EmployeeDetails details = 101;}
>>>
>>> message EmployeeDetails {
>>> required string department = 1;
>>> repeated Skill skills = 2;}
>>>
>>> message Skill {
>>> required int64 id = 1;
>>> required string name = 2;}
>>>
>>> I can create a model and serialize it to JSON using
>>> JsonFormat.printer().print(...):
>>>
>>> ExtensionRegistry registry = 
>>> ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
>>> UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
>>> userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
>>> EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
>>> EmployeeModel.EmployeeDetails.newBuilder();
>>> employeeBuilder.setDepartment("Department 1")
>>>.addSkills(EmployeeModel.Skill.newBuilder()
>>>  .setId(10_000)
>>>  .setName("Skill 10_")
>>>  .build())
>>>.addSkills(EmployeeModel.Skill.newBuilder()
>>>  .setId(11_000)
>>>  .setName("Skill 11_")
>>>  .build());
>>>
>>> userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
>>> final String json = JsonFormat.printer().print(userBuilder.build());
>>>
>>> However deserializing the generated JSON back to Java objects fails with
>>> com.google.protobuf.InvalidProtocolBufferException: Cannot find field:
>>> details in message demo.User:
>>>
>>> UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
>>> JsonFormat.parser().merge(json, userBuilder2);
>>>
>>> And there doesn't seem to be a way to pass an ExtensionRegistry to
>>> JsonFormat.parser()either.
>>>
>>> Is there a way to make this *Protobuf → JSON → Protobuf*
>>>  serialization/deserialization chain work?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.

Re: [protobuf] Re: How to speedup serialization and deserialization process?

[Marc Gravell]

I did a test locally using protobuf-net (since that is what I'm most
familiar with); to get an output of about 3,784,000 I used a count of
172000 items in the inner array - does that sound about right? Then I
tested it in a loop as per this gist: https://gist.github.com/mgravell/
10a21970531485008731d700b89ec732

The timings I get there are about 25ms to serialize, 40ms to deserialize -
although my machine is quite fast, so this may mean you're pretty much
getting "about right" numbers. What sort of numbers are you looking for
here? Note that in .NET the first run will always be slightly slower due to
JIT.

Additionally, I know nothing about the zmq overheads - are you including
the zmq cost in your numbers?

If you want to squeeze the last few drops of performance, you usually can -
for example, by looking at whether zmq allows you to pass a stream or span
or an *oversized* array - again, I'm not as familiar with the Google C#
version as I am with protobuf-net, but *if* (and it is a huge "if") the
"ToByteArray()" is essentially writing to a MemoryStream then calling
ToArray, you can probably avoid an extra blit and some allocs by providing
your own re-used memory-stream and using GetBuffer to access the oversized
array, remembering to limit to just the first .Length bytes. But again a
lot of this depends on specifics of zmq and the Google C# version. It is
almost certainly diminishing returns.

So: what numbers are you *looking to get*? what would be "acceptable"? And
how complex is your data? is what you've shown the *only* data you need to
transfer? if so, it might not be a bad candidate for fully manual explicit
serialization not involving a library - just a payload of:

Height [int32 fixed 4 bytes]
Width [int32 fixed 4 bytes]
Time [int64 fixed 8 bytes]
ElementCount [int32 fixed 4 bytes]
then for each element: 16 bytes consisting of
X [float fixed 4 bytes]
Y [float fixed 4 bytes]
Z [float fixed 4 bytes]
RGB [int32 fixed 4 bytes]

this would require manual coding, but would typically outperform other
options - but would be more brittle and would require you to be reasonably
good at IO code.

Personally, I'd probably leave it alone...

Marc



On 22 December 2017 at 13:03, Ravi  wrote:

> Any suggestions, please?
>
>
> On Wednesday, December 20, 2017 at 10:00:31 PM UTC+9, Ravi wrote:
>>
>> I have defined the Protocol Buffers message file as follows:
>>
>> syntax = "proto3";
>> package Tutorial;
>> import "google/protobuf/timestamp.proto";
>> option csharp_namespace = "Tutorial";
>>
>> message PointCloud {
>>   int32 width  = 1;
>>   int32 height = 2;
>>
>>   message Point {
>> float x = 1;
>> float y = 2;
>> float z = 3;
>> fixed32 rgb = 4;
>>   }
>>   repeated Point points = 3;
>>   google.protobuf.Timestamp timestamp = 4;
>> }
>>
>> This is how I am preparing the data and serializing it in C#:
>> using Google.Protobuf;
>> using Tutorial;
>> using ZeroMQ;
>>
>> PointCloud pointCloud = new PointCloud();
>> pointCloud.Height = Height
>> pointCloud.Width = Width;
>> pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);
>>
>> for (var index = 0; index < POINTS3D_COUNT; index++) {
>>   PointCloud.Types.Point point = new PointCloud.Types.Point {
>> X = points3D[index].X,
>> Y = points3D[index].Y,
>> Z = points3D[index].Z,
>> Rgb = (uint)((red << 16) | (green << 8) | blue)
>>   };
>>
>>   pointCloud.Points.Add(point);
>> }
>>
>> zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));
>>
>> This is how I deserialize the data in C++:
>> while (receive) {
>>   zmq::message_t msg;
>>   int rc = zmq_socket.recv();
>>   if (rc) {
>> Tutorial::PointCloud point_cloud;
>> point_cloud.ParseFromArray(msg.data(), msg.size());
>>   }point_cloud.ParseFromArray(msg.data(), msg.size())
>> }
>>
>> I am able to get the data back properly. However, the serialization and
>> deserialization process seems slow.
>>
>>- I used *System.Diagnostics.Stopwatch *in C# and noticed that
>>*pointCloud.ToByteArray()* is taking 100ms approximately.
>>- Similarly I used *std::chrono::steady_clock::now()* in C++ and
>>noticed that *point_cloud.ParseFromArray(msg.data(), msg.size())* is
>>taking 96ms approximately.
>>- Just for information, the length of the byte array is roughly
>>3,784,000.
>>
>>
>> *I want to know how to speed up serialization and deserialization
>> process?*
>>
>> -
>> Thanks
>> Ravi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the 

[protobuf] Re: How to speedup serialization and deserialization process?

[Ravi]

Any suggestions, please?

On Wednesday, December 20, 2017 at 10:00:31 PM UTC+9, Ravi wrote:
>
> I have defined the Protocol Buffers message file as follows:
>
> syntax = "proto3";
> package Tutorial;
> import "google/protobuf/timestamp.proto";
> option csharp_namespace = "Tutorial";
>
> message PointCloud {
>   int32 width  = 1;
>   int32 height = 2;
>
>   message Point {
> float x = 1;
> float y = 2;
> float z = 3;
> fixed32 rgb = 4;
>   }
>   repeated Point points = 3;
>   google.protobuf.Timestamp timestamp = 4;
> }
>
> This is how I am preparing the data and serializing it in C#:
> using Google.Protobuf;
> using Tutorial;
> using ZeroMQ;
>
> PointCloud pointCloud = new PointCloud();
> pointCloud.Height = Height
> pointCloud.Width = Width;
> pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);
>
> for (var index = 0; index < POINTS3D_COUNT; index++) {
>   PointCloud.Types.Point point = new PointCloud.Types.Point {
> X = points3D[index].X,
> Y = points3D[index].Y,
> Z = points3D[index].Z,
> Rgb = (uint)((red << 16) | (green << 8) | blue)
>   };
>
>   pointCloud.Points.Add(point);
> }
>
> zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));
>
> This is how I deserialize the data in C++:
> while (receive) {
>   zmq::message_t msg;
>   int rc = zmq_socket.recv();
>   if (rc) {
> Tutorial::PointCloud point_cloud;
> point_cloud.ParseFromArray(msg.data(), msg.size());
>   }point_cloud.ParseFromArray(msg.data(), msg.size())
> }
>
> I am able to get the data back properly. However, the serialization and 
> deserialization process seems slow. 
>
>- I used *System.Diagnostics.Stopwatch *in C# and noticed that 
>*pointCloud.ToByteArray()* is taking 100ms approximately.
>- Similarly I used *std::chrono::steady_clock::now()* in C++ and 
>noticed that *point_cloud.ParseFromArray(msg.data(), msg.size())* is 
>taking 96ms approximately.
>- Just for information, the length of the byte array is roughly 
>3,784,000.
>
>
> *I want to know how to speed up serialization and deserialization process?*
>
> -
> Thanks
> Ravi
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.