Re: [protobuf] How to handle options in compiler plugin?

['Feng Xiao' via Protocol Buffers]

On Thu, Aug 30, 2018 at 4:14 PM Ankit Patel 
wrote:

> Hi Kenton,
>
> I am trying to do something similiar. I am writing a java plugin and I
> have defined my custom service options and method options but inside my
> plugin I see I only have access to FileDescriptorProto and not the
> FileDescriptor itself. when I try to access these options as you just
> suggested from the FileDecriptorProto they are null.
>
Link the option definitions into your plugin, and when parsing
CodeGeneratorRequest in your plugin, use an ExtensionRegistry with all
option definitions registered. After that you should be able to get
FileDescriptorProtos with option values.

To get FileDescriptor, you will need to call FileDescriptor.buildFile()
yourself.


> Please recommend on what is the suggested solution to make this work.
> Thanks
>
> On Tuesday, April 20, 2010 at 5:53:37 PM UTC-7, Kenton Varda wrote:
>>
>> You do not need to compile your option declarations into protoc.  You
>> only need to compile them into your plugin binary.  Files which use your
>> options must import the .proto file that defines them.  protoc will then
>> parse the option definitions dynamically.
>>
>> For example, you could create a file php_options.proto:
>>
>>   import "google/protobuf/descriptor.proto";
>>
>>   package protobuf_php;
>>
>>   message PhpFileOptions {
>> optional bool skip_unknown = 1;
>> optional string namespace = 2;
>>   }
>>
>>   extend google.protobuf.FileOptions {
>> optional PhpFieldOptions file_opt = 1004;
>>   }
>>
>> Now users could do:
>>
>>   import "php_options.proto"
>>
>>   option (protobuf_php.file_opt.skip_unknown) = true;
>>   option (protobuf_php.file_opt.namespace) = "My/Namespace";
>>
>> In your plugin, you can read these options like:
>>
>>
>>   
>> file_descriptor->options().GetExtension(protobuf_php::file_opt).skip_unknown()
>>
>> Note that the number 1004 above is a number that I have assigned
>> specifically to you!  Usually people have to e-mail me to request these (as
>> the comments in descriptor.proto say), but rather than wait for you to
>> actually do so I went ahead and assigned you a number.  Note that you can
>> also define other kinds of options (e.g. field options); use extension
>> number 1004 for these as well.  Please let me know your project URL once it
>> has one so that I can keep track of it.
>>
>> On Tue, Apr 20, 2010 at 3:08 PM, Andrew Brampton 
>> wrote:
>>
>>> Thanks for your reply Kenton,
>>>
>>> I've been going round in circles with the documentation but completely
>>> missed the section on custom options. Ok, so that has helped a little
>>> but I'm still not sure how to apply it. So here is some more
>>> information. I have the sample addressbook.proto file and at the top
>>> it has a couple of options:
>>>
>>> option java_package = "com.example.tutorial";
>>> option java_outer_classname = "AddressBookProtos";
>>>
>>> All I want to do is add another line like so:
>>>
>>> option php_skip_unknown = true;
>>> or something like:
>>> option php_namespace = "My/Namespace";
>>>
>>> Which, in the first case, would tell my compiler to skip over unknown
>>> fields instead of storing them, and the second put the generated files
>>> in a particular namespace.
>>>
>>> Now I see from the documentation I can extend FileOptions, but where
>>> do I save the proto file with my new options in it? Would I need to
>>> recompile the protoc compiler? At the moment I'm using protoc
>>> installed from apt, and giving it the following command line:
>>>
>>> protoc --php_out . --plugin=protoc-gen-php=./protoc-gen-php my.proto
>>>
>>> Thanks for any guidance you can give me.
>>> Andrew
>>>
>>> On Tue, Apr 20, 2010 at 10:38 PM, Kenton Varda 
>>> wrote:
>>> > It sounds like you are using custom options incorrectly, independent
>>> of your
>>> > plugin.  That error message is being produced by protoc before it even
>>> tries
>>> > to call your plugin.  Please double-check the docs on custom options.
>>> If
>>> > you can't figure out what's wrong, show us the .proto code you are
>>> using to
>>> > define and then use your option.
>>> >
>>> > On Tue, Apr 20, 2010 at 4:24 AM, Andrew Brampton 
>>> wrote:
>>> >>
>>> >> Hi,
>>> >> I've nearly finished writing a compiler plugin for generating PHP
>>> >> code. I now wanted to support some additional options. However, I'm
>>> >> not sure how to begin. When I have a custom option in my proto file,
>>> >> protoc seems to die with this error:
>>> >>
>>> >> my.proto:2:8: Option "php_skip_unknown" unknown.
>>> >>
>>> >> This seems to happen before CodeGenerator's generate method is called.
>>> >> So I'm not sure how I handle this.
>>> >>
>>> >> Also, are there some test proto files, with binary data, which I can
>>> >> use to fully validate my compiler?
>>> >>
>>> >> thanks
>>> >> Andrew
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups
>>> >> "Protocol Buffers" group.
>>> >> To post to this group, send email to 

Re: [protobuf] Re: Message imports from different locations and stub generation

['Feng Xiao' via Protocol Buffers]

I have written some explanation of protoc "-I" flag and import path here:
https://github.com/xfxyjwf/protobuf/blob/protoc/docs/protoc.md

 To run protoc correctly:
1. Determine the import path for every one of your .proto files. In your
case, I think it should be "dms/v1/dms.proto" and "metric/protobuf.proto".
2. Use the chosen import paths in all your .proto files. E.g., change your
dms.proto to use 'import "metric/protobuf.proto";' instead.
3. Construct "--proto_path" arguments where protoc can find the imported
files using the chosen import paths. In your case, it should be "-I." which
can be omitted because it's the default.

Putting everything together:
a. change dms.proto to do 'import "metric/protobuf.proto";'
b. run 'protoc dms/v1/dms.proto --go_out=..."

and it should work.

The other error:
--go_out: protoc-gen-go: The system cannot find the file specified.

is about missing the go protoc plugin. You should be able to fix that by
installing the right plugin and make sure it can be found in PATH. If in
doubt, check:

$ which protoc-gen-go

If it prints nothing, your go protoc plugin is not installed correctly.

On Tue, Aug 28, 2018 at 5:05 AM Dzintars Klavins 
wrote:

> And when i am using:
>>
>
> protoc --proto_path=dms/v1 dms.proto --go_out=plugins=grpc:D:\Code\Go\src
>
> i get a error:
>
> protobuf.proto: File not found.
> dms.proto: Import "protobuf.proto" was not found or had errors.
> dms.proto:17:66: "domain.protobuf.Empty" is not defined.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] how to build executable protoc-gen-javalite

['Feng Xiao' via Protocol Buffers]

Can you help creat a github issue for this?

On Wed, Aug 22, 2018 at 06:07 Luan Cooper  wrote:

> Hi
>
> *We need to build protoc/protoc-gen-javalite ourself, because we changed
> some java lite code generate behavior*
>
> I followed Github C++ Installation Wiki
> https://github.com/protocolbuffers/protobuf/blob/21fc01b63c7f60bae4e230c817a82bd8571a4a70/src/README.md
> $ ./autogen.sh
> $ ./configure
> $ make
> $ make check
> $ sudo make install
>
> and protoc works well, but javalite does not work
>
> this is how I use javalite
>
> $ protoc --plugin=/usr/local/lib --javalite_out=tracker-builder-lite/src/
> main/java/ tracker.proto
>
> protoc-gen-javalite: program not found or is not executable
>
> --javalite_out: protoc-gen-javalite: Plugin failed with status code 1.
>
> this is where I install proto lib to
>
> $ l /usr/local/lib/libproto*
>
> -rwxr-xr-x  1 root  admin   467K  8 22 21:01 /usr/local/lib/libprotobuf-
> lite.15.dylib
>
> -rw-r--r--  1 root  admin   5.4M  8 22 21:01 /usr/local/lib/libprotobuf-
> lite.a
>
> lrwxr-xr-x  1 root  admin25B  8 22 21:01 /usr/local/lib/libprotobuf-
> lite.dylib -> libprotobuf-lite.15.dylib
>
> -rwxr-xr-x  1 root  admin   979B  8 22 21:01 /usr/local/lib/libprotobuf-
> lite.la
>
> -rwxr-xr-x  1 root  admin   3.2M  8 22 21:01 /usr/local/lib/libprotobuf.
> 15.dylib
>
> -rw-r--r--  1 root  admin54M  8 22 21:01 /usr/local/lib/libprotobuf.a
>
> lrwxr-xr-x  1 root  admin20B  8 22 21:01 /usr/local/lib/libprotobuf.
> dylib -> libprotobuf.15.dylib
>
> -rwxr-xr-x  1 root  admin   949B  8 22 21:01 /usr/local/lib/libprotobuf.la
>
> -rwxr-xr-x  1 root  admin   2.7M  8 22 21:01 /usr/local/lib/libprotoc.
> 15.dylib
>
> -rw-r--r--  1 root  admin93M  8 22 21:01 /usr/local/lib/libprotoc.a
>
> lrwxr-xr-x  1 root  admin18B  8 22 21:01 /usr/local/lib/libprotoc.
> dylib -> libprotoc.15.dylib
>
> -rwxr-xr-x  1 root  admin   967B  8 22 21:01 /usr/local/lib/libprotoc.la
>
> Is there anything wrong ? What did I miss?
>
> Can anybody help me ?
>
>
> Cooper
> Shanghai Xingin Ltd
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Importing any.proto in .proto files

['Feng Xiao' via Protocol Buffers]

On Tue, Aug 21, 2018 at 8:27 AM akshay patil  wrote:

> Thanks a lot for the suggestions. I can import the any.proto correctly
> into my proto files.
>
> Wanted to ask about importing other proto files not provided by
> google protobuf but different python libraries.
> eg. tensorflow/core/framework/tensor.proto  or onnx/onnx.proto
>
Protocol compiler can't detect those files. You will need to use the
--proto_path option to specify where those protos are located.

For example, suppose onnx/onnx.proto is at /some/path/onnx/onnx.proto, try:

protoc --proto_path=/some/path your_proto.proto


>
> i.e
>
> import "onnx/onnx.proto";
>
> message Test {
> onnx.TensorProto var = 1;
> }
>
>
> Do I have to copy over the files and use them or the protoc compiler can
> detect that from my site-packages?
> Same is the case for other proto files eg. import
> "tensorflow/core/framework/tensor.proto"
>
> On Tuesday, August 21, 2018 at 12:16:12 AM UTC-5, Feng Xiao wrote:
>>
>>
>>
>> On Mon, Aug 20, 2018 at 10:02 PM Siddharth 
>> wrote:
>>
>>> Hi feng,
>>>
>>> Can we convert JSON object into Google.protobuf.Any format? If yes, can
>>> you provide an example. Thanks for the help.
>>>
>> Check out the Any unittest examples:
>>
>> https://github.com/google/protobuf/blob/master/java/util/src/test/java/com/google/protobuf/util/JsonFormatTest.java#L835
>>
>>
>>>
>>> Thanks,
>>> Sid
>>>
>>> On Tue, Aug 21, 2018, 10:27 AM 'Feng Xiao' via Protocol Buffers <
>>> prot...@googlegroups.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Aug 20, 2018 at 9:24 PM akshay patil  wrote:
>>>>
>>>>> I have protoc installed which I can run via command line. I did
>>>>> install protobuf using conda and when I check in my site-packages in my
>>>>> python environment under ` site-packages/google/protobuf ` I do not see 
>>>>> any
>>>>> *.proto files
>>>>>
>>>> They are not in your python site-packages, but should be installed
>>>> along side with the protoc binary. Check where your protoc is and look
>>>> around to see if the protos are install there.
>>>>
>>>> Try compile the following proto and see if protoc can find the right
>>>> imports:
>>>>
>>>> syntax = "proto3";import "google/protobuf/any.proto";
>>>>
>>>> message Test {
>>>>google.protobuf.Any var1 = 1;}
>>>>
>>>> If you are unsure whether your proto definition is valid (your original
>>>> one has a invalid import path and incomplete type name for Any), try it
>>>> here:
>>>> https://unpkg.com/google-protocol-compiler@0.0.6/index.html
>>>>
>>>> If you can get it working it on this site, you should be able to get it
>>>> working locally as well.
>>>>
>>>>
>>>>>
>>>>> On Monday, August 20, 2018 at 11:09:10 PM UTC-5, Feng Xiao wrote:
>>>>>>
>>>>>> Have you install protocol compiler, i.e., protoc? These protos should
>>>>>> be installed along with the protoc binary if you get it from our release
>>>>>> page.
>>>>>>
>>>>>> On Mon, Aug 20, 2018 at 8:13 PM akshay patil 
>>>>>> wrote:
>>>>>>
>>>>>>>  I am trying to define my own messages representing python classes
>>>>>>> that I want to serialize. However, I need to import any.proto or
>>>>>>> timestamp.proto in my .proto files to represent certain messages
>>>>>>>
>>>>>>> syntax = "proto3";import "/google/protobuf/any.proto";
>>>>>>>
>>>>>>> message Test {
>>>>>>>Any var1 = 1;}
>>>>>>>
>>>>>>>
>>>>>>> However, seems like the protobuf installation does not come with any
>>>>>>> .proto files. I am using python 3.5 and have protobuf 3.5.2 installed. I
>>>>>>> tried to look under google/protobuf/ but just found the compiled
>>>>>>> files like ant_pb2.py and timestamp_pb2.py. I am following docs
>>>>>>> here to import any.proto -
>>>>>>> https://developers.google.com/protocol-buffers/docs/proto3#any
>>>>>>>
>>>>>>> --
>

[protobuf] Re: Protobuf is moving to its own github organization!

['Feng Xiao' via Protocol Buffers]

The move is complete. Protobuf is now live at the new location:
https://github.com/protocolbuffers/protobuf/issues

Everything should continue to work (all issues/PRs will be moved, all links 
will be redirect, git clone with the old location will continue to work).

If there are any issues, please comment on this issue:
https://github.com/protocolbuffers/protobuf/issues/4796

On Tuesday, June 19, 2018 at 5:13:36 PM UTC-7, Feng Xiao wrote:
>
> Dear Protobuf users,
>
> Since 3.0.0 release, we have brought official support for several new 
> languages into protobuf. Due to that growth, we are considering moving the 
> project to its own github organization. This can help us better manage the 
> project and improve the support we provide to the community.
>
> An organization will allow us to add support for more languages outside 
> the main repo. We can include repos for helpful tools/utilities that are 
> not suitable to bundled with the protobuf core library. Having them all 
> under a single organization can help make them more discoverable.
>
> The github organization we are considering moving to is:
> https://github.com/protocolbuffers 
> <https://www.google.com/url?q=https://github.com/protocolbuffers=D=152945679959=AFQjCNFzMNLL69HlvnZLwISb_0Gbe73Nnw>
>
> After the move, the main protobuf repository will be: 
> https://github.com/protocolbuffers/protobuf 
> <https://www.google.com/url?q=https://github.com/protocolbuffers/protobuf=D=152945679959=AFQjCNH4yeH1AK-5Z-_y8BAiNjTx3kIozw>
> .
>
> While we believe this move will be seamless for us and the users as 
> everything will be redirected, we want to make sure we have considered all 
> corner cases. Please help us with that by filling up this survey:
>
>
> https://docs.google.com/forms/d/e/1FAIpQLSeH1ckwm6ZrSfmtrOjRwmF3yCSWQbbO5pTPqPb6_rUppgvBqA/viewform
>
> Suggestions on how we should structure the organization are welcome.
>
> - Google Protobuf Team
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Importing any.proto in .proto files

['Feng Xiao' via Protocol Buffers]

On Mon, Aug 20, 2018 at 10:02 PM Siddharth 
wrote:

> Hi feng,
>
> Can we convert JSON object into Google.protobuf.Any format? If yes, can
> you provide an example. Thanks for the help.
>
Check out the Any unittest examples:
https://github.com/google/protobuf/blob/master/java/util/src/test/java/com/google/protobuf/util/JsonFormatTest.java#L835


>
> Thanks,
> Sid
>
> On Tue, Aug 21, 2018, 10:27 AM 'Feng Xiao' via Protocol Buffers <
> protobuf@googlegroups.com> wrote:
>
>>
>>
>> On Mon, Aug 20, 2018 at 9:24 PM akshay patil  wrote:
>>
>>> I have protoc installed which I can run via command line. I did install
>>> protobuf using conda and when I check in my site-packages in my python
>>> environment under ` site-packages/google/protobuf ` I do not see any
>>> *.proto files
>>>
>> They are not in your python site-packages, but should be installed along
>> side with the protoc binary. Check where your protoc is and look around to
>> see if the protos are install there.
>>
>> Try compile the following proto and see if protoc can find the right
>> imports:
>>
>> syntax = "proto3";import "google/protobuf/any.proto";
>>
>> message Test {
>>google.protobuf.Any var1 = 1;}
>>
>> If you are unsure whether your proto definition is valid (your original
>> one has a invalid import path and incomplete type name for Any), try it
>> here:
>> https://unpkg.com/google-protocol-compiler@0.0.6/index.html
>>
>> If you can get it working it on this site, you should be able to get it
>> working locally as well.
>>
>>
>>>
>>> On Monday, August 20, 2018 at 11:09:10 PM UTC-5, Feng Xiao wrote:
>>>>
>>>> Have you install protocol compiler, i.e., protoc? These protos should
>>>> be installed along with the protoc binary if you get it from our release
>>>> page.
>>>>
>>>> On Mon, Aug 20, 2018 at 8:13 PM akshay patil  wrote:
>>>>
>>>>>  I am trying to define my own messages representing python classes
>>>>> that I want to serialize. However, I need to import any.proto or
>>>>> timestamp.proto in my .proto files to represent certain messages
>>>>>
>>>>> syntax = "proto3";import "/google/protobuf/any.proto";
>>>>>
>>>>> message Test {
>>>>>Any var1 = 1;}
>>>>>
>>>>>
>>>>> However, seems like the protobuf installation does not come with any
>>>>> .proto files. I am using python 3.5 and have protobuf 3.5.2 installed. I
>>>>> tried to look under google/protobuf/ but just found the compiled
>>>>> files like ant_pb2.py and timestamp_pb2.py. I am following docs here
>>>>> to import any.proto -
>>>>> https://developers.google.com/protocol-buffers/docs/proto3#any
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Protocol Buffers" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to protobuf+u...@googlegroups.com.
>>>>> To post to this group, send email to prot...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Importing any.proto in .proto files

['Feng Xiao' via Protocol Buffers]

Have you install protocol compiler, i.e., protoc? These protos should be
installed along with the protoc binary if you get it from our release page.

On Mon, Aug 20, 2018 at 8:13 PM akshay patil  wrote:

>  I am trying to define my own messages representing python classes that I
> want to serialize. However, I need to import any.proto or timestamp.proto
> in my .proto files to represent certain messages
>
> syntax = "proto3";import "/google/protobuf/any.proto";
>
> message Test {
>Any var1 = 1;}
>
>
> However, seems like the protobuf installation does not come with any
> .proto files. I am using python 3.5 and have protobuf 3.5.2 installed. I
> tried to look under google/protobuf/ but just found the compiled files
> like ant_pb2.py and timestamp_pb2.py. I am following docs here to import
> any.proto - https://developers.google.com/protocol-buffers/docs/proto3#any
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to workaround enum sibling issue in Java

['Feng Xiao' via Protocol Buffers]

On Fri, Jul 20, 2018 at 10:22 PM jiandai via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hi there,
>
> I am creating a protobuf message to match with a JSON data in legacy
> application. The problem is that legacy JSON has mulitple enum data, and
> some of them have the same enum name, like "NONE".
>
> I know protobuf requires enum name uniqueness on the scope of package and
> protobuf convert to JSON based on enum name. I was wondering, in my case
> here, is there any way that I can,
>
> 1. workaround protobuf enum sibling constriction and build the enums with
> the same name in a same package? Or,
> 2. request protobuf converter to convert enum into different name, and
> vice versa?
>
You can put enums with the same name in the same package but they have to
be contained in separate message types. I.e.:

message Foo {
  enum MyEnum {
NONE = 0;
  }
}

message Bar {
  enum MyEnum {
NONE = 0;
  }
}

Because we only use the short name in JSON, the above 2 enums will end up
with the same JSON value "NONE".


>
> Many thanks in advance for the nice help.
>
> Jian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Java API: creating message by name

['Feng Xiao' via Protocol Buffers]

On Mon, Jul 9, 2018 at 2:45 PM John Lilley  wrote:

> Well, apparently I am really off base.  Given a Descriptor I cannot figure
> out how to create the right message. I *thought* this was the right
> approach:
>  Descriptors.Descriptor desc = // look up the descriptor
>  Builder builder = desc.toProto().newBuilderForType();
>  Message message = builder.mergeFrom(requestBytes).build();
>
>
> But no.  Can anyone help me with this?  I need to go from the full name of
> a message to its builder and I'm not finding anything like MessageFactory()
> or DescriptorPool in Java.
>
There is no DecriptorPool/MessageFactory in Java because there is no easy
way to automatically build such a DescriptorPool/MessageFactory with all
protos in the class path. There are a few options for Java:
1. Build a registry from message name to proto message type manually.
2. Use protoc to output a FileDescriptorSet for all of your .proto files,
and use this FileDescriptorSet to build a DescriptorPool-like structure
yourself. You will be using FileDescriptor.buildFile() to convert
FileDescriptorProtos in the FileDescriptorSet to FileDescriptor and then
traverse through the FileDescriptors to build a map from messageName to
Descriptor.
3. Write a protoc plugin to produce Java code to create a registry for all
of your .proto files.

All the above options are used by some projects inside Google. (2), (3) are
integrated into our build system with runtime library support and will take
quite some effort to recreate from scratch. Your best option is probably
(1).


> Thanks
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] VS2017 issue (#4773/#4679) : seeking advice on workaround

['Feng Xiao' via Protocol Buffers]

You can use the release version of protobuf as a workaround. The bug
doesn't happen in release build.

On Wed, Jun 27, 2018 at 1:38 PM John Lilley  wrote:

> I am starting a POC project involving protobuf, which must work on
> Windows.  I've run into issues #4773 and #4679.  Is there any workaround
> for VS2017?  If not, what is the best advice to continue development on
> Windows?
> Thanks
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Performance of JsonFormat.printer() for converting proto to json.

['Feng Xiao' via Protocol Buffers]

On Mon, Jun 25, 2018 at 10:19 PM Jiten Gupta  wrote:

> Hi Feng
>
> Thanks for quick reply. Is there any alternative to parse proto to JSON
> which performance is a concern.
>
There isn't an official alternative.


>
> Regards
> Jiten
>
> On Mon, Jun 25, 2018 at 10:51 PM, Feng Xiao  wrote:
>
>>
>>
>> On Mon, Jun 25, 2018 at 10:02 AM Jiten Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> I am using google proto and we need to convert proto payload   to JSON
>>> to be sent out for monitoring and logging purpose. Size of the payload is
>>> around 10KB. We are using JsonFormat.printer().print for proto to json
>>> conversion.
>>>
>>> We are running on 4 cores CPU and we need to do around 30-40 such
>>> conversion at a time.
>>>
>>> When I do profiling, I see there is quite a bit CPU time is spent on
>>> proto to JSON converson. ( Screen shot attached)
>>>
>>> I JsonFormat.printer().print best way to do proto to json conversion ?
>>>
>> It's the official way, but maybe not the best performant way. If
>> performance is a concern, you probably should consider using proto binary
>> directly?
>>
>>
>>>
>>> REgards
>>> Jiten
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Performance of JsonFormat.printer() for converting proto to json.

['Feng Xiao' via Protocol Buffers]

On Mon, Jun 25, 2018 at 10:02 AM Jiten Gupta  wrote:

> Hi,
>
> I am using google proto and we need to convert proto payload   to JSON to
> be sent out for monitoring and logging purpose. Size of the payload is
> around 10KB. We are using JsonFormat.printer().print for proto to json
> conversion.
>
> We are running on 4 cores CPU and we need to do around 30-40 such
> conversion at a time.
>
> When I do profiling, I see there is quite a bit CPU time is spent on proto
> to JSON converson. ( Screen shot attached)
>
> I JsonFormat.printer().print best way to do proto to json conversion ?
>
It's the official way, but maybe not the best performant way. If
performance is a concern, you probably should consider using proto binary
directly?


>
> REgards
> Jiten
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Protobuf is moving to its own github organization!

['Feng Xiao' via Protocol Buffers]

Dear Protobuf users,

Since 3.0.0 release, we have brought official support for several new 
languages into protobuf. Due to that growth, we are considering moving the 
project to its own github organization. This can help us better manage the 
project and improve the support we provide to the community.

An organization will allow us to add support for more languages outside the 
main repo. We can include repos for helpful tools/utilities that are not 
suitable to bundled with the protobuf core library. Having them all under a 
single organization can help make them more discoverable.

The github organization we are considering moving to is:
https://github.com/protocolbuffers 


After the move, the main protobuf repository will be: 
https://github.com/protocolbuffers/protobuf 

.

While we believe this move will be seamless for us and the users as 
everything will be redirected, we want to make sure we have considered all 
corner cases. Please help us with that by filling up this survey:

https://docs.google.com/forms/d/e/1FAIpQLSeH1ckwm6ZrSfmtrOjRwmF3yCSWQbbO5pTPqPb6_rUppgvBqA/viewform

Suggestions on how we should structure the organization are welcome.

- Google Protobuf Team

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Strings and Arenas

['Feng Xiao' via Protocol Buffers]

On Thu, Jun 7, 2018 at 8:01 AM aram  wrote:

> I am working on the same problem in my project (arena-allocated strings),
> and came across this topic. It is 2.5 years old now, so I wonder if
> arena-allocated strings or StringPiece class is going to be included in
> official protobuf releases any time soon?
>
Unfortunately this has happened yet. The main issue is how to adopt abseil
in protobuf library to be able to use the new string_view type and Cord. We
haven't figured that out yet. You can follow-up on
https://github.com/google/protobuf/issues/1896 and we will post updates
there.


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto3 Default Value Conformance

['Feng Xiao' via Protocol Buffers]

On Tue, May 29, 2018 at 11:30 AM Cyrus Katrak 
wrote:

> I am implementing a protobuf implementation for a new language and I am
> failing the conformance test named:
>
>
> Required.Proto3.ProtobufInput.RepeatedScalarSelectsLast.UINT64.ProtobufOutput
>
> The test input appears to be an optional (non-repeated) field that appears
> multiple times on the wire, with the last occurrence having a value of zero
> (the default value for the field).
>
> The expected test output appears to be the same field, appearing once on
> the wire, with a value of zero.
>
> I am confused why this is the expected output given the Langauge Guide for
> proto3 states:
> > Also note that if a scalar message field is set to its default, the
> value will not be serialized on the wire.
>
> Any clarification would be much appreciated!
>
The conformance test specifies output in text format, but it does not
compare your output against the text format directly. Instead, it will
parse the text data into proto, and serialize it again. In this case, your
output will be compared to an empty byte array and your implementation has
failed this test.


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Errors building Protobuf 3.5.1 with Visual Studio 2005

['Feng Xiao' via Protocol Buffers]

On Tue, May 29, 2018 at 9:42 AM Eric Pederson  wrote:

> Hi all:
>
> I am trying to build Protobuf (3.5.1) using Visual Studio 2005 (I know,
> it's ancient, but we are stuck with it for now).  I am getting these errors:
>
>
> Compiling...
> main.cc
> src\google/protobuf/stubs/hash.h(309) : error C2977: 'stdext::hash_map' : too 
> many template arguments
> C:\Program Files (x86)\Microsoft Visual Studio 
> 8\VC\include\hash_map(89) : see declaration of 'stdext::hash_map'
> src\google/protobuf/stubs/hash.h(319) : see reference to class template 
> instantiation 'google::protobuf::hash_map' being compiled
> src\google/protobuf/stubs/hash.h(309) : error C2955: 'stdext::hash_map' : use 
> of class template requires template argument list
> C:\Program Files (x86)\Microsoft Visual Studio 
> 8\VC\include\hash_map(89) : see declaration of 'stdext::hash_map'
> src\google/protobuf/stubs/hash.h(311) : error C2977: 'stdext::hash_map' : too 
> many template arguments
> C:\Program Files (x86)\Microsoft Visual Studio 
> 8\VC\include\hash_map(89) : see declaration of 'stdext::hash_map'
>
>
> Is there a workaround for this?   Or is Protobuf 3 not compatible with VS
> 2005?   I am able to build Protobuf 2.6.1 with VS 2005.
>
If you have to use vs2005, protobuf 2.6.1 might be the latest version you
can use. From protobuf 3.0.0, we require at least vs 2008 to build.


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Do protoc 3.5.1 and 2.6.1 generate compatible output?

['Feng Xiao' via Protocol Buffers]

On Tue, May 22, 2018 at 1:05 PM Tomáš Volf  wrote:

> We are thinking about switching to newer protocol buffer (current 3.5.1)
> but keeping on using proto2 syntax. Under this condition (proto2), does
> library created using 3.5.1 generate compatible messages with one compiled
> using 2.6.1?
>
If you are talking about the wire format, yes, proto 3.5.1 is fully
compatible with 2.6.1. If you are using the same .proto file, they should
generate exactly the same wire format data given the same proto message.


>
> From what I understand it should but I want to make sure.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Making message reader strict, rejecting non-canonical messages

['Feng Xiao' via Protocol Buffers]

On Wed, May 16, 2018 at 7:39 AM Dmitry Timofeev 
wrote:

> Hi,
>
> I consider if Protocol Buffers can be possibly used in an application that
> requires canonical representation of messages coming from external source.
>
> The encoding and proto3 guide [1, 2] include several requirements for a
> parser that make it accept non-canonical data (this list is probably not
> exhaustive):
>   - Message fields may appear in any order
>   - There are might be multiple instances of the same *non-repeated* field
>   - Message may contain unknown fields
>   - ¿Default values of primitives may appear on the wire
>   - Map entries may appear in any order
>   - Repeated fields of primitives may be packed or unpacked.
>
> 1. Is there any natural way to extend the parser with checks of canonical
> form?
>
No.


> By "natural" I mean a compiler and/or runtime plugin, something that does
> not require a fork of the project.
>
2. If not, does such optional feature make sense in Protocol Buffers? Would
> you accept an option that makes the generated reader code 'strict',
> rejecting non-canonical representations, and, consequently, not
> forward-compatible?
>
Also no here. Compatibility is one of the main reasons to use protobuf
because it allows you to evolve your protocol without breaking anyone in a
complex system. If you don't need compatibility at all (i.e., you will
never change your protocol), using a C++ struct will be much more
performant than protobuf because you can skip the whole
parsing/serialization cost.

There is a way to mimic the behavior you want though:
1. parse the input data to a proto message
2. check if the proto message has any unknown fields; if any, report error
3. serialize the proto message using deterministic serialization (
https://github.com/google/protobuf/blob/master/src/google/protobuf/io/coded_stream.h#L842
)
4. compare the serialized data against the input data; if they match, the
input data is in the "canonical form"; if not, report error

It will incur an additional serialization cost, but can get you close
enough to the canonical form.

>
> Thanks,
> Dmitry
>
> [1] https://developers.google.com/protocol-buffers/docs/encoding
> [2] https://developers.google.com/protocol-buffers/docs/proto3
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is proto3 specification still beta?

['Feng Xiao' via Protocol Buffers]

On Mon, Apr 30, 2018 at 6:37 AM Gary Pennington  wrote:

> This is probably a silly question, but:
>
> "Is protocol buffers language 3 a finished specification?"
>
> At various points on the gRPC pages, e.g.:
> https://grpc.io/docs/tutorials/basic/java.html, the text specifically
> mentions that "the proto3 version of the protocol buffers language, which
> is currently in beta release".
>
> I tried trawling around the protocol buffers website and can't find a
> categorical statement.
>
> Can anyone provide clarification?
>
Proto3 has been used in production for quite a while. It's in GA status.
That gPRC page needs some updates.


>
> Thanks,
>
> Gary
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to use a different protoc version?

['Feng Xiao' via Protocol Buffers]

On Thu, Apr 26, 2018 at 6:22 PM elmer  wrote:

> Hi there,
>
> When I want use the gRPC and make the C++ files, I got the following error:
>
> error: This file was generated by an older version of protoc which is
> error: incompatible with your Protocol Buffer headers. Please
> error: regenerate this file with a newer version of protoc.
>
> It seems that I have installed different versions in my system (macOS). I
> checked my protoc version using the command "protoc --version", I got
>
> libprotoc 3.2.0
>
> Then, I try to upgrade my protoc using the "brew upgrade protobuf", I got
>
> protobuf 3.5.1_1 already installed
>
> So I also noticed the version 3.5.1 protoc is in 
> /usr/local/opt/protobuf/share/doc/protobuf
> . And in /usr/local/opt there is a protobuf folder and a protobuf@3.5
> folder. So I think the error I got when I make the C++ file is caused
> because the 3.5.1 version of protoc is needed but the 3.2.0 version is used.
>
> Am I right? If yes, how can I switch from 3.2.0 to the 3.5.1 version?
>
I suggest you uninstall the versions you don't need and only keep one
version at a time. That would make things easier.


>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default value not accessible in proto3

['Feng Xiao' via Protocol Buffers]

On Thu, Apr 26, 2018 at 2:12 AM love sidhu  wrote:

> Hi,
>
> This is how my response message is defined in proto file.
>
> message Response{
>  bool isOk = 1;
> }
>
>
> When I set this bool value in gRPC server to false, I'm not receiving it
> on client side
>
If you receive an empty Response message, it means the value of the isOk
field is the default which is false. Why do you care to send it over the
wire?


> but If I set this bool value to true I receive it on client side.
>
> I read this link:
> https://stackoverflow.com/questions/41089035/protobuf-default-values
> and saw that default values are not sent on wire and in case of boolean
> variables, False is default value. I don't know how true it is?
>
>
> Is there some general proper way of how to handle this for all types of
> variables(int32, bool etc.)
>

> Thanks,
> Lovepreet
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] return to arena (C++)

['Feng Xiao' via Protocol Buffers]

Arena is not supposed to be used as a memory pool. The recommended pattern
is to use one arena per rpc request/response. You can tweak that to one
arena multiple rpc request/response but the number of messages you want to
put into the arena should be limited. It won't work well if you use one
arean the entire lifetime of the program.

On Mon, Apr 9, 2018 at 8:07 AM John Scholvin 
wrote:

> Can someone confirm for me that there's no way to return a message's
> memory to the arena when you're done with it? I was expecting the arena to
> work something like boost::pool where I can give memory back when I'm done
> with it, but it appears that's not part of the API here.
>
> My use case is a large number of messages with relatively short lifespans.
> What happens is that the arena just grows in size until application exit,
> which won't work. If there's another way to support this workflow without
> all the heap traffic of the default implementation, I'd love to hear about
> it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf arena, perfomance

['Feng Xiao' via Protocol Buffers]

On Fri, Apr 6, 2018 at 5:02 AM  wrote:

> Hello, everybody.
> I would like to detail some moments about protobuf arena.
> In my project there is a structure with ~100 fields.
> I compared elapsed time for parse and serialize methods with arena and
> without it.
>
> 1.Arena:
> 
> google::protobuf::Arena arena;
> ProtoStruct message  =
> google::protobuf::Arena::CreateMessage();
> while (is_testing) {
>
> packet.ParseFromArray(...)
> ...read fields, change fields
> message.SerializeToArray(...);
>
> }
>
> 2.No arena:
>
> ProtoStruct message
> while (is_testing) {
>
> packet.ParseFromArray(...)
> ...read fields, change fields
> message.SerializeToArray(...);
>
> }
>
> The results didn't show any difference.
> I used 3.3.1.189 version and cloned from master branch, which allocates
> strings in arena.
>
> From this topic (https://github.com/google/protobuf/issues/4327)
> I see that there is no benefits from arena usage now.
> Is It true or I do something wrong?
>
Right now opensource protobuf arena only works well when you have a deep
message hierarchy but without string fields (which is very unlikely in most
common use cases). I suggest you wait for the support of STRING_PIECE to be
in place before trying arena again.


>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Building with debug symbols?

['Feng Xiao' via Protocol Buffers]

On Sun, Apr 1, 2018 at 3:20 AM Aaron Glenn  wrote:

> Hi,
>
> I am the next foolish person in line trying to get any version of protobuf
> 3.x compiled and tests passing on OpenBSD. Release 3.5.1 compiles without
> much fuss but when running tests, a segmentation fault occurs at the start
> of JsonStreamParserTest.
>
> [--] 70 tests from JsonStreamParserTest
> [ RUN  ] JsonStreamParserTest.SimpleTrue
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0cd0eddfdd54 in
> testing::Sequence::AddExpectation(testing::Expectation const&) const ()
> from /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgmock.so.0.0
> (gdb) bt
> #0  0x0cd0eddfdd54 in
> testing::Sequence::AddExpectation(testing::Expectation const&) const ()
> from /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgmock.so.0.0
> #1  0x0cceb7261033 in
> testing::internal::MockSpec (google::protobuf::StringPiece, bool)>::InternalExpectedAt(char const*,
> int, char const*, char const*) ()
> #2  0x0cceb7266825 in
> google::protobuf::util::converter::ExpectingObjectWriter::RenderBool(google::protobuf::StringPiece,
> bool) ()
> #3  0x0cceb72805f6 in
> google::protobuf::util::converter::JsonStreamParserTest_SimpleTrue_Test::TestBody()
> ()
> #4  0x0cd1178b8d81 in void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*) ()
>from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #5  0x0cd1178af48a in testing::Test::Run() () from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #6  0x0cd1178af5d8 in testing::TestInfo::Run() () from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #7  0x0cd1178af6b5 in testing::TestCase::Run() () from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #8  0x0cd1178af9c8 in testing::internal::UnitTestImpl::RunAllTests()
> () from /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #9  0x0cd1178b92f1 in bool
> testing::internal::HandleExceptionsInMethodIfSupported bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*) ()
>from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #10 0x0cd1178afca7 in testing::UnitTest::Run() () from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgtest.so.0.0
> #11 0x0cd19db06519 in main () from
> /usr/ports/pobj/protobuf-3.5.1/build-amd64/src/.libs/libgmock_main.so.0.0
> #12 0x0cceb6e05026 in _start ()
> (gdb)
>
> I am woefully out of my element, and concerted Google/GitHub/mailing list
> searching did not produce any actionable data for me. I fully understand no
> developers have access to OpenBSD machines in their testing infra (and
> while that can be arranged to whatever specifics required, I understand the
> project is not interested in having an OpenBSD machine in their infra to
> reproduce this on).
>
> My questions are:
>
> 1) how can I build with debug symbols to get `backtrace full` output
> working as intended? where do I put the -g flag to get passed to gcc in the
> CMake makefile system? do I pass this with ./configure?
>
If you are using cmake, do:
cmake -DCMAKE_BUILD_TYPE=Debug

or if you want to pass specific C++ flags:
cmake -DCMAKE_CXX_FLAGS=-g

If you are using autotools (i.e., ./configure && make), do:
./configure CXXFLAGS=-g


> 2) would anyone be so kind as to venture a guess or three where I should
> be looking or attempting next?
>
The issue could be in gmock, or the way gmock is built. I suggest you try
different ways to build the tests and see if that makes any difference. For
example, try build the tests statically, try both autotools build and cmake
build.


>
> Again, I understand no one can reproduce this due to lack (and seemingly
> desire) of having an OpenBSD machine. I am that someone. I am highly
> motivated to get this working. I humbly ask for any and all assistance
> anyone can be bothered to provide.
>

> Many thanks,
> Aaron
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at 

Re: [protobuf] package com.google.protobuf does not exist error while trying to build ProtoBuf Java component

['Feng Xiao' via Protocol Buffers]

Try following the instructions in examples/README.md and define CLASSPATH
properly:
https://github.com/google/protobuf/tree/master/examples#java

On Sun, Apr 1, 2018 at 10:19 AM Prag Thiru  wrote:

> Previously I was able to run make all and make java. I did a make clean after
> which make java fails with com/example/tutorial/AddressBookProtos.java:95:
> error: package com.google.protobuf does not exist.
>
> I followed this SO post
> ,
> but it fails
> 
> .
>
>
> How can I fix this error? Thanks!
>
>
> examples $protoc --version
> libprotoc 3.5.1
>
> examples $make java
> make: Nothing to be done for `java'.
>
> examples $make clean
> rm -f add_person_cpp list_people_cpp add_person_java list_people_java 
> add_person_python list_people_python
> rm -f javac_middleman AddPerson*.class ListPeople*.class 
> com/example/tutorial/*.class
> rm -f protoc_middleman addressbook.pb.cc addressbook.pb.h addressbook_pb2.py 
> com/example/tutorial/AddressBookProtos.java
> rm -f *.pyc
> rm -f protoc_middleman_go tutorial/*.pb.go add_person_go list_people_go
> rmdir tutorial 2>/dev/null || true
> rmdir com/example/tutorial 2>/dev/null || true
> rmdir com/example 2>/dev/null || true
> rmdir com 2>/dev/null || true
>
> examples $make java
> protoc $PROTO_PATH --cpp_out=. --java_out=. --python_out=. addressbook.proto
> javac -cp $CLASSPATH AddPerson.java ListPeople.java 
> com/example/tutorial/AddressBookProtos.java
> com/example/tutorial/AddressBookProtos.java:95: error: package 
> com.google.protobuf does not exist
>   com.google.protobuf.GeneratedMessageV3 implements
>  ^
> com/example/tutorial/AddressBookProtos.java:19: error: package 
> com.google.protobuf does not exist
>   com.google.protobuf.MessageOrBuilder {
>
> ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] MutableExtension crash in Release only

['Feng Xiao' via Protocol Buffers]

On Tue, Mar 27, 2018 at 5:24 AM Vylsain  wrote:

> Hi,
>
> I'm trying to implement polymorphism with protobuf. I want to have several
> levels, each new level extending the previous one.
> I have a crash when calling MutableExtension.
>
> My .proto file looks like :
>
>
> *package aapb;*
>
> *message A*
> *{*
> * optional int32 fooA = 1;*
>
> * extensions 1000 to max;*
> *}*
>
> *message B*
> *{*
> * extend A*
> * {*
>
> *optional B b = 1000;*
>
> * }*
>
> * optional int32 fooB = 1;*
>
> * extensions 1000 to max;*
> *}*
>
> *message C*
> *{*
> * extend B*
>
> * {*
>
> *optional C c = 1000;*
>
> * }*
>
> * optional int32 fooC = 1;*
>
> * extensions 1000 to max;*
> *}*
>
>
>
> On the C++ side, I have the corresponding classes :
>
> *class A*
> *{*
> *virtual void WriteExtensions(aapb::A *pA){}*
> *}*
>
> *class B : public A*
> *{*
> *virtual void WriteExtensions(aapb::A *pA)*
> *{*
> *aapb::B* pB = pA->MutableExtension(aapb::B::b);*
> *pB->fooB = 1;*
> *WriteExtensions(pB);*
> *}*
>
> *virtual void WriteExtensions(aapb::B *pB){}*
> *}*
>
> *class C : public B*
> *{*
> *virtual void WriteExtensions(aapb::B *pB)*
> *{*
> *aapb::C* pC = pB->MutableExtension(aapb::C::c);
>//crash happens here*
> *pC->fooC = 2;*
> *}*
> *}*
>
> Then, in my program I just create shared_ptr and aapb::A objects and
> call C's WriteExtensions method with aapb::A reference as an argument.
> It works as expected in Debug and I retrieve the correct data when reading
> the file.
> But in Release, I have a segfault when writing the file, when it reaches
> the MutableExtension call of the C class.
> To be more precise, the segfault happens on the call of 
> *google::protobuf::internal::ExtensionSet::MutableMessage(int,
> unsigned char, google::protobuf::MessageLite const&,
> google::protobuf::FieldDescriptor const*) ()*
>
> I'm struggling to see where the problem is coming from and wanted to make
> sure the protobuf part is correct before I look for other causes.
> So you guys see anything wrong in the way I use this ?
>
I don't see a problem there, except that "pB->fooB = 1; " won't compile.
It's likely the problem is not in the code shown above.


>
> Thanks
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] JSON Serialization Performance

['Feng Xiao' via Protocol Buffers]

On Thu, Mar 22, 2018 at 8:23 AM, Edward Clark  wrote:

> Howdy,
>
> I'm working on a project that recently needed to insert data represented
> by protobufs into elasticsearch.  Using the built in JSON serialization we
> were able to quickly get data into elasticsearch, however, the JSON
> serialization seems to be rather slow when compared to generating with a
> library like rapidjson.  Is this expected or is a likely we're doing
> something wrong?
>
It's expected for proto-to-JSON conversion to be slower (and likely much
slower) than a dedicated JSON library converting objects designed to
represent JSON objects to JSON. It's like comparing a library that converts
rapidjson::Document to protobuf binary format against protobuf binary
serialization. The latter is definitely going to be faster no matter how
you optimize the former. Proto objects are just not designed to be
efficiently converted to JSON.

There are ways to improve the proto to JSON conversion though, but at the
end of day it won't going to beat proto to proto binary serialization so
usually performance sensitive services will just support proto binary
format instead.


> Below is info on what we're using, and relative serialization performance
> results.  Surprisingly, rapidjson serialization was faster than protobufs
> binary serialization in some cases, which leads me to believe I'm doing
> something wrong.
>
> Ubuntu 16.04
> GCC 7.3, std=c++17, libstdc++11 string api
> Protobuf 3.5.1.1 compiled with -O3, proto3 syntax
>
> I've measure the performance of 3 cases, serializing the protobuf to
> binary, serializing the protobuf to JSON via MessageToJSONString, and
> building a rapidjson::Document from the protobuf and then serializing that
> to JSON.  All tests use the same message with different portions of the
> message populated, 100,000 iterations.  The json generated from the
> protobuf and rapidjson match exactly.
>
> Test 1, a single string field populated.
> proto binary: 0.01s
> proto json:0.50s
> rapidjson: 0.02s
>
> Test 2, 1 top level string field, 1 nested object with 3 more string
> fields.
> proto binary: 0.02s
> proto json:1.06s
> rapidjson: 0.05s
>
> Test 3, 2 string fields, and 1 ::google::protobuf::ListValue containing
> doubles of the format, [[[double, double], [double, double], ...]], 36
> pairs of doubles total.
> *proto binary: 1.50s*
> *proto json:8.87s*
> *rapidjson: 0.41s*
>
I think this is because of your choice of using
google::protobuf::ListValue. That type (along with
google::protobuf::Value/Struct) is specifically designed to mimic arbitrary
JSON content with proto and is far from efficient compared to protobuf
primitive types. I would just use a "repeated double" to represent these 36
pairs of doubles.


>
> Protobuf binary serialization code:
> std::string toJSON(Message const& msg) { return
> msg.SerializeAsString(); }
>
> Protobuf json serialization code:
> std::string toJSON(Message const& msg) { return
> msg.SerializeAsString(); }
> std::string json;
> ::google::protobuf::util::MessageToJsonString(msg,
> std::addressof(json));
> return json;
> }
>
> Rapidjson serialization code:
> // It's a lengthy section of code manually populating the document.
> Of note, empty strings and numbers set to 0 are omitted from the JSON as
> the protobuf does.  The resulting JSON is exactly the same as the protobuf
> json.
>
> Any info on how to improve the protobuf to JSON serialization would be
> greatly appreciated!
>
> Thanks,
> Ed
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Impossible to launch ubuntu unity after protobuf install

['Feng Xiao' via Protocol Buffers]

Do you happen to install protobuf into /usr instead of the default
/usr/local?

ubuntu 16.04 comes with a default installation of protobuf library and it's
used by unity. Your new installation likely has overridden the default
libprotobuf.so and leads to compatibility issues.

On Tue, Mar 20, 2018 at 3:22 AM, Rémi Pautrat 
wrote:

> Hello everyone,
>
> I have downloaded and compiled the C++ installation of protobuf yesterday
> on my Ubuntu 16.04 and everything seemed to be fine (make check passed
> everything in particular).
>
> Now today when I restarted my computer, I wasn't able to open unity: the
> launcher and top pannel have disappeared and I can't open a terminal with
> the classical shortcut Ctrl + Alt + T for example. When I launch unity in a
> terminal, I get the following error:
>
> /usr/lib/x86_64-linux-gnu/unity/compiz-config-profile-setter:
>  symbol lookup error: /usr/lib/x86_64-linux-gnu/libcompizconfig.so.0:
> undefined symbol: _ZNK6google8protobuf7Message9SpaceUsedEv
>
> It seems that the protobuf installation has messed up with compiz and
> unity.
>
> I have tried several things to fix unity, but nothing worked:
>
>- Reinstall ubuntu-desktop with
>sudo apt-get install --reinstall ubuntu-desktop
>- Reinstall libcompizconfig and also compiz entirely with apt-get.
>- Reinstall unity with apt-get
>- Uninstall protobuf with
>sudo make uninstall
>
>
> Do you have any idea why this happened and how to fix this? I would like
> to completely undo the protobuf install and have unity back as before, but
> obviously sudo make uninstall wasn't enough...
>
Try reinstalling the protobuf package that comes along with ubuntu.
Something like:

sudo apt-get install --reinstall libprotobuf8 libprotobuf-lite8

I'm not sure which version of protobuf is on your ubuntu. libprotobuf8 is
the version the comes with ubuntu 14.04. On ubuntu 16.04, it's probably
libprotobuf9 or similar.


>
> Thanks in advance for your help!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protocol Buffer + AIDL

['Feng Xiao' via Protocol Buffers]

On Fri, Mar 9, 2018 at 1:36 PM, Matt W  wrote:

> Is it possible to have the generated code implement Parcelable?
>

No (assuming you are using the official Java implementation).  I think you
can implement a simple wrapper for this purpose.


> My System Service is receiving Events that were serialized with Google
> Protocol Buffers, and I would like to deserialize inside the service rather
> than the manager. Any non primitive sent across AIDL needs to implement
> Parcelable.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Override equals and hashCode for generated Java code?

['Feng Xiao' via Protocol Buffers]

Protobuf generated classes can't be subclassed. I suggest create a wrapper
type and implement the hashCode()/equals() there.

To generate one .java file for each message, use "option
java_multiple_files = true;" in your .proto file.

On Wed, Feb 28, 2018 at 5:41 AM Kaushal Hooda 
wrote:

> Hi,
>
> The default behaviour for equals/hashCode is checking every single field
> for comparison/generating the hash.
>
> This is problematic for us - we need to override this to use only few
> fields (e.g., an ID field). Is there any way to do this, apart from editing
> the generated java code every time I run protoc?
>
> Also, is it possible to generate one .java file for each message type,
> instead of having them all inside one huge file for the entire schema?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protocol message had invalid UTF-8 (C++ to Java)

['Feng Xiao' via Protocol Buffers]

On Mon, Feb 26, 2018 at 5:54 AM  wrote:

> Hello,
>
> I'm hoping someone can help me out a bit here..
>
> I've developed a C++ GRPC server (Question is protobuf specific though),
> that talks with a Java client. The C++ server takes data from an older C++
> API and converts it to protobuf and then uses GRPC to send the data.
>
> However it appears that there may be a bug in the older API and i'm
> occasionally being sent garbage in some char*, which when they get to java
> result in an Protocol message had invalid UTF-8.
>
> It seems slightly odd that on the C++ side the protobuf is happy to accept
> and encode invalid data, but the Java side throws an exception.
>
> Is there anyway from the C++ side to validate the messages strings before
> they are sent. I've noticed if i convert the message to JSON the strings
> just stop at the first invalid character they reach. I looked at trying to
> find a simple UTF8 validation in C++ but not had much luck with this (Sorry
> C++ is really noy my main language)
>
You can use protobuf's VerifyUtf8String() method:
https://github.com/google/protobuf/blob/master/src/google/protobuf/wire_format_lite.h#L345


>
> Developing in Visual Studio 2017 with C++11 support
>
> Thanks,
> Robin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How do I create a field mask for a field of a repeated field?

['Feng Xiao' via Protocol Buffers]

On Sun, Feb 18, 2018 at 9:05 PM Shelley Tong 
wrote:

> Let's say you have a proto defined as the following:
>
> message Zoo {
> repeated Cat cats = 1;
> }
>
> Let's say cat looks like this:
>
> message Cat {
> google.protobuf.StringValue name = 1
> Decimal age = 2;
> }
>
> If I want to reduce the amount of data in Zoo such that sample data only
> contains Cat with only the name field, how do I do that?
>
> Example: A sample Zoo object will look like this:
>
> {
> Cat: [{
> name = "sam";
> },
> {
> name = "hester";
> }]
> }
>
> I'm trying to do this with one field mask:
>
> FieldMask zoo_mask = FieldMask.newBuilder()
> .addPaths("cats")
> .build();
>
> Zoo getMaskedZoo(FieldMask mask, Zoo zoo) {
> Zoo.Builder zooDest = Zoo.newBuilder();
> FieldMaskUtil.merge(zoo_mask, zoo, zooDest);
> return zooDest.build();
> }
>
> How do I make it so that only the name displays for each Cat?
>
This is not supported at the moment, i.e., you can't use 'cats.name' in the
FieldMask because 'cats' is a repeated field. FieldMaskUtil is more of a
helper utility for a specific use case of protobuf. In your case, I think
you can simply use protobuf reflection API to do the same.

> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "unwrap descriptor class before comparison of RepeatedField types" into release

['Feng Xiao' via Protocol Buffers]

On Fri, Feb 16, 2018 at 2:31 AM <x...@rubynerd.net> wrote:

> Sorry to double post, but this has stagnated - please can we get some
> movement on this?
>
Jisi is on vacation and will be back in two weeks. I added him to your pull
request. There isn't a concrete timeline for the release though. Patch
releases are mostly done on request depending on the urgency of the bug
fixes.


>
> Even a timeline of when someone could look at it would be solid gold at
> this stage - this feels a lot like stonewalling.
>
> Thx,
> @rubynerd
>
> On Tuesday, January 30, 2018 at 11:58:39 PM UTC, x...@rubynerd.net wrote:
>>
>> Absolutely amazing - please can we land this in 3.5.2 then?
>>
>> On Tuesday, January 30, 2018 at 11:39:31 PM UTC, Feng Xiao wrote:
>>>
>>>
>>>
>>> On Tue, Jan 30, 2018 at 3:22 PM, <x...@rubynerd.net> wrote:
>>>
>>>> Hey,
>>>>
>>>> I'm trying to get https://github.com/google/protobuf/pull/2519 into a
>>>> release - as it was merged into master, I think it's been forgotten! :(
>>>>
>>>> I've opened https://github.com/google/protobuf/pull/4254 to try and
>>>> get the patch into the next 3.5.x release, but I'm grasping at straws here.
>>>>
>>>> Please can someone take a look at this situation, and help me get this
>>>> merged? My original patch was opened on the 20th of December 2016, so I'd
>>>> really like to put this to bed.
>>>>
>>> +Jisi is making a 3.5.2 release based off the 3.5.x branch.
>>>
>>>
>>>>
>>>>
>>>> Thanks,
>>>> @rubynerd
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+u...@googlegroups.com.
>>>> To post to this group, send email to prot...@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Generating a api descriptor proto in bazel for a grpc service

['Feng Xiao' via Protocol Buffers]

On Wed, Feb 7, 2018 at 7:14 PM, Marquis Taliaferro <
marquis.taliafe...@gmail.com> wrote:

> Is there a way to generating a api descriptor proto in bazel for a grpc
> service.  I dont mind writing my own rule but i would prefer not to just
> use a genrule for this.  What im trying to do is described here
> 
>  in
> step 1.
>
> So basically given a proto_library rule like this
>
> proto_library(
> name = "user_proto",
> srcs = ["user.proto"],
> deps = [
> ":error_proto",
> "//src/main/proto/overloaded/common:common_proto",
>  "@googleapi//:annotations_proto"]
> )
>
> how would i generate a descriptor proto that is equivalant to being
> generated on the commend line such as
>
> protoc \
> *--include_imports \*
> *--include_source_info \*
> --proto_path=. \
> --descriptor_set_out=api_descriptor.pb \
> user.proto
>
> If there are some docs that im missing that could help that would also be
> very appreciated.
>

I think the bazel folks may answer this question better since they
implemented the proto_library rule in bazel. You can post the question
there as well. As I remember, the proto_library rule already produces a
descriptor output but only for the proto in "srcs". So when you build
"user_proto", a descriptor file for user.proto is produced but not with
"include_imports", nor with "include_source_info". See:
https://github.com/bazelbuild/bazel/issues/4201#issuecomment-348673194
https://github.com/bazelbuild/bazel/issues/4201#issuecomment-351055194


>
> Thanks,
> Marquis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is Any safe to use for java in proto3?

['Feng Xiao' via Protocol Buffers]

On Thu, Feb 8, 2018 at 12:42 PM, Mark Hoffman  wrote:

> Hi,
>
> I have a little concern using the 'Any' type in java due to this message
> in the docs:
>
> *Currently the runtime libraries for working with Any types are under
> development*.
>
> Does anyone know what the status is for production use?
>
It's ready for production use. The above is just saying more new APIs for
Any are coming in some other languages but no existing API will change. In
the case of Java, the support for Any is completed and there is no plan to
add more APIs at the moment.


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Porting Protocol Buffer Library to Green Hills Integrity

['Feng Xiao' via Protocol Buffers]

On Wed, Feb 7, 2018 at 4:32 PM, Alex Chen  wrote:

> I am trying to port the protocol buffer library to Green Hills Integrity
> on a PPC platform. When compiling the library, I ran into the following
> compiler error:
>
> "..\vendor\protobuf\src\google/protobuf/reflection_internal.h", line 306:
> error #1001:
>   class member designated by a using-declaration must be visible
> in a
>   direct base class
> using RepeatedFieldAccessor::Add;
>
> Below is the code snippet where the error occured in reflection_internal.h:
>
> class RepeatedPtrFieldStringAccessor : public RepeatedPtrFieldWrapper
> {
>   typedef void Field;
>   typedef void Value;
>   using RepeatedFieldAccessor::Add;
>
>  public:
>   RepeatedPtrFieldStringAccessor() {}
> };
>
> It seems the statement "using RepeatedFieldAccessor::Add" is trying to
> access the member function Add() that is two levels higher in the class
> hierarchy, which is not allowed by the compiler because it is not a direct
> base class. If my explanation is confusing, I wrote a small snippet of code
> below that simulates the same problem:
>
> class A
> {
> public:
> virtual void foo() {}
> };
>
> template
> class B : public A
> {
> public:
> virtual void foo() {}
> };
>
> class C : public B
> {
> using A::foo;
> };
>
> I am to successfully compile the protobuf library using Microsoft Visual
> Studio and Linux GNUC, but not MULTI Integrity compiler. I consulted with
> Green Hills and they say that syntax is illegal. They even pointed me to
> this link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32039
>
Reading the thread reminds me how difficult C++ is... Is there any way I
can run this MULTI Integrity compiler on my macosx, linux or windows? Or
can you ask Green Hills what other compiler can reproduce the same problem?


>
> Does anyone know how to workaround this issue? I rather not have to modify
> the protobuf library code to fix it.
>
> Thanks,
>
> -Alex
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Porting Protocol Buffer Library to Green Hills Integrity

['Feng Xiao' via Protocol Buffers]

I think moving "using RepeatedFieldAccessor::Add" to the parent class
RepeatedPtrFieldWrapper might fix the issue. Could you give it a try? If it
works, could you also help make a pull request? I don't have a compiler
that can reproduce the error you saw.

On Wed, Feb 7, 2018 at 4:53 PM, Feng Xiao <xiaof...@google.com> wrote:

> Can you create an issue on our github repo?
> https://github.com/google/protobuf
>
> On Wed, Feb 7, 2018 at 4:32 PM, Alex Chen <a4c...@gmail.com> wrote:
>
>> I am trying to port the protocol buffer library to Green Hills Integrity
>> on a PPC platform. When compiling the library, I ran into the following
>> compiler error:
>>
>> "..\vendor\protobuf\src\google/protobuf/reflection_internal.h", line
>> 306: error #1001:
>>   class member designated by a using-declaration must be visible
>> in a
>>   direct base class
>> using RepeatedFieldAccessor::Add;
>>
>> Below is the code snippet where the error occured in
>> reflection_internal.h:
>>
>> class RepeatedPtrFieldStringAccessor : public
>> RepeatedPtrFieldWrapper {
>>   typedef void Field;
>>   typedef void Value;
>>   using RepeatedFieldAccessor::Add;
>>
>>  public:
>>   RepeatedPtrFieldStringAccessor() {}
>> };
>>
>> It seems the statement "using RepeatedFieldAccessor::Add" is trying to
>> access the member function Add() that is two levels higher in the class
>> hierarchy, which is not allowed by the compiler because it is not a direct
>> base class. If my explanation is confusing, I wrote a small snippet of code
>> below that simulates the same problem:
>>
>> class A
>> {
>> public:
>> virtual void foo() {}
>> };
>>
>> template
>> class B : public A
>> {
>> public:
>> virtual void foo() {}
>> };
>>
>> class C : public B
>> {
>> using A::foo;
>> };
>>
>> I am to successfully compile the protobuf library using Microsoft Visual
>> Studio and Linux GNUC, but not MULTI Integrity compiler. I consulted with
>> Green Hills and they say that syntax is illegal. They even pointed me to
>> this link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32039
>>
>> Does anyone know how to workaround this issue? I rather not have to
>> modify the protobuf library code to fix it.
>>
>> Thanks,
>>
>> -Alex
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Porting Protocol Buffer Library to Green Hills Integrity

['Feng Xiao' via Protocol Buffers]

Can you create an issue on our github repo?
https://github.com/google/protobuf

On Wed, Feb 7, 2018 at 4:32 PM, Alex Chen  wrote:

> I am trying to port the protocol buffer library to Green Hills Integrity
> on a PPC platform. When compiling the library, I ran into the following
> compiler error:
>
> "..\vendor\protobuf\src\google/protobuf/reflection_internal.h", line 306:
> error #1001:
>   class member designated by a using-declaration must be visible
> in a
>   direct base class
> using RepeatedFieldAccessor::Add;
>
> Below is the code snippet where the error occured in reflection_internal.h:
>
> class RepeatedPtrFieldStringAccessor : public RepeatedPtrFieldWrapper
> {
>   typedef void Field;
>   typedef void Value;
>   using RepeatedFieldAccessor::Add;
>
>  public:
>   RepeatedPtrFieldStringAccessor() {}
> };
>
> It seems the statement "using RepeatedFieldAccessor::Add" is trying to
> access the member function Add() that is two levels higher in the class
> hierarchy, which is not allowed by the compiler because it is not a direct
> base class. If my explanation is confusing, I wrote a small snippet of code
> below that simulates the same problem:
>
> class A
> {
> public:
> virtual void foo() {}
> };
>
> template
> class B : public A
> {
> public:
> virtual void foo() {}
> };
>
> class C : public B
> {
> using A::foo;
> };
>
> I am to successfully compile the protobuf library using Microsoft Visual
> Studio and Linux GNUC, but not MULTI Integrity compiler. I consulted with
> Green Hills and they say that syntax is illegal. They even pointed me to
> this link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32039
>
> Does anyone know how to workaround this issue? I rather not have to modify
> the protobuf library code to fix it.
>
> Thanks,
>
> -Alex
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Generating Code With Parent Directory Imports?

['Feng Xiao' via Protocol Buffers]

On Wed, Jan 31, 2018 at 2:57 PM, Jonnny <irs...@gmail.com> wrote:

> By 'Protobuf Python Library' you mean the Python Package called `protobuf
> ? I think It's already included in my package as dependency. But Google
> protobuf is just one example. Most people around here using Scala, and they
> all have 'scalapb' imported into their proto files. The path is `*scalapb*
> /*scalapb*.*proto`. *It's not something I can chance as it's a third
> party proto, and it's not a Google protobuf so using the `protobuf` package
> won't solve this issue. I can manually add it into my `/usr/local/include`
> folder, but this mean the package is not self-contained and the user needs
> to compile his own scalapb pb2 file, and manually add it to
> `/usr/local/include` if he wants my package to work.
>
Is there not a Pypi package containing scalapb/scalapb_pb2.py already? How
about ask the author of scalapb/scalapb.proto to provide such a python
package?


>
> It means again, I have to keep the `scalapb` folder set into the root of
> my project, or play around with PYTHONPATH. I wasn't able to find online
> Python projects/packages that uses Python complied proto code, so I was
> thinking perhaps there is a way around it. But if there is not a better
> way, and that's consider the 'standard', I'm guessing I can do one of the
> three below:
>
> 1. On a normal project, `scalapb` should be sitting on the root of the
> project (ie, the folder where I'm running the script from).
> 2. Packages are more complex, because the 'root' includes things like
> `setup.py` and `README.srt`, but not the actual package code. The package
> sits in a directory inside the root Folder. For example in the Request
> package (https://github.com/requests/requests),  the actual code run from
> within the `request` folder. So even if I'll keep the generated code
> directory structure in the `requests` folder, it won't work necause the
> relative path starts from the Root folder and not `requests` folder. Adding
> the folder structure into the root of the project is messy (because you
> need to defines those as package in `setup.py`, and perhaps even supply
> __init__.py file).
> 3. Playing with PYTHONPATH like so:
>
>
> import sys
> sys.path.append('./my_package_name/proto')
>
>
> This code need to be called every-time the package is called (probably need 
> to sit inside the `__init__.py I'm guessing).
>
>
> There is a 4th option, but one that I do not like much. In case I have
> small amount of `pb2` files, I can manually edit the `pb2` imports by
> adding my folder after the code is compiled.
>
> I would just go ahead with (1). What's so bad about having a scalapb
directory in your root directory? If you are concerned with conflicts with
other project that may also include scalapb, you can just create a separate
python package for the scalapb/scalapb_pb2.py, deploy it to pypi, and share
it with all other projects that depend on the same thing.


>
>
>
>
>
> On Wednesday, January 31, 2018 at 11:39:08 PM UTC+2, Feng Xiao wrote:
>>
>>
>>
>> On Wed, Jan 31, 2018 at 8:54 AM, Jonnny <irs...@gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> I have a couple of micro-services, and each has couple of proto files.
>>> It's common for one micro-service proto file to reference another service
>>> proto file using import like so:
>>>
>>> import "google/protobuf/struct.proto";
>>> import "myService/v1/message.proto";
>>> import "myOtherService/v2/parser.proto";
>>>
>>> We use a Gradle script that's responsible on creating the folder
>>> structure and fetch all proto dependencies from different git repositories.
>>> So in the example above the Gradle script will output will be a folder
>>> named `google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each
>>> of the folders will hold the proto files needed. Once the folder structure
>>> is OK, I can use `protoc` to generate the Python code.
>>>
>>> Here's where the problem begins. The generated `_pb2` code import
>>> modules based on the folder structure. So the import inside the `_pb2` file
>>> will look like so:
>>>
>>> from google.protobuf import struct_pb2 as 
>>> google_dot_protobuf_dot_struct__pb2
>>>
>>>
>>> That's a relative path import, and it means that in order for my Python
>>> program to work, the root folder where the Python script running from,
>>> should have the `google/protobuff` folder for this import to work.Now
>>> because I have multiple micro-services, I don't appreciated the fact I have
>>> to 

Re: [protobuf] Generating Code With Parent Directory Imports?

['Feng Xiao' via Protocol Buffers]

On Wed, Jan 31, 2018 at 8:54 AM, Jonnny  wrote:

> Hello everyone,
>
> I have a couple of micro-services, and each has couple of proto files.
> It's common for one micro-service proto file to reference another service
> proto file using import like so:
>
> import "google/protobuf/struct.proto";
> import "myService/v1/message.proto";
> import "myOtherService/v2/parser.proto";
>
> We use a Gradle script that's responsible on creating the folder structure
> and fetch all proto dependencies from different git repositories. So in the
> example above the Gradle script will output will be a folder named
> `google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each of the
> folders will hold the proto files needed. Once the folder structure is OK,
> I can use `protoc` to generate the Python code.
>
> Here's where the problem begins. The generated `_pb2` code import modules
> based on the folder structure. So the import inside the `_pb2` file will
> look like so:
>
> from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
>
>
> That's a relative path import, and it means that in order for my Python
> program to work, the root folder where the Python script running from,
> should have the `google/protobuff` folder for this import to work.Now
> because I have multiple micro-services, I don't appreciated the fact I have
> to add 3 new folders in my main Python project (it's even more painful when
> you create a Pip package). What I was hoping to do is to open a directory
> called `proto`, and add all that directory structure into it - but because
> imports are relative, this won't work. I tried to to play with couple of
> `protoc` options like `PROTO_PATH`, or run `protoc` from a parent
> directory, so the `_pb2` files will get created with a different relative
> path I can work with like so:
>
> from proto.google.protobuf import struct_pb2 as 
> google_dot_protobuf_dot_struct__pb2
>
>
> But I wasn't able to do so. A dirty option would be adding the `proto`
> folder into my `PYTHONPATH`. It's not pretty, and I'm guessing there is
> probably a simple solution for that I'm missing with `protoc`.
>
You can probably change the path for your own message.proto/parser.proto,
but for protobuf's google/protobuf/struct.proto (and any other protos under
google/protobuf), it's the only option to import it as
"google/protobuf/struct.proto". Nothing else would work. It just has to be
imported as "google/protobuf/struct.proto" in any .proto file and has to be
imported as "google.protobuf.struct_pb2" in any python file. It's part of
protobuf public API and won't (and can't) be changed.

Where is protobuf python library installed on your machine? If you have it
installed, google.protobuf.struct_pb2 should be part of it and you
shouldn't need to put google/protobuf/struct_pb2.py in your own project
directory.


>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "unwrap descriptor class before comparison of RepeatedField types" into release

['Feng Xiao' via Protocol Buffers]

On Tue, Jan 30, 2018 at 3:22 PM,  wrote:

> Hey,
>
> I'm trying to get https://github.com/google/protobuf/pull/2519 into a
> release - as it was merged into master, I think it's been forgotten! :(
>
> I've opened https://github.com/google/protobuf/pull/4254 to try and get
> the patch into the next 3.5.x release, but I'm grasping at straws here.
>
> Please can someone take a look at this situation, and help me get this
> merged? My original patch was opened on the 20th of December 2016, so I'd
> really like to put this to bed.
>
+Jisi is making a 3.5.2 release based off the 3.5.x branch.


>
>
> Thanks,
> @rubynerd
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Tags and values

['Feng Xiao' via Protocol Buffers]

On Mon, Jan 22, 2018 at 3:08 PM, Ashwin Kini  wrote:

> Hi all,
>
> While defining messages can the *tags *be non continous? The
> documentation never mentions the same
>
> message abc {
> optional uint32 xyz = *100*;
> optional uint32 def = *200*;
> optional unit32 abc = *0xDEAD01*;
>
> }
>
> The documentation says it needs to be unique. Agreed. Can they be non
> continous as explained in example?
>
It can, but I would not recommend it. You should generally make the tag
numbers as small as possible and as dense as possible. Smaller tags need
less bytes to encode and dense tags allow compiler to optimize the parsing
switch case better.


> . Also does it take HEX values as far as it falls in the range mentioned
> in the doc?
>
It should be supported.


>
> One more question is, can repeated fields be made required?
>
The general recommendation is not to use required fields, however if you
have a way to guarantee that all repeated fields have at least one element,
changing it to required should be wire compatible.


> I did a lot of digging, seems its not possible. Can anybody confirm? I
> understand required should be completely avoided.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Python CPP Implementation

['Feng Xiao' via Protocol Buffers]

On Mon, Jan 15, 2018 at 4:34 AM, Matthew Heritage <
matthew.c.herit...@gmail.com> wrote:

> According to the docs, the C++ Python implementation is experimental, but
> this has been the case in the docs for 7 years.
> Is it really still experimental or are the docs out of date?
>
A link to the said doc? The doc needs update. Python C++ support is not
experimental any more and has been used in production both inside/outside
Google. +jieluo


> Is there a date when the default Python implementation will be switched to
> C++ from Python? The performance difference is huge.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Performance for Messages containing big (1...100MByte) binary blobs

['Feng Xiao' via Protocol Buffers]

On Thu, Jan 11, 2018 at 4:58 AM,  wrote:

> Thanks for the reply :)
>
> I was planning to use the "bytes" Proto-Type from the proto3 language
> specification (see here: https://developers.google.com/
> protocol-buffers/docs/proto3).
> It seems to be suited to transfer binary blobs.
>
> However in the Techniques section of the ProtocolBuffers documentation (
> https://developers.google.com/protocol-buffers/docs/techniques) it is
> mentioned, that "Protocol Buffers are not designed to handle large
> messages." without any further explanation. So I wonder if this also
> applies for "bytes"-typed message fields.
>
Using a bytes field of 100MB is fine. That satement "Protocol Buffers are
not designed to handle large messages." is more or less obsolete now
because we see many people sending protos as large as 2GB and they are
supported.



>
>
> On Thursday, January 11, 2018 at 1:20:26 PM UTC+1, Mario Emmenlauer wrote:
>>
>>
>> Hi,
>>
>> On 11.01.2018 13:13, rscho...@gmail.com wrote:
>> > Hi,
>> > How do Protocol Buffers (C++) perform when messages contain big binary
>> blobs? With big I mean 1 ... 100MByte.
>>
>> My impression was that its not very suitable for that. Its a while
>> ago that I checked but there was no plain binary encoding back then.
>> To send binary files, I encoded them in some ASCII encoding (Base64
>> I think) and sent them as strings. The encoding takes some CPU, and
>> the data size goes up.
>>
>> I'd be happy to learn otherwise?! And possibly things have changed in
>> the meantime...
>>
>> Cheers,
>>
>> Mario Emmenlauer
>>
>>
>> --
>> BioDataAnalysis GmbH, Mario Emmenlauer  Tel. Buero: +49-89-74677203
>> <+49%2089%2074677203>
>> Balanstr. 43
>> 
>> mailto: memmenlauer * biodataanalysis.de
>> D-81669 München  http://www.biodataanalysis.de/
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

['Feng Xiao' via Protocol Buffers]

On Wed, Dec 20, 2017 at 4:18 PM, Behrang Saeedzadeh <behran...@gmail.com>
wrote:

> Unfortunately I don't own the actual protocol buffer models and can't
> change their definitions.
>
> What I actually want to do is this: I have a Protocol buffer model in
> byte[] format. I want to deserialize it back to Java objects but then
> replace all the ID fields in the model, including its extensions.
>
> For example, I have a "template" Employee model in byte[] format, and I
> want to create new Java objects using that template. After deserializing
> the byte array to a Java object, I want to change the UserHeader.userID and
> ID of all the skills in it.
>
> Is there a fluent API or a concise way available to do this?
>
Why not just use protobuf binary format? For example:

byte[] data = userBuilder.build().toByteArray();

UserModel.User.Builder userBuilder2 =
UserModel.User.newBuilder().mergeFrom(data);


>
>
>
> On Thursday, December 21, 2017 at 10:56:10 AM UTC+11, Feng Xiao wrote:
>>
>> JsonFormat doesn't support extensions. You can replace extensions with
>> google.protobuf.Any if you want to use the proto with JsonFormat.
>>
>> On Wed, Dec 20, 2017 at 3:39 PM, Behrang Saeedzadeh <behr...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> *down vote**favorite*
>>> <https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel#>
>>>
>>> *Cross-post from
>>> StackOverflow: 
>>> https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
>>> <https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel>*
>>>
>>> I have 2 Protobuf models:
>>> User:
>>>
>>> package demo;
>>>
>>> option java_package = "com.stackoverflow.question";
>>> option java_outer_classname = "UserModel";
>>>
>>> message User {
>>>
>>> message UserHeader {
>>> required int64 userId = 1;
>>> }
>>>
>>> required UserHeader header = 1;
>>>
>>> extensions 100 to 200;}
>>>
>>> Employee:
>>>
>>> import "person.proto";
>>> package demo;
>>>
>>> option java_package = "com.stackoverflow.question";
>>> option java_outer_classname = "EmployeeModel";
>>>
>>> extend demo.User {
>>> optional EmployeeDetails details = 101;}
>>>
>>> message EmployeeDetails {
>>> required string department = 1;
>>> repeated Skill skills = 2;}
>>>
>>> message Skill {
>>> required int64 id = 1;
>>> required string name = 2;}
>>>
>>> I can create a model and serialize it to JSON using
>>> JsonFormat.printer().print(...):
>>>
>>> ExtensionRegistry registry = 
>>> ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
>>> UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
>>> userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
>>> EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
>>> EmployeeModel.EmployeeDetails.newBuilder();
>>> employeeBuilder.setDepartment("Department 1")
>>>.addSkills(EmployeeModel.Skill.newBuilder()
>>>  .setId(10_000)
>>>  .setName("Skill 10_")
>>>  .build())
>>>.addSkills(EmployeeModel.Skill.newBuilder()
>>>  .setId(11_000)
>>>  .setName("Skill 11_")
>>>  .build());
>>>
>>> userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
>>> final String json = JsonFormat.printer().print(userBuilder.build());
>>>
>>> However deserializing the generated JSON back to Java objects fails with
>>> com.google.protobuf.InvalidProtocolBufferException: Cannot find field:
>>> details in message demo.User:
>>>
>>> UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
>>> JsonFormat.parser().merge(json, userBuilder2);
>>>
>>> And there doesn't seem to be a way to pass an Extensi

Re: [protobuf] How to fix “com.google.protobuf.InvalidProtocolBufferException: Cannot find field” thrown from JsonFormat.parser().merge(…)?

['Feng Xiao' via Protocol Buffers]

JsonFormat doesn't support extensions. You can replace extensions with
google.protobuf.Any if you want to use the proto with JsonFormat.

On Wed, Dec 20, 2017 at 3:39 PM, Behrang Saeedzadeh 
wrote:

>
>
> *down vote**favorite*
> 
>
> *Cross-post from
> StackOverflow: 
> https://stackoverflow.com/questions/47903567/how-to-fix-com-google-protobuf-invalidprotocolbufferexception-cannot-find-fiel
> *
>
> I have 2 Protobuf models:
> User:
>
> package demo;
>
> option java_package = "com.stackoverflow.question";
> option java_outer_classname = "UserModel";
>
> message User {
>
> message UserHeader {
> required int64 userId = 1;
> }
>
> required UserHeader header = 1;
>
> extensions 100 to 200;}
>
> Employee:
>
> import "person.proto";
> package demo;
>
> option java_package = "com.stackoverflow.question";
> option java_outer_classname = "EmployeeModel";
>
> extend demo.User {
> optional EmployeeDetails details = 101;}
>
> message EmployeeDetails {
> required string department = 1;
> repeated Skill skills = 2;}
>
> message Skill {
> required int64 id = 1;
> required string name = 2;}
>
> I can create a model and serialize it to JSON using JsonFormat.printer().
> print(...):
>
> ExtensionRegistry registry = 
> ExtensionRegistry.newInstance();EmployeeModel.registerAllExtensions(registry);
> UserModel.User.Builder userBuilder = UserModel.User.newBuilder();
> userBuilder.setHeader(UserModel.User.UserHeader.newBuilder().setUserId(1000));
> EmployeeModel.EmployeeDetails.Builder employeeBuilder = 
> EmployeeModel.EmployeeDetails.newBuilder();
> employeeBuilder.setDepartment("Department 1")
>.addSkills(EmployeeModel.Skill.newBuilder()
>  .setId(10_000)
>  .setName("Skill 10_")
>  .build())
>.addSkills(EmployeeModel.Skill.newBuilder()
>  .setId(11_000)
>  .setName("Skill 11_")
>  .build());
>
> userBuilder.setExtension(EmployeeModel.details, employeeBuilder.build());
> final String json = JsonFormat.printer().print(userBuilder.build());
>
> However deserializing the generated JSON back to Java objects fails with
> com.google.protobuf.InvalidProtocolBufferException: Cannot find field:
> details in message demo.User:
>
> UserModel.User.Builder userBuilder2 = UserModel.User.newBuilder();
> JsonFormat.parser().merge(json, userBuilder2);
>
> And there doesn't seem to be a way to pass an ExtensionRegistry to JsonFo
> rmat.parser()either.
>
> Is there a way to make this *Protobuf → JSON → Protobuf* 
> serialization/deserialization
> chain work?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Feng Xiao' via Protocol Buffers]

 want to develop against protobuf.
> Debian packages that depend on protobuf could build protobuf (and ABSL) and
> statically link against it to build runnable binaries. It would not be
> possible to have library packages in Debian that depend on protobuf.
>

> Thoughts?
> Josh
>
> On Wed, Dec 20, 2017 at 10:58 AM, Feng Xiao <xiaof...@google.com> wrote:
>
>> +abseil...@googlegroups.com, +proto...@gogolegroups.com
>>
>> CCed public google groups so László can reply to the group. BCCed
>> internal mailing lists.
>>
>> On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman <haber...@google.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I wanted to kick off a conversation about the future of Protobuf and
>>> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
>>> Protobuf package, and has also expressed interest in packaging Abseil.
>>>
>>> Debian currently distributes a binary shared library for protobuf
>>> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
>>> time there is an ABI-incompatible change to the library:
>>>
>>> https://www.debian.org/doc/debian-policy/#shared-libraries
>>>
>>> Since protobuf does not guarantee any kind of binary compatibility, we
>>> have traditionally updated our SONAME for every minor release of protobuf.
>>> We are currently at major version 15:
>>>
>>> https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab
>>> 95c773192b5d908c5a61/src/Makefile.am#L186
>>>
>>> This has worked reasonably well so far. But we want to start depending
>>> on Abseil (https://abseil.io) which does not do releases at all (so
>>> doesn't even have a major version number to track). How would this work for
>>> Debian?
>>>
>>> Abseil's philosophy is that users will build from source. But Debian is
>>> inherently a binary distribution. We have to figure out how to reconcile
>>> these two philosophies. I don't know the answer, but I'm going to throw out
>>> a few straw man ideas as a starting point for discussion.
>>>
>>> 1. Distribute libprotobuf and libabsl as static libraries only.
>>>
>>> Debian policy says that libraries can be distributed in static form only
>>> if that is what is intended by upstream:
>>>
>>> https://www.debian.org/doc/debian-policy/#static-libraries
>>>
>>> Since both libprotobuf and libabsl are disavowing any binary
>>> compatibility guarantees, that implies that they should not be distributed
>>> as .so's. Any Debian packages or user programs that want to use protobuf
>>> have to link it statically.
>>>
>>> The tricky part is that libprotobuf.a and libabsl.a would need to match
>>> each other exactly. They would need to be compiled from exactly the same
>>> ABSL headers. This implies that libprotobuf-dev would need to pin itself to
>>> one specific libabsl-dev version -- the one it was compiled against. Other
>>> libraries that depend on ABSL (like TensorFlow) would also need to be
>>> pinned to exactly the same libabsl-dev version, otherwise it would be
>>> impossible to install them all at the same time. So realistically all
>>> packages that depend on ABSL would need to be updated at the same time.
>>>
>>> 2. Distribute libprotobuf and libabsl as shared libraries.
>>>
>>> This is against the philosophy of ABSL. But if Debian updated its ABSL
>>> infrequently enough, it could just treat each ABSL update as a separate
>>> major version. Sort of like we do with protobuf now.
>>>
>>> The tricky part is if two different libraries pass ABSL types across
>>> library boundaries. For example, imagine that a user program was developing
>>> against libabsl.so.2 but also using libprotobuf.so.15 that was linked
>>> against libabsl.so.1. If a user got an absl::string_view from protobuf,
>>> this could be unsafe for the user to pass directly to ABSL functions. The
>>> representation of absl::string_view could have changed between the two
>>> versions of libabsl.
>>>
>>> The all-static option (1) seems to solve this problem by pinning all of
>>> the ABSL dependencies so that they all rev together. But this solution
>>> could become unwieldy if too many packages depend on ABSL. Also any library
>>> that exposes ABSL types in its API would have to be part of this lock-step
>>> ABSL upgrade.
>>>
>>> 3. Remove protobuf from Debian completely, ask users to build from
>>> source.
>>>
>>> Since the philosophy of ABSL is that users will always build from
>>> source, we could ask that users do this if they want to develop against
>>> protobuf. Debian packages that depend on protobuf could build protobuf (and
>>> ABSL) and statically link against it to build runnable binaries. It would
>>> not be possible to have library packages in Debian that depend on protobuf.
>>>
>>> Thoughts?
>>> Josh
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

['Feng Xiao' via Protocol Buffers]

+abseil...@googlegroups.com, +proto...@gogolegroups.com

CCed public google groups so László can reply to the group. BCCed internal
mailing lists.

On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman  wrote:

> Hi all,
>
> I wanted to kick off a conversation about the future of Protobuf and
> Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the
> Protobuf package, and has also expressed interest in packaging Abseil.
>
> Debian currently distributes a binary shared library for protobuf
> (libprotobuf.so). Debian policy requires that the SONAME is bumped every
> time there is an ABI-incompatible change to the library:
>
> https://www.debian.org/doc/debian-policy/#shared-libraries
>
> Since protobuf does not guarantee any kind of binary compatibility, we
> have traditionally updated our SONAME for every minor release of protobuf.
> We are currently at major version 15:
>
> https://github.com/google/protobuf/blob/0ba8eea655a5e40d19ab95c773192b
> 5d908c5a61/src/Makefile.am#L186
>
> This has worked reasonably well so far. But we want to start depending on
> Abseil (https://abseil.io) which does not do releases at all (so doesn't
> even have a major version number to track). How would this work for Debian?
>
> Abseil's philosophy is that users will build from source. But Debian is
> inherently a binary distribution. We have to figure out how to reconcile
> these two philosophies. I don't know the answer, but I'm going to throw out
> a few straw man ideas as a starting point for discussion.
>
> 1. Distribute libprotobuf and libabsl as static libraries only.
>
> Debian policy says that libraries can be distributed in static form only
> if that is what is intended by upstream:
>
> https://www.debian.org/doc/debian-policy/#static-libraries
>
> Since both libprotobuf and libabsl are disavowing any binary compatibility
> guarantees, that implies that they should not be distributed as .so's. Any
> Debian packages or user programs that want to use protobuf have to link it
> statically.
>
> The tricky part is that libprotobuf.a and libabsl.a would need to match
> each other exactly. They would need to be compiled from exactly the same
> ABSL headers. This implies that libprotobuf-dev would need to pin itself to
> one specific libabsl-dev version -- the one it was compiled against. Other
> libraries that depend on ABSL (like TensorFlow) would also need to be
> pinned to exactly the same libabsl-dev version, otherwise it would be
> impossible to install them all at the same time. So realistically all
> packages that depend on ABSL would need to be updated at the same time.
>
> 2. Distribute libprotobuf and libabsl as shared libraries.
>
> This is against the philosophy of ABSL. But if Debian updated its ABSL
> infrequently enough, it could just treat each ABSL update as a separate
> major version. Sort of like we do with protobuf now.
>
> The tricky part is if two different libraries pass ABSL types across
> library boundaries. For example, imagine that a user program was developing
> against libabsl.so.2 but also using libprotobuf.so.15 that was linked
> against libabsl.so.1. If a user got an absl::string_view from protobuf,
> this could be unsafe for the user to pass directly to ABSL functions. The
> representation of absl::string_view could have changed between the two
> versions of libabsl.
>
> The all-static option (1) seems to solve this problem by pinning all of
> the ABSL dependencies so that they all rev together. But this solution
> could become unwieldy if too many packages depend on ABSL. Also any library
> that exposes ABSL types in its API would have to be part of this lock-step
> ABSL upgrade.
>
> 3. Remove protobuf from Debian completely, ask users to build from source.
>
> Since the philosophy of ABSL is that users will always build from source,
> we could ask that users do this if they want to develop against protobuf.
> Debian packages that depend on protobuf could build protobuf (and ABSL) and
> statically link against it to build runnable binaries. It would not be
> possible to have library packages in Debian that depend on protobuf.
>
> Thoughts?
> Josh
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf and Golang: How to generate a map string interface using proto3?

['Feng Xiao' via Protocol Buffers]

On Tue, Dec 19, 2017 at 10:01 AM,  wrote:

> Hi, I am new in using Protobuf for golang.
>
> https://github.com/golang/protobuf
>
>
> Using golang Protobuf, within *package **foo*, I created *message
> TaskJobEntity* which is a combination of other messages:
>
>
> message TaskJobEntity {
> JobEntity job = 1;
> TaskEntity task = 2;
> map dependent_jobs = 3;
>
> }
>
>
> Which is used within *package foo* response as part a *map*:
>
>
> message ListTaskJobsResponse {
> *map**** task_jobs* = 1;
> ServiceStatus status = 2;
> }
>
> Now within another Protobuf *package bar*, I would like to define
> response that includes *message TaskJobEntity* defined in *package **foo*,
> however, as far as I can determine, a *proto3* package cannot reference
> another proto3 package (is this determination correct?).
>
What makes you think that? You can certainly import and use other proto3
packages.


>
> My goal is to add *map task_jobs* to
> *package bar* defined response.
>
>
> Here is how I approaching to get to this goal. I have found that map of
> *interface{}* is not supported in Protobuf. So, is using
> *google.protobuf.Any* the next best solution?
>
>
> message ImportStatsResponse {
> string task_id = 1;
> // map task_jobs = 2;
> // map task_jobs = 2;
> map task_jobs = 2;
> ServiceStatus status = 3;
> }
>
>
> Recommendations?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] ParseFromArray always returning true

['Feng Xiao' via Protocol Buffers]

On Wed, Dec 6, 2017 at 4:39 PM, 'Allen Vicencio' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> If all my fields in a protobuf are `optional`, will ParseFromArray ever
> return false? I am doing some tests right now, and I am testing two
> scenarios:
>
>
>> Person
>
> optional string name;
>
>
>  Vehicle
>
>  optional int64 wheels;
>
>
> 1. Empty buffer is passed into Person#ParseFromArray. Returns true.
> 2. Instance of Person is serialized into a buffer. Buffer is passed into
> Vehicle#ParseFromArray. Returns true.
>
> Is this expected behavior?
>
Unless you have required fields, protobuf parsing will always return true
as long as the input data is valid proto wire format.


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Status of protobuf-java 2.x / 3.x compatibility

['Feng Xiao' via Protocol Buffers]

On Tue, Dec 5, 2017 at 1:33 PM, Chris Thunes <cthu...@brewtab.com> wrote:

> Thanks Feng. It seems like the GeneratedMessage / GeneratedMessageV3 split
> introduced between 3.0.0-beta-4 and the 3.0.0 final release caused the
> java/compatability_tests suite to start failing when run against 2.5.0 /
> 2.6.1 ("java/compatibility_tests/v2.5.0/test.sh 2.5.0" passes at git tag
> v3.0.0-beta-4 but fails at v3.0.0).
>
This is expected. Only 3.0.0-beta-4 is intended to be compatible with 2.5.0/
2.6.1. The final 3.0.0 release is not compatible with 2.5.0/2.6.1. 3.0.0
release is compatible with 3.0.0-beta-4 though. So a migration path could
be:
1. Move individual components to depend on 3.0.0-beta-4 instead of 2.5.0/
2.6.1. Since 3.0.0-beta-4 is compatible with 2.5.0/2.6.1 you don't need to
upgrade everything at once but can do it incrementally.
2. Once every component is on 3.0.0-beta-4, upgrade the dependency to 3.x.x.

This is how we do the migration inside google, but I guess it's hard to do
the same in the opensource world because you don't have control over your
dependencies.


> In 3.0.0 and newer, what is the role of the non-V3 variants of
> GeneratedMessage, SingleFieldBuilder, etc? Are these classes simply
> vestigial at this point or do they still provide some benefit to
> compatibility (even if not 100%).
>
They are still needed to be compatible with older beta releases like
3.0.0-beta-3 etc.


>
> Thanks again,
> Chris
>
> On Mon, Dec 4, 2017 at 8:24 PM, Feng Xiao <xiaof...@google.com> wrote:
>
>>
>>
>> On Mon, Dec 4, 2017 at 9:00 AM, Chris Thunes <cthu...@brewtab.com> wrote:
>>
>>>
>>> I'm looking at options for moving some applications that currently
>>> depend on protobuf-java 2.5.0 to a more recent version. This is made
>>> complicated by the fact that we have a mixure of internal and external
>>> dependencies (Hadoop & HBase) which depend on protobuf-java. My
>>> understanding is that this will require these dependencies to move to a 3.x
>>> release sychronously (i.e. regenerate using a 3.x protoc and update
>>> protobuf-java to a corresponding release).
>>>
>>> However, looking through release notes and protobuf source code it seems
>>> like some attempts have been made to address the source and binary
>>> compatibility issues between 2.5/2.6.1 and 3.x. Specifically,
>>>
>>>- The 3.0.0-beta-4 release notes
>>><https://github.com/google/protobuf/blob/v3.0.0-beta-4/CHANGES.txt>
>>>mention runtime updates "to be compatible with 2.5.0/2.6.1 generated
>>>protos".
>>>- A number of classes have "V3" variants where the non-V3 variants
>>>appear to exist solely in an attempt to maintain binary compatibility 
>>> with
>>>pre-3.x generated code.
>>>
>>> Running the compatibility tests in java/compatibility_tests/v2.5.0 it
>>> appears that source and binary incompatibilities still exist.
>>>
>>>
>>> I'm curious if anyone can shed some light on this effort and its status
>>> or provide suggestions for migrating to a recent protobuf release under
>>> these circumstances.
>>>
>> Protobuf 2.5.0/2.6.1 should be compatible with 3.0.0-beta-4 if you only
>> uses protobuf public APIs. That's unfortunately not the case with Hadoop &
>> HBase though. They introduced a class into com.google.protobuf package
>> with the sole purpose to access protobuf package private classes
>> <https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java>.
>>  As
>> such there is no way you can upgrade to protobuf 3.x if the version of
>> Hadoop & HBase you use still depends on protobuf 2.5/2.6 private symbols.
>>
>>
>>> Thanks,
>>>
>>> Chris
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Status of protobuf-java 2.x / 3.x compatibility

['Feng Xiao' via Protocol Buffers]

On Mon, Dec 4, 2017 at 9:00 AM, Chris Thunes  wrote:

>
> I'm looking at options for moving some applications that currently depend
> on protobuf-java 2.5.0 to a more recent version. This is made complicated
> by the fact that we have a mixure of internal and external dependencies
> (Hadoop & HBase) which depend on protobuf-java. My understanding is that
> this will require these dependencies to move to a 3.x release sychronously
> (i.e. regenerate using a 3.x protoc and update protobuf-java to a
> corresponding release).
>
> However, looking through release notes and protobuf source code it seems
> like some attempts have been made to address the source and binary
> compatibility issues between 2.5/2.6.1 and 3.x. Specifically,
>
>- The 3.0.0-beta-4 release notes
>
>mention runtime updates "to be compatible with 2.5.0/2.6.1 generated
>protos".
>- A number of classes have "V3" variants where the non-V3 variants
>appear to exist solely in an attempt to maintain binary compatibility with
>pre-3.x generated code.
>
> Running the compatibility tests in java/compatibility_tests/v2.5.0 it
> appears that source and binary incompatibilities still exist.
>
>
> I'm curious if anyone can shed some light on this effort and its status or
> provide suggestions for migrating to a recent protobuf release under these
> circumstances.
>
Protobuf 2.5.0/2.6.1 should be compatible with 3.0.0-beta-4 if you only
uses protobuf public APIs. That's unfortunately not the case with Hadoop &
HBase though. They introduced a class into com.google.protobuf package with
the sole purpose to access protobuf package private classes
.
As
such there is no way you can upgrade to protobuf 3.x if the version of
Hadoop & HBase you use still depends on protobuf 2.5/2.6 private symbols.


> Thanks,
>
> Chris
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Issues identified in version 3.4.0 by security scan

['Feng Xiao' via Protocol Buffers]

Can you create an issue for this on https://github.com/google/protobuf ?

On Wed, Nov 15, 2017 at 6:26 AM, 
wrote:

> Tho following 3 issues were flagged by a security scan where we use
> Protobuf v3.4.0. Will these be resolved in an upcoming version?
>
>
> 1. Unchecked CString Convertion
>
> The element Param at line 273 of /Protobuf/objectivec/
> GPBCodedOutputStream.m contains a C-String that was converted from a
> CFString object. The length of Param was not checked after conversion.
>
> Method: (void) writeStringNoTag:(const NSString *) value
>
> Line: 294
>
> result = [value getBytes:state_.bytes + state_.position
>
>
> 2. Unchecked CString Convertion
>
> The element CFStringGetCStringPtr at line 273 of /Protobuf/objectivec/
> GPBCodedOutputStream.m contains a C-String that was converted from a
> CFString object. The length of CFStringGetCStringPtr was not checked after 
> conversion.
>
>
> Method: (void) writeStringNoTag:(const NSString *) value
>
> Line: 281
>
> CFStringGetCStringPtr((CFStringRef)value, kCFStringEncodingUTF8);
>
>
> 3. Improper Resource Shutdown or Release
>
> The application's - method in /Pods/Protobuf/objectivec/
> GPBCodedOutputStream.m defines and initializes the output object at 173.
> This object encapsulates a limited computing resource, such as open file
> streams, database connections, or network streams. This resource is not
> properly closed and released in all situations.
>
> Method: (instancetype)initWithOutputStream:(NSOutputStream *)output
>
> Line: 177
>
> [output open];
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Golang: issues/discussions/pull requests

['Feng Xiao' via Protocol Buffers]

Some new updates from go protobuf developers:
https://github.com/golang/protobuf/issues/454#issuecomment-343250894

An announcement about this change will be posted in this discussion group 
soon.

On Tuesday, October 24, 2017 at 12:06:22 PM UTC-7, Feng Xiao wrote:
>
> On Tue, Oct 24, 2017 at 8:12 AM Josh Humphries <jh...@bluegosling.com> 
> wrote:
>
>> In the past, when I've asked questions in this mailing list or hoped to 
>> discuss details of the protobuf package for Go, I've been pointed to the 
>> Github project.
>>
>> And, for some period, I had luck having discussions on Github issues and 
>> even getting pull requests looked at. However, lately, I've been having a 
>> very hard time getting questions answered or fixes reviewed.
>>
>> Is there something big that is diverting attention from the engineers 
>> working on Go protobuf stuff? Any suggestions on who I can tag in a Github 
>> issue or pull request? I understand that reviews could take time to get to 
>> (though the backlog is not very large and the rate of incoming PRs looks 
>> pretty low). But I can't even get a reply to any question or remark, much 
>> less a review.
>>
>> Is there another communication channel I should be trying to use? Maybe 
>> Slack?
>>
> +Pherl Liu <liuj...@google.com> should know who maintains Go protobuf now.
>  
>
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C++ - Protobuf descriptor static memory leaks

['Feng Xiao' via Protocol Buffers]

On Thu, Nov 9, 2017 at 11:53 AM, Arpit Baldeva  wrote:

> Hi,
>
> Currently, protobuf generated code makes allocations in the static
> descriptor objects. For example, valgrind reports
>
>  
>Memcheck:Leak
>fun:malloc
>fun:_ZN20PassthroughAllocator5AllocEmPKcj
>fun:_Znwm
>fun:_ZN6google8protobuf14DescriptorPoolC1EPNS0_
> 18DescriptorDatabaseEPNS1_14ErrorCollectorE
>fun:_ZN6google8protobuf12_GLOBAL__N_1L17InitGeneratedPoolEv
>fun:_ZN6google8protobuf8internal16FunctionClosure03RunEv
>fun:_ZN6google8protobuf18GoogleOnceInitImplEPlPNS0_7ClosureE
>fun:_ZN6google8protobuf14GoogleOnceInitEPlPFvvE
>fun:_ZN6google8protobuf12_GLOBAL__N_121InitGeneratedPoolOnceEv
>fun:_ZN6google8protobuf14DescriptorPool24InternalAddGeneratedFileEPKvi
>fun:_ZN5Blaze51protobuf_blaze_2fprotobuf_2fblazeextensions_2eproto12_
> GLOBAL__N_118AddDescriptorsImplEv
>fun:_ZN6google8protobuf8internal16FunctionClosure03RunEv
>fun:_ZN6google8protobuf18GoogleOnceInitImplEPlPNS0_7ClosureE
>fun:_ZN6google8protobuf14GoogleOnceInitEPlPFvvE
>fun:_ZN5Blaze51protobuf_blaze_2fprotobuf_2fblazeextensions_
> 2eproto14AddDescriptorsEv
>fun:_ZN5Blaze51protobuf_blaze_2fprotobuf_2fblazeextensions_
> 2eproto27StaticDescriptorInitializerC1Ev
>fun:_Z41__static_initialization_and_destruction_0ii
>fun:_GLOBAL__sub_I_blazeextensions.pb.cc
>
> Is there a convenience function that could deallocate these objects so
> that I don't have to suppress these allocations? Found this relevant issue
> - https://codereview.appspot.com/7918 - 9 years ago.
>
You can call ShutdownProtobufLibrary() before program exiting and after
that there should be no such "leaks".


>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Avoid C++11 warning about switch on proto3 enum w/o default clause?

['Feng Xiao' via Protocol Buffers]

On Mon, Oct 30, 2017 at 6:19 PM, <chhs...@mesosphere.io> wrote:

> Hi Feng,
>
> Thanks for answering.
>
> As suggested by the official documentation:
> https://developers.google.com/protocol-buffers/docs/
> reference/cpp-generated#enum
> We can handle the unknown enum value through either a `default` clause or
> an `if` check as follows:
>
> // Wrong version of client-side code:
> int HandleFoo(Foo foo) {
>   if (Foo::Type_IsValid(foo.type()) {
> switch (foo.type()) {
>   case FOO: return ...;
>   case BAR: return ...;
>   case google::protobuf::kint32min:
>   case google::protobuf::kint32max:
> // UNREACHABLE.
> }
>   } else {
> // Handle the unknown enum value properly.
>   }
> }
>
> So this way we can ensure that the unknown enum values are handled,
> and also the compiler can check if an enum value is missing in the
> `switch` statement for us.
> Is it future-proof to use `google::protobuf::kint32max` or
> `*_INT_MAX_SENTINEL_DO_NOT_USE_`?
>
No, it will break right away when we add another
"*_XXX_SENTINEL_DO_NOT_USE_", or when we want to rename "XXX_DO_NOT_USE" to
"XXX_DO_NOT_USE_PLEASE_ADD_DEFAULT_CASE".


> It seems to me that there's some merit to formalize this in the proto3
> language. Thoughts?
>
> Best,
> Chun-Hung
>
>
> On Monday, October 30, 2017 at 12:29:26 PM UTC-7, Feng Xiao wrote:
>>
>>
>>
>> On Mon, Oct 30, 2017 at 9:57 AM, <chh...@mesosphere.io> wrote:
>>
>>> Hi all,
>>>
>>> I'm from the Apache Mesos community and we're working on things that
>>> uses proto3.
>>> We usually turn on compiler options to report missing values in switch
>>> statements,
>>> and don't do a "default" clause. This gives us the benefit of capturing
>>> missing enum
>>> values during compile time. However, it seems that for proto3's open
>>> enum values,
>>> we need to either manually adding those `*_INT_MAX_SENTINEL_DO_NOT_USE_`
>>> values
>>> in switch statements, or add a default clause to avoid such errors but
>>> then we lose
>>> the benefit I mentioned above. It seems that the compiler is not clever
>>> enough
>>> to infer that clauses for those sentinel values are not needed for
>>> switch statements
>>> that are inside `if(Enum_IsValid(...))` statements.
>>>
>>> My question is, what's the best practice if I don't want to add a
>>> default clause?
>>> Is it safe to add either `*_INT_MAX_SENTINEL_DO_NOT_USE_` symbols
>>> or `google::protobuf::kint32max` into switch statements, such that these
>>> will
>>> remain unchanged in future versions of protobuf library? Or is there a
>>> better way
>>> to address this?
>>>
>> The recommendation is to always have a default case when handling proto3
>> enums. Unlike regular enums that fall in a defined list of possible values,
>> proto3 enums can have unrecognized values when a client receives data from
>> a remote server with a new enum definition. To ensure such unrecognized
>> values are correctly handled, a default case must be added.
>>
>> For example, suppose you have a client built with such a proto:
>> message Foo {
>>   enum Type {
>> FOO = 0;
>> BAR = 1;
>>   }
>>   Type type = 1;
>> }
>>
>> and later someone adds a new value into the enum and uses the new
>> definition on the server side:
>> message Foo {
>>   enum Type {
>> FOO = 0;
>> BAR = 1;
>> BAZ = 2;
>>   }
>>   Type type = 1;
>> }
>>
>> If your client receives data from the server with type = BAZ, foo.type()
>> will return 2, which you can't catch with any predefined values. If you
>> don't have default case, undefined behavior ensures.
>>
>> // Wrong version of client-side code:
>> int HandleFoo(Foo foo) {
>>   switch (foo.type()) {
>> case FOO: return ...;
>> case BAR: return ...;
>> case *_INT_MAX_SENTINEL_DO_NOT_USE: return ...;
>>   }
>>   // You can never get here, or maybe you can?
>> }
>>
>> // The correct version
>> int HandleFoo(Foo foo) {
>>   switch (foo.type()) {
>> case FOO: return ...;
>> case BAR: return ...;
>> default: return ...;  // There is an enum value we don't know about,
>> handle it appropriately.
>>   }
>>   // You can never get here.
>> }
>>
>>
>>>
>>> Thanks,
>>> Chun-Hung
>>>
>>> --
&g

Re: [protobuf] Avoid C++11 warning about switch on proto3 enum w/o default clause?

['Feng Xiao' via Protocol Buffers]

On Mon, Oct 30, 2017 at 9:57 AM,  wrote:

> Hi all,
>
> I'm from the Apache Mesos community and we're working on things that uses
> proto3.
> We usually turn on compiler options to report missing values in switch
> statements,
> and don't do a "default" clause. This gives us the benefit of capturing
> missing enum
> values during compile time. However, it seems that for proto3's open enum
> values,
> we need to either manually adding those `*_INT_MAX_SENTINEL_DO_NOT_USE_`
> values
> in switch statements, or add a default clause to avoid such errors but
> then we lose
> the benefit I mentioned above. It seems that the compiler is not clever
> enough
> to infer that clauses for those sentinel values are not needed for switch
> statements
> that are inside `if(Enum_IsValid(...))` statements.
>
> My question is, what's the best practice if I don't want to add a default
> clause?
> Is it safe to add either `*_INT_MAX_SENTINEL_DO_NOT_USE_` symbols
> or `google::protobuf::kint32max` into switch statements, such that these
> will
> remain unchanged in future versions of protobuf library? Or is there a
> better way
> to address this?
>
The recommendation is to always have a default case when handling proto3
enums. Unlike regular enums that fall in a defined list of possible values,
proto3 enums can have unrecognized values when a client receives data from
a remote server with a new enum definition. To ensure such unrecognized
values are correctly handled, a default case must be added.

For example, suppose you have a client built with such a proto:
message Foo {
  enum Type {
FOO = 0;
BAR = 1;
  }
  Type type = 1;
}

and later someone adds a new value into the enum and uses the new
definition on the server side:
message Foo {
  enum Type {
FOO = 0;
BAR = 1;
BAZ = 2;
  }
  Type type = 1;
}

If your client receives data from the server with type = BAZ, foo.type()
will return 2, which you can't catch with any predefined values. If you
don't have default case, undefined behavior ensures.

// Wrong version of client-side code:
int HandleFoo(Foo foo) {
  switch (foo.type()) {
case FOO: return ...;
case BAR: return ...;
case *_INT_MAX_SENTINEL_DO_NOT_USE: return ...;
  }
  // You can never get here, or maybe you can?
}

// The correct version
int HandleFoo(Foo foo) {
  switch (foo.type()) {
case FOO: return ...;
case BAR: return ...;
default: return ...;  // There is an enum value we don't know about,
handle it appropriately.
  }
  // You can never get here.
}


>
> Thanks,
> Chun-Hung
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Integrate proto2 messages from a device to C#

['Feng Xiao' via Protocol Buffers]

On Fri, Oct 27, 2017 at 10:16 AM, cleal  wrote:

> I have to integrate a third part protocol from a gps device, they send me
> the .proto files, they are using the schema proto2.
>
> I should receive the binary messages and I need desarialize it in order to
> work with the location data.  But I see in forums that C# only works for
> proto3.  There is no way to migrate the messages because they are from a
> hardware device from a third part.
>
> I have never working with the Protocol Buffers, how I should start?
>
Proto2 and proto3 are binary compatible, so if the .proto files don't
contain any proto2-only features (e.g., extensions), you can convert them
to proto3 syntax and it should be able to deserialize the same binary
message.

If these .proto files can't be converted to proto3, there are third-party
C# protobuf implementations that you can use. For example:
https://github.com/mgravell/protobuf-net


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to use custom options in official docs

['Feng Xiao' via Protocol Buffers]

On Wed, Oct 25, 2017 at 5:38 AM,  wrote:

> the example in https://developers.google.com/protocol-buffers/docs/
> proto#customoptions below
>
> message FooOptions {
>   optional int32 opt1 = 1;
>   optional string opt2 = 2;
> }
>
> extend google.protobuf.FieldOptions {
>   optional FooOptions foo_options = 1234;
> }
>
> // usage:
> message Bar {
>   optional int32 a = 1 [(foo_options).opt1 = 123, (foo_options).opt2 = "baz"];
>   // alternative aggregate syntax (uses TextFormat):
>   optional int32 b = 2 [(foo_options) = { opt1: 456 opt2: "xaa" }];
> }
>
> assume I have a message Bar here:
>
> var msg Bar
> _, md := ForMessage()
> md.Field[0].GetOptions().String()  only a string, How do i parse that ??
>
> How to I get the exact value of Bar.b.opt1 which is 123 (default value)
>
> What I want is like this
>
> md.Field[0].GetOptions().GetOpt1()   // 123
> md.Field[0].GetOptions().GetOpt2()// baz
> md.Field[1].GetOptions().GetOpt1()   // 456
> md.Field[1].GetOptions().GetOpt2()   // xaa
>
> Can we achieve that ?
>
The exact API to access custom options is language specific, It generally
uses the same syntax as accessing extensions. For go, you can post the
question to https://github.com/golang/protobuf


> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Deserializing of protobf Message from JSON string in C++

['Feng Xiao' via Protocol Buffers]

On Wed, Oct 25, 2017 at 1:46 AM, Giri Guntipalli <giri.guntipa...@gmail.com>
wrote:

> I am also facing similar issue but problem seems to happen only when
> library is complied with -O2 or –Os(optimize for size)  with gcc tool chain.
>
>
>
> looks like extensions are being treated as dead code and removed by gcc
> optimization, any thoughts on overcoming this problem.
>
The extension has to be explicitly referenced or the linker will strip them
out.


>
>
> Thank you
>
> Giri
> On Thursday, 6 June 2013 00:15:14 UTC+5:30, Feng Xiao wrote:
>>
>>
>>
>>
>> On Wed, Jun 5, 2013 at 2:44 AM, Channakeshava S C <channake...@gmail.com>
>> wrote:
>>
>>> I am facing problem in getting protobuf Message with extension from JSON
>>> string  {"xyz": [1234,2345,3456],"extensions":{"bar": [1,11,111]}}
>>> which is generated from Foo.
>>>
>>> message Foo {
>>>  repeated int32 xyz = 11;
>>>  extensions 100 to 199;
>>> }
>>>
>>> extend Foo {
>>>  repeated int32 bar = 123;
>>> }
>>>
>>> The word "extensions" in the JSON stirng is just added to differentiate
>>> with other normal fileds.
>>>
>>> Now, when I try to deserialize protobuf, i.e protobuf Message from JSON
>>> string. Facing difficulty here.
>>>
>>> fd is NULL, when FindExtensionByName or FindKnownExtensionByName is
>>> called .
>>>
>>> Foo foo;
>>> Message *x = 
>>> const FiledDescriptor *fd;
>>>
>>> fd = x->GetDescriptor()->FindExtensionByName("bar")or
>>> fd = x->GetReflection()->FindKnownExtensionByName("bar");
>>>
>> Is the extension defined in the same .proto file with the message?  The
>> second one should work.
>>
>>
>>>
>>> Could you point out any error in steps which I am following or Is there
>>> any other better approach?
>>>
>>>
>>> Thanks,
>>> Channakeshava
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/protobuf?hl=en.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Golang: issues/discussions/pull requests

['Feng Xiao' via Protocol Buffers]

On Tue, Oct 24, 2017 at 8:12 AM Josh Humphries 
wrote:

> In the past, when I've asked questions in this mailing list or hoped to
> discuss details of the protobuf package for Go, I've been pointed to the
> Github project.
>
> And, for some period, I had luck having discussions on Github issues and
> even getting pull requests looked at. However, lately, I've been having a
> very hard time getting questions answered or fixes reviewed.
>
> Is there something big that is diverting attention from the engineers
> working on Go protobuf stuff? Any suggestions on who I can tag in a Github
> issue or pull request? I understand that reviews could take time to get to
> (though the backlog is not very large and the rate of incoming PRs looks
> pretty low). But I can't even get a reply to any question or remark, much
> less a review.
>
> Is there another communication channel I should be trying to use? Maybe
> Slack?
>
+Pherl Liu  should know who maintains Go protobuf now.


>
> 
> *Josh Humphries*
> jh...@bluegosling.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf well-known types and C++

['Feng Xiao' via Protocol Buffers]

On Sat, Oct 21, 2017 at 4:39 PM, Antonia Elsen 
wrote:

> What is the proper way to include well-known types in proto files?
> I'm hoping to use FieldMasks and Any among others.
> So far I haven't had any luck.
> I've tried importing the proto files and using the namespace in the type
> name;
>
> ex:
> import "include/google/protobuf/field_mask.proto"
> ...
> google.protobuf.FieldMask field_mask = 1;
>
>
> with no success.
> Should importing the well-known type be necessary, or are they included
> with the protoc compiler?
>
You can follow the example proto in protobuf repo:
https://github.com/google/protobuf/blob/master/examples/addressbook.proto#L13

Follow this readme file to build the example:
https://github.com/google/protobuf/tree/master/examples


>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Best practice for effecting inheritance/polymorphism in proto3

['Feng Xiao' via Protocol Buffers]

On Wed, Oct 18, 2017 at 3:04 PM, Jeff White  wrote:

> There seem to be 2 techniques to effect pseudo-inheritance/polymorphism.
>
> oneof:
>
> message Animal {
> string name = 1;
> oneof animal_type {
> Bird bird = 2;
> Dog dog = 3;
> }
> }
> message Bird {
> bool can_fly = 1;
> }
> message Dog {
> string breed = 1;
> }
>
> mixin common fields:
>
> message Animal {
>   string name = 1;
> }
>
> message Bird {
> Animal animal = 1;
> bool can_fly = 2;
> }
>
> message Dog {
> Animal animal = 1;
> string breed = 2;
> }
>
> Using oneof makes it easier to define a generic collection of Animals,
> though all the subtypes need to be defined by the author of Animal.
> Using the mixin strategy makes it easier for any "subclass" to extend the
> base type.
>
> I'm trying to understand all the tradeoffs and am curious if people have
> developed best practices on this. Are there other considerations?
>
I think the oneof approach is more common and more useful. Protobuf wire
format doesn't carry message type info so if you use the second mixin
pattern and send a Bird or Dog over the wire to another client, the other
client can't really tell whether it's a Bird or Dog. Instead if you are
sending a oneof, the other client can distinguish the difference by
inspecting the oneof field.

If you don't want to define all types in the base Animal type, you can use
the Any message instead:
message Animal {
  string name = 1;
  google.protobuf.Any any = 2;
}

Then if the other client knows about Bird, it can check whether the type is
actually bird:

Animal animal = ...;
Bird bird;
if (animal.any().UnpackTo()) {
  // This is a bird.
  ...
}

>
> jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "ParseFromString()" and "JsonStringToMessage()" have inconsistent behavior when parsing an unrecognized enum value.

['Feng Xiao' via Protocol Buffers]

On Tue, Oct 10, 2017 at 8:15 PM, Qian Zhang <zhq527...@gmail.com> wrote:

> Note that you can use the always_print_enums_as_ints option to work around
>> this problem:
>
> https://github.com/google/protobuf/blob/master/src/
>> google/protobuf/util/json_util.h#L73
>>
>
>
> With this option enum value will printed as integers and will be accepted
>> by JsonStringToMessage.
>
>
> "always_print_enums_as_ints" is an option for the method
> "MessageToJsonString()", but the issue that I am talking about is the
> method "JsonStringToMessage()", the only option that
> "JsonStringToMessage()" accepts is "JsonParseOptions.ignore_unknown_fields",
> but I do not think it can help here.
>
> This is the expected behavior. The issue at core is that protobuf message
>> is only able to hold unknown binary data. It's designed that way with its
>> UnknownFieldSet data structure and you can not achieve the same with other
>> formats.
>
>
> But that behavior is not desired for us :-( We are implementing a server
> which can accept both protobuf serialized string and JSON string from
> clients, and then the server will call "ParseFromString()" and
> "JsonStringToMessage" to get the protobuf message from the serialized
> string and JSON string respectively. But now these two methods have
> different behaviors, "ParseFromString()" can successfully parse a
> serialized string which contains unrecognized enum value, but
> "JsonStringToMessage()" will fail to do that. This makes our server can not
> behave consistently for protobuf serialized string and JSON string.
>
> Any suggestions? Thanks!
>
There really isn't a good way to solve this. The server has to know the
enum definition to be able to accept it in JSON format. You will either
need to make sure the server is updated with the new definition before
sending it JSON data using the new enum names, or use integer instead of
enum names in the request.


>
>
> Regards,
> Qian Zhang
>
> On Wed, Oct 11, 2017 at 5:20 AM, Feng Xiao <xiaof...@google.com> wrote:
>
>> Note that you can use the always_print_enums_as_ints option to work
>> around this problem:
>> https://github.com/google/protobuf/blob/master/src/google/
>> protobuf/util/json_util.h#L73
>>
>> With this option enum value will printed as integers and will be accepted
>> by JsonStringToMessage.
>>
>> On Tue, Oct 10, 2017 at 2:18 PM, Feng Xiao <xiaof...@google.com> wrote:
>>
>>>
>>>
>>> On Tue, Oct 10, 2017 at 12:29 AM, Qian Zhang <zhq527...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using protobuf-3.3.0, and I found when parsing an unrecognized
>>>> enum value for an optional enum field, the behaviors of "ParseFromString()"
>>>> and "JsonStringToMessage()" are different. "ParseFromString()" will succeed
>>>> and the field's getter method will return the default enum value, but
>>>> "JsonStringToMessage" will fail with an error:
>>>>
>>>>> invalid value "xxx" for type TYPE_ENUM
>>>>>
>>>>
>>>> Is this a bug of protobuf-3.3.0? IMHO, "ParseFromString()" and
>>>> "JsonStringToMessage()" should have consistent behavior.
>>>>
>>> This is the expected behavior. The issue at core is that protobuf
>>> message is only able to hold unknown binary data. It's designed that way
>>> with its UnknownFieldSet data structure and you can not achieve the same
>>> with other formats.
>>>
>>>
>>>>
>>>>
>>>> Thanks,
>>>> Qian
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to protobuf@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "ParseFromString()" and "JsonStringToMessage()" have inconsistent behavior when parsing an unrecognized enum value.

['Feng Xiao' via Protocol Buffers]

Note that you can use the always_print_enums_as_ints option to work around
this problem:
https://github.com/google/protobuf/blob/master/src/google/protobuf/util/json_util.h#L73

With this option enum value will printed as integers and will be accepted
by JsonStringToMessage.

On Tue, Oct 10, 2017 at 2:18 PM, Feng Xiao <xiaof...@google.com> wrote:

>
>
> On Tue, Oct 10, 2017 at 12:29 AM, Qian Zhang <zhq527...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using protobuf-3.3.0, and I found when parsing an unrecognized enum
>> value for an optional enum field, the behaviors of "ParseFromString()" and
>> "JsonStringToMessage()" are different. "ParseFromString()" will succeed and
>> the field's getter method will return the default enum value, but
>> "JsonStringToMessage" will fail with an error:
>>
>>> invalid value "xxx" for type TYPE_ENUM
>>>
>>
>> Is this a bug of protobuf-3.3.0? IMHO, "ParseFromString()" and
>> "JsonStringToMessage()" should have consistent behavior.
>>
> This is the expected behavior. The issue at core is that protobuf message
> is only able to hold unknown binary data. It's designed that way with its
> UnknownFieldSet data structure and you can not achieve the same with other
> formats.
>
>
>>
>>
>> Thanks,
>> Qian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "ParseFromString()" and "JsonStringToMessage()" have inconsistent behavior when parsing an unrecognized enum value.

['Feng Xiao' via Protocol Buffers]

On Tue, Oct 10, 2017 at 12:29 AM, Qian Zhang  wrote:

> Hi,
>
> I am using protobuf-3.3.0, and I found when parsing an unrecognized enum
> value for an optional enum field, the behaviors of "ParseFromString()" and
> "JsonStringToMessage()" are different. "ParseFromString()" will succeed and
> the field's getter method will return the default enum value, but
> "JsonStringToMessage" will fail with an error:
>
>> invalid value "xxx" for type TYPE_ENUM
>>
>
> Is this a bug of protobuf-3.3.0? IMHO, "ParseFromString()" and
> "JsonStringToMessage()" should have consistent behavior.
>
This is the expected behavior. The issue at core is that protobuf message
is only able to hold unknown binary data. It's designed that way with its
UnknownFieldSet data structure and you can not achieve the same with other
formats.


>
>
> Thanks,
> Qian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is it possible to retrieve an absolute path to a proto-file from a Protoc plugin?

['Feng Xiao' via Protocol Buffers]

On Mon, Sep 25, 2017 at 5:21 AM, Andrew Pashkin 
wrote:

> Hello, everyone,
>
> I have a case where I need to read non-Protobuf files that are referenced
> from a target proto-file by a relative path. I use this approach to embed
> some information in a language different than Protobuf.  It seems like
> Protoc plugins only have access to the path of target proto-files relative
> to one of the directories provided using --proto_path command-line
> options of protoc.  And I'm trying to find a way to locate the target
> proto-file on the disk and with that locate the other files referenced from
> it by relative paths. Does anybody know a way how to do that?
>
This is not possible. If you look at the protoc-plugin protocol message, it
contains only the processed FileDescriptorProto without any original file
paths:
https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/plugin.proto#L90

Most people embed additional info in .proto files using custom options. I
think you can do the same .

> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] table_driven_parsing option in protobuf 3.4 C++ library

['Feng Xiao' via Protocol Buffers]

On Mon, Sep 18, 2017 at 3:13 PM, Todd Lipcon  wrote:

> Hi folks,
>
> I noticed that protobuf 3.4 includes the new 'table_driven_parsing' option
> for C++ code generation, but can't seem to find any docs about this feature.
>
> From a quick skim of the code, it seems like the goal here is to reduce
> the amount of per-PB generated code, and instead replace it with a generic
> method which can parse/serialize any protobuf based on a constant table
> indicating the field/type/class-offset mappings. My guess is the purpose
> here is to reduce code size and thus increase instruction cache locality
> for applications which handle either very large PBs or a large number of
> varying PB types, and potentially to unify the dynamic and generated
> parsing code paths.
>
> Is my understanding correct?
>
You are spot-on :)

Besides what you mentioned, this table-driven approach will also allow us
to add more specialized parser implementations (e.g., a parsing method that
reads from char* directly without a wrapping CodedInputStream, or maybe
support an entirely different wire format for better performance in some
specific context) without causing code bloat.



> Is there any speed benefit to using table-driven parsing for applications
> which deal with a relatively smaller set of unique PB types? Would be
> interested to hear about experiences at Google or elsewhere with this new
> option.
>
It's still highly experimental and I would not recommend using this option
at the moment (unless you want to improve and contribute code to this
implementation). Internal benchmarks show that it's slower than generated
code and the code size reduction is less than we expected. We might enable
it to replace the reflection-based dynamic parsing code since that's an
easy win, but more work is required before it can replace generated code.


>
> -Todd
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to get value of protobuf custom option in Java?

['Feng Xiao' via Protocol Buffers]

Suppose you have two .proto files, an options.proto where the extension is
defined, and a foo.proto where the custom option is used.

If both proto files are actually part of your protoc plugin (i.e., your
plugin includes the generated Java class of both .proto files), you should
be able to access the option values in foo.proto without problem.

If options.proto is part of your protoc plugin, and foo.proto is the proto
file that's being compiled by protoc + your plugin, you need to provide an
ExtensionRegistry when parsing the CodeGeneratorRequest message. After that
custom options included in the ExtensionRegistry should be accessible.

On Fri, Sep 1, 2017 at 5:34 PM, Noel Yap  wrote:

> Hmm, I'm wondering if that starting block never actually executes.
>
> On Fri, Sep 1, 2017, 17:05 Noel Yap  wrote:
>
>> I'm seeing the generated code having in a `static` init block:
>> ```
>> com.google.protobuf.Descriptors.FileDescriptor
>>   .internalBuildGeneratedFileFrom(descriptorData,
>> new com.google.protobuf.Descriptors.FileDescriptor[] {
>>   com.example.proto.options.api_docs.ApiDocsOptionsProto.
>> getDescriptor(),
>>   com.example.proto.options.ServerOptionsProto.getDescriptor(),
>> }, assigner);
>> com.google.protobuf.ExtensionRegistry registry =
>> com.google.protobuf.ExtensionRegistry.newInstance();
>> registry.add(com.example.proto.options.ServerOptionsProto.server);
>> com.google.protobuf.Descriptors.FileDescriptor
>> .internalUpdateFileDescriptor(descriptor, registry);
>> com.example.proto.options.api_docs.ApiDocsOptionsProto.
>> getDescriptor();
>> com.example.proto.options.ServerOptionsProto.getDescriptor();
>> ```
>>
>> On Fri, Sep 1, 2017 at 4:54 PM Adam Cozzette 
>> wrote:
>>
>>> You may also need to provide an ExtensionRegistry as described here
>>> 
>>> when you parse the proto, in order for the extension to be visible.
>>>
>>> On Fri, Sep 1, 2017 at 3:55 PM, Noel Yap  wrote:
>>>
 One piece of possibly important information. The code being written is
 for a `protoc` plugin. I've updated the Stackoverflow question to state
 that.

 On Fri, Sep 1, 2017 at 3:42 PM Noel Yap  wrote:

> proto.getDescriptor().getFile().getOptions() didn't work. If I print
> that out, I see:
>
> java_package: \"com.google.protobuf\"\njava_outer_classname:
> \"DescriptorProtos\"\noptimize_for: SPEED\ngo_package: \"
> github.com/golang/protobuf/protoc-gen-go/descriptor;descriptor\
> 
> "\nobjc_class_prefix: \"GPB\"\ncsharp_namespace:
> \"Google.Protobuf.Reflection\"
>
> I've also tried `proto.getOptions()`. That shows the options like
> `java_package` but `hasExtension` for the extension I'm setting is still
> returning `false`.
>
> On Fri, Sep 1, 2017 at 2:53 PM Adam Cozzette 
> wrote:
>
>> I posted an answer on your StackOverflow post but let me know if you
>> still have any trouble with that.
>>
>> On Wed, Aug 30, 2017 at 4:13 PM, Noel Yap  wrote:
>>
>>> https://stackoverflow.com/questions/45970986/how-to-get-
>>> value-of-protobuf-custom-option-in-java
>>>
>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to protobuf+unsubscr...@googlegroups.com.
>>
>>
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is there any way to add a prefix to canonical name?

['Feng Xiao' via Protocol Buffers]

On Fri, Aug 11, 2017 at 1:41 PM, 'Jungho Ahn' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hello,
>
> I'm trying to import protos from a third party repo, but they don't have
> proper canonical names.
>
> For example, there is a repo X which has
>
> a/p1.proto
> a/p2.proto with (import "a/p1.proto")
>
>
> Since 'a' is too generic, I'd like to add a kind of prefix to the
> canonical names like "external/a/p1.proto", "external/a/p2.proto" to avoid
> conflicts with existing protos.
> But I'm not sure whether this possible since a/p2.proto imports p1.proto
> as "a/p1.proto" already.
>
> Is there any workaround for this?
>
Unfortunately there isn't a workaround here. Basically "a/p1.proto" becomes
the canonical name of that proto file and it can't be changed unless you
can update all references of that import path. If repo X only provides the
.proto files but not the generated code in their library, you can just
update all the import paths in their protos. However, if generated code is
part of X library, there isn't much you can do other than asking owners of
X library to use a better path for their protos.


>
> Thanks,
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Compiler error with GCC targeting ARM - Atomic32 vs AtomicWord

['Feng Xiao' via Protocol Buffers]

On Thu, Aug 10, 2017 at 9:53 AM, Brad Larson <bklar...@gmail.com> wrote:

>
>
> On Wednesday, August 9, 2017 at 1:13:54 PM UTC-5, Feng Xiao wrote:
>>
>> Hi Brad, can you send your patch as a pull request? It's easier to review
>> the diff there.
>>
>
> Hi Feng, thank you for the reply.  I really didn't have a fix in mind and
> was looking for more of a discussion on what a better fix might be.
> However I consulted with a coworker and we came up with this:
> https://github.com/google/protobuf/pull/3480.  It does solve my problem,
> but I'm unable to test on the wide variety of hardware platforms supported
> by GPB.
>
Thanks! I think your pull request makes the code cleaner. Made a few minor
style comments there.


>
>
>>
>> On Wed, Aug 9, 2017 at 8:27 AM, Brad Larson <bkla...@gmail.com> wrote:
>>
>>>
>>>
>>> On Monday, August 7, 2017 at 1:13:46 PM UTC-5, Brad Larson wrote:
>>>>
>>>> Hello, I'm getting a compiler error when using GCC to target ARM
>>>> platforms.  I've patched this locally, but would like to upstream my patch
>>>> and am not sure on the best generally-acceptable solution.
>>>>
>>>> src/google/protobuf/io/coded_stream.h: In static
>>>> member function 'static bool google::protobuf::io::CodedOut
>>>> putStream::IsDefaultSerializationDeterministic()':
>>>> C:\projects\cycling\carlton\external\protobuf\src/google/pro
>>>> tobuf/io/coded_stream.h:868:53: error: invalid conversion from
>>>> 'google::protobuf::internal::AtomicWord* {aka int*}' to 'const volatile
>>>> Atomic32* {aka const volatile long int*}' [-fpermissive]
>>>>  return google::protobuf::internal::Ac
>>>> quire_Load(_serialization_deterministic_);
>>>>
>>>>  ^
>>>>
>>>> On my platform, int32 is defined as long int and intptr_t is defined
>>>> as int (They both are 4 bytes).  This is causing the above error.
>>>>
>>>> Locally, I can easily change the type of Atomic32 to intptr_t or just
>>>> int, or change the type of AtomicWord to int32 or intptr_t.  But this
>>>> isn't generally acceptable of course.
>>>>
>>>> I'm not sure on what a better solution might be.  I see we already have
>>>> a check #if GOOGLE_PROTOBUF_ARCH_POWER to typedef Atomic32 to intptr_t
>>>> in some cases, and extra logic around #if defined(__ILP32__) ||
>>>> defined(GOOGLE_PROTOBUF_OS_NACL).  Could this be simplified with a #if
>>>> sizeof intptr_t == sizeof int32 check (or similar)?
>>>>
>>>> Very open to any suggestions on how to upstream a fix for this!
>>>>
>>>> Thanks,
>>>> Brad
>>>>
>>>
>>>
>>> If it is helpful, here are the current typedefs, from atomicops.h:
>>>
>>> #if defined(GOOGLE_PROTOBUF_ARCH_POWER)
>>> #if defined(_LP64) || defined(__LP64__)
>>> typedef int32 Atomic32;
>>> typedef intptr_t Atomic64;
>>> #else
>>> typedef intptr_t Atomic32;
>>> typedef int64 Atomic64;
>>> #endif
>>> #else
>>> typedef intptr_t Atomic32;
>>> #ifdef GOOGLE_PROTOBUF_ARCH_64_BIT
>>> // We need to be able to go between Atomic64 and AtomicWord implicitly.
>>> This
>>> // means Atomic64 and AtomicWord should be the same type on 64-bit.
>>> #if defined(__ILP32__) || defined(GOOGLE_PROTOBUF_OS_NACL)
>>> // NaCl's intptr_t is not actually 64-bits on 64-bit!
>>> // http://code.google.com/p/nativeclient/issues/detail?id=1162
>>> // sparcv9's pointer type is 32bits
>>> typedef int64 Atomic64;
>>> #else
>>> typedef intptr_t Atomic64;
>>> #endif
>>> #endif
>>> #endif
>>>
>>> // Use AtomicWord for a machine-sized pointer.  It will use the Atomic32
>>> or
>>> // Atomic64 routines below, depending on your architecture.
>>> typedef intptr_t AtomicWord;
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Compiler error with GCC targeting ARM - Atomic32 vs AtomicWord

['Feng Xiao' via Protocol Buffers]

Hi Brad, can you send your patch as a pull request? It's easier to review
the diff there.

On Wed, Aug 9, 2017 at 8:27 AM, Brad Larson  wrote:

>
>
> On Monday, August 7, 2017 at 1:13:46 PM UTC-5, Brad Larson wrote:
>>
>> Hello, I'm getting a compiler error when using GCC to target ARM
>> platforms.  I've patched this locally, but would like to upstream my patch
>> and am not sure on the best generally-acceptable solution.
>>
>> src/google/protobuf/io/coded_stream.h: In static
>> member function 'static bool google::protobuf::io::CodedOut
>> putStream::IsDefaultSerializationDeterministic()':
>> C:\projects\cycling\carlton\external\protobuf\src/google/pro
>> tobuf/io/coded_stream.h:868:53: error: invalid conversion from
>> 'google::protobuf::internal::AtomicWord* {aka int*}' to 'const volatile
>> Atomic32* {aka const volatile long int*}' [-fpermissive]
>>  return google::protobuf::internal::Acquire_Load(_serializat
>> ion_deterministic_);
>>
>>  ^
>>
>> On my platform, int32 is defined as long int and intptr_t is defined as
>> int (They both are 4 bytes).  This is causing the above error.
>>
>> Locally, I can easily change the type of Atomic32 to intptr_t or just int,
>> or change the type of AtomicWord to int32 or intptr_t.  But this isn't
>> generally acceptable of course.
>>
>> I'm not sure on what a better solution might be.  I see we already have a
>> check #if GOOGLE_PROTOBUF_ARCH_POWER to typedef Atomic32 to intptr_t in
>> some cases, and extra logic around #if defined(__ILP32__) ||
>> defined(GOOGLE_PROTOBUF_OS_NACL).  Could this be simplified with a #if
>> sizeof intptr_t == sizeof int32 check (or similar)?
>>
>> Very open to any suggestions on how to upstream a fix for this!
>>
>> Thanks,
>> Brad
>>
>
>
> If it is helpful, here are the current typedefs, from atomicops.h:
>
> #if defined(GOOGLE_PROTOBUF_ARCH_POWER)
> #if defined(_LP64) || defined(__LP64__)
> typedef int32 Atomic32;
> typedef intptr_t Atomic64;
> #else
> typedef intptr_t Atomic32;
> typedef int64 Atomic64;
> #endif
> #else
> typedef intptr_t Atomic32;
> #ifdef GOOGLE_PROTOBUF_ARCH_64_BIT
> // We need to be able to go between Atomic64 and AtomicWord implicitly.
> This
> // means Atomic64 and AtomicWord should be the same type on 64-bit.
> #if defined(__ILP32__) || defined(GOOGLE_PROTOBUF_OS_NACL)
> // NaCl's intptr_t is not actually 64-bits on 64-bit!
> // http://code.google.com/p/nativeclient/issues/detail?id=1162
> // sparcv9's pointer type is 32bits
> typedef int64 Atomic64;
> #else
> typedef intptr_t Atomic64;
> #endif
> #endif
> #endif
>
> // Use AtomicWord for a machine-sized pointer.  It will use the Atomic32 or
> // Atomic64 routines below, depending on your architecture.
> typedef intptr_t AtomicWord;
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Protobuf on ARM pads integers

['Feng Xiao' via Protocol Buffers]

On Tue, Aug 1, 2017 at 4:13 AM, Brian Savage  wrote:

> This problem has reared it's ugly head again.  This time, some enums were
> changed and it caused a crash in an unrelated message when serialized.
> Padding the message with a bool type at the end of the message fixed it.
>
> It is scary that touching an unrelated part of the .proto file can cause a
> seg fault (memory free) in the protobuf code when you touch something
> totally unrelated.
>
What version of protobuf are you using? When you bulid protobuf, does it
pass the unit tests?


>
>
>
>
>
> On Thursday, April 28, 2016 at 5:58:42 PM UTC-4, Brian Savage wrote:
>>
>> I'm using an ARM Cortex A9 running Ubuntu Linux.  My application uses
>> protobufs.  I have successfully implemented much of my interface, but there
>> are a couple of messages that have an integer that is getting padded.  For
>> example, if  send decimal 50 (0x32), when it's parsed from the
>> protomessage, I get 0x32 0x00.  The raw data on the send and receive sides
>> match up (ethernet).  Any ideas?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Contribution Guide?

['Feng Xiao' via Protocol Buffers]

On Tue, Jul 18, 2017 at 7:04 AM, Brian Bustamante  wrote:

> Hi Adam,
>
> Who would be the best person to talk about possibly adding support for a
> new language to the protobuf compiler?
> I have opened a mostly feature complete proto3 PR for F#:
> https://github.com/google/protobuf/pull/3378
>
We can't accept a new language implementation into the protobuf repo unless
there is a Google engineer who can take on the maintenance work for the new
language, that invokes not only code changes and answering questions, but
also work like setting up automatic tests, making regular releases and
completing documentation. Right now these are tied to our internal
services/tools so an external contributor can't really help much there. If
there is a Google engineer who is willing to take on the work for F#, I
think we are happy to have him/her work with you on getting the F# support
into protobuf repo, but otherwise we don't really have the resource to
support a new language right now and the only option is to implement the
new language support in your own repo...

Jisi is the tech lead of protobuf team and will be the ultimate decider on
this. I saw he already replied on your pull request.


>
>
> Thanks,
> Brian
>
>
> On Monday, July 17, 2017 at 3:14:01 PM UTC-4, Adam Cozzette wrote:
>>
>> I think for these small tweaks and fixes it doesn't matter much to us
>> whether there's an associated GitHub issue or not, so whichever way you
>> prefer is fine with me!
>>
>> On Mon, Jul 17, 2017 at 11:03 AM, Brad Larson  wrote:
>>
>>>
>>>
>>> On Monday, July 17, 2017 at 12:20:22 PM UTC-5, Adam Cozzette wrote:

 Hi Brad,

 We don't have any real contribution guide but just accept pull requests
 fairly informally. Probably the one really important guideline is that if
 you want to add a substantial new feature or make a major change, it's best
 to talk to us (the protobuf team) first before spending much time
 implementing it. If it's a simple tweak or bugfix or whatever, though, feel
 free to just send off a pull request. We just tend to be somewhat
 conservative about making major changes, because we have have to preserve
 backward compatibility with a great deal of existing code, and we are also
 hesitant to add new features or options that will create extra maintenance
 work in the future.

 Your pull request looks great (thanks for sending that); I am just
 waiting for the CI runs to finish but I'll merge it once they've run
 successfully.

 Adam

>>>
>>> Thanks Adam!  Your guidelines above are very reasonable and I appreciate
>>> the response.  I see my pull request has just been merged so I'll start
>>> cleaning up the other handful of minor fixes I have for our more strict
>>> build environment and push them up.
>>>
>>> One follow-up question - for small tweaks/fixes like these, which mostly
>>> are addressing compiler warnings, is an associated github issue helpful?
>>> Or does it just create more process on your end?  Either way is fine with
>>> me!  :)
>>>
>>> Thanks again,
>>> Brad
>>>
>>>
>>>

 On Fri, Jul 14, 2017 at 10:14 AM, Brad Larson 
 wrote:

> Hi all!  I'm curious if there is a contribution guide for protobuf, or
> anything I should be doing to help get my patch merged.  It is very small
> and fixes a build error in our (warning-aggressive) environment at 
> $DAYJOB.
>
> I've created an issue - https://github.com/google/protobuf/issues/3356
> and a pull request - https://github.com/google/protobuf/pull/3357
>
> Reading a bit more on other pull requests, I'm not sure if I should
> have added this to the master branch or the 3.3.x branch?  I'm just 
> hopeful
> to get this merged so we can run stock GPB in the future.  I have a few
> other small tweaks like this as well, but want to get this one taken care
> of first.
>
> Thanks!
> Brad
>
> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are 

Re: [protobuf] Where to find Protobuf Javascript tutorial?

['Feng Xiao' via Protocol Buffers]

On Sat, Jul 15, 2017 at 8:28 PM, hce h  wrote:

> Hi,
>
> We are using javascript for web front end development and we are using the
> protobuf to communicate with backend web applications. Currently we are
> using GO to do protobuf communication between the backend application
> server which runs on C++ and the front end  running on Javascript, but I've
> just found there is a protobuf interface for Javascript as well, we can
> actually simplify the process directly to use javascript to fetch the
> protobuf messages with backend application without GO. Is it a good idea? I
> have since been investigating the javascript protobuf capability but I
> could not find javascript progotbuf examples and tutorials, the link
> https://developers.google.com/protocol-buffers/docs/tutorials shows
> tutorials for every program language except javascript. Appriciate anyone
> could point me likes for Javascript prototobuf examples and tutorials.
>
> I have also downloaded protobuf-js-3.3.0.zip, but it looks like it is
> similar to
> protobuf-cpp-3.3.0.tar.gz
> which I built on Linux platform, I guess the
> protobuf-js-3.3.0.zip
> is not the binary for javascript, link the C++ I need to build it as well,
> correct?
> 
>
protobuf-js-3.3.0.zip is the original source code for C++ + Javascript. If
you only need the already-built protobuf js package, try npm
google-protobuf:
https://www.npmjs.com/package/google-protobuf

Note that you will need to download protoc binary to do the code generation.


>
>
> Thank you.
>
> Kind regards,
>
>  - j
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] A timeout in this test probably indicates a real bug.

['Feng Xiao' via Protocol Buffers]

The failing test case
is: CsharpBootstrapTest.GeneratedCsharpDescriptorMatches

This is an known issue. It happens when you use protobuf-cpp-3.3.0.tar.gz
which doesn't contain the C# source file for that test case to pass. We
fixed the issue in master and you can safely ignore the test failure.

On Wed, Jul 12, 2017 at 1:44 PM, Sara Quigley <sara.quig...@gmail.com>
wrote:

> Sure. Here it is.
>
> Thanks,
> Sara
>
> On Wed, Jul 12, 2017 at 1:40 PM, Feng Xiao <xiaof...@google.com> wrote:
>
>> Can you attach the full log file?
>>
>> On Wed, Jul 12, 2017 at 1:27 PM, Sara Quigley <sara.quig...@gmail.com>
>> wrote:
>>
>>> Hi there,
>>>
>>> I was installing Protocol Buffers and it looks like during my "make
>>> check", the protobuf-test failed.
>>>
>>> /Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
>>> FAIL: protobuf-test
>>> PASS: protobuf-lazy-descriptor-test
>>> PASS: protobuf-lite-test
>>> PASS: google/protobuf/compiler/zip_output_unittest.sh
>>> PASS: google/protobuf/io/gzip_stream_unittest.sh
>>> PASS: protobuf-lite-arena-test
>>> PASS: no-warning-test
>>> /Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
>>> make[6]: Nothing to be done for `all-am'.
>>> 
>>> 
>>> Testsuite summary for Protocol Buffers 3.3.0
>>> 
>>> 
>>> # TOTAL: 7
>>> # PASS:  6
>>> # SKIP:  0
>>> # XFAIL: 0
>>> # FAIL:  1
>>> # XPASS: 0
>>> # ERROR: 0
>>> 
>>> 
>>> See src/test-suite.log
>>> Please report to protobuf@googlegroups.com
>>> 
>>> 
>>> make[4]: *** [test-suite.log] Error 1
>>> make[3]: *** [check-TESTS] Error 2
>>> make[2]: *** [check-am] Error 2
>>> make[1]: *** [check] Error 2
>>> make: *** [check-recursive] Error 1
>>>
>>>
>>>
>>> I took a look at the test-suite.log and there's an info line when the
>>> errors begin to occur:
>>> [libprotobuf INFO google/protobuf/descriptor_unittest.cc:6547] A
>>> timeout in this test probably indicates a real bug.
>>>
>>> Shall I open up an issue on GitHub or is this the correct place to
>>> report this kind of thing?
>>>
>>> Thanks,
>>> Sara
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] A timeout in this test probably indicates a real bug.

['Feng Xiao' via Protocol Buffers]

Can you attach the full log file?

On Wed, Jul 12, 2017 at 1:27 PM, Sara Quigley 
wrote:

> Hi there,
>
> I was installing Protocol Buffers and it looks like during my "make
> check", the protobuf-test failed.
>
> /Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
> FAIL: protobuf-test
> PASS: protobuf-lazy-descriptor-test
> PASS: protobuf-lite-test
> PASS: google/protobuf/compiler/zip_output_unittest.sh
> PASS: google/protobuf/io/gzip_stream_unittest.sh
> PASS: protobuf-lite-arena-test
> PASS: no-warning-test
> /Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
> make[6]: Nothing to be done for `all-am'.
> 
> 
> Testsuite summary for Protocol Buffers 3.3.0
> 
> 
> # TOTAL: 7
> # PASS:  6
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  1
> # XPASS: 0
> # ERROR: 0
> 
> 
> See src/test-suite.log
> Please report to protobuf@googlegroups.com
> 
> 
> make[4]: *** [test-suite.log] Error 1
> make[3]: *** [check-TESTS] Error 2
> make[2]: *** [check-am] Error 2
> make[1]: *** [check] Error 2
> make: *** [check-recursive] Error 1
>
>
>
> I took a look at the test-suite.log and there's an info line when the
> errors begin to occur:
> [libprotobuf INFO google/protobuf/descriptor_unittest.cc:6547] A timeout
> in this test probably indicates a real bug.
>
> Shall I open up an issue on GitHub or is this the correct place to report
> this kind of thing?
>
> Thanks,
> Sara
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Need help with compilation of protobuf with a large enum for java

['Feng Xiao' via Protocol Buffers]

On Tue, Jul 11, 2017 at 5:05 PM, Minimol Thomas <minimoltho...@gmail.com>
wrote:

> Thank you Feng for replying. Are there any plans to back port these
> changes to 2.6.1 ? Could you please point me to a commit id ?
>
There isn't any plan to port things back to 2.6.1 at the moment. The change
was implemented inside Google and is part of this enormous integration
commit:
https://github.com/google/protobuf/commit/cf14183bcd5485b4a71541599ddce0b35eb71352#diff-f6bd6179dc2afd0ccf0d9d113c95f2e1

I believe the change in java_enum.cc is what brings down the code size a
little bit:
https://github.com/google/protobuf/blob/cf14183bcd5485b4a71541599ddce0b35eb71352/src/google/protobuf/compiler/java/java_enum.cc#L95




>
> On Tuesday, July 11, 2017 at 1:58:38 PM UTC-7, Feng Xiao wrote:
>>
>>
>>
>> On Tue, Jul 11, 2017 at 1:06 PM, Minimol Thomas <minimo...@gmail.com>
>> wrote:
>>
>>> Compilation of java code generated using protobuf compiler fails giving
>>> the error, "code too large".
>>> We have an enumeration with 6000 values. And the code generated for this
>>> enum is causing the problem.
>>> I would not be able to make any changes to the enumeration as the
>>> protofile is owned/maintained by another team.
>>> The java code generated using protobuf compiler 3 for same protofile
>>> compiles without any issues.
>>> Do we have any option to continue using protobuf compiler version 2.6.1
>>> and solve "code too large" problem.
>>> Note: I am already using "option java_multiple_files = true;"
>>>
>>
>>>
>> Please suggest
>>>
>> I remember we implemented something in protobuf to reduce the code size
>> for enums and that's probably why you don't get the error with newer
>> version of protocol compiler. If you have to use protobuf 2.6.1, I think
>> you will have to patch up protoc 2.6.1 yourself and adopt the same fix used
>> in latest version of protoc.
>>
>>
>>>
>>> Thanks,
>>> Minimol
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] [Java] Proper way to add an item to a List?

['Feng Xiao' via Protocol Buffers]

On Tue, Jul 11, 2017 at 4:53 PM, Laird Nelson  wrote:

> I hope this is the right group.
>
> I've generated a Java API for Helm's Tiller using gRPC and protocol
> buffers.  You can see the javadocs here: https://microbean.
> github.io/microbean-helm/apidocs/index.html  Look at the hapi.* packages.
>
> As I'm working with the generated API, I need to add one kind of thing to
> another thing's list of such things.
>
> Specifically (and arbitrarily) I need to add a (sub-) Chart.Builder to a
> (top-level) Chart.Builder's "dependencies", only if it isn't already
> present.
>
> I can do this via the following generated methods:
> https://microbean.github.io/microbean-helm/apidocs/hapi/
> chart/ChartOuterClass.Chart.Builder.html#addDependencies-
> hapi.chart.ChartOuterClass.Chart.Builder- (Adds a Chart.Builder)
> https://microbean.github.io/microbean-helm/apidocs/hapi/
> chart/ChartOuterClass.Chart.Builder.html#addDependenciesBuilder-- (I
> don't know what this does.)
>
> It seems from looking at the generated source code that if I call build()
> on the "top level" Chart.Builder, it will cascade to the Chart.Builders
> present in the return value of getDependenciesBuilderList() and call
> build() on them too.
>
Actually you can't add a sub-builder to another builder. If you look at the
implementation of addDependencies(Builder), it just calls builder.build()
right away and add the resulted Chart object:
https://microbean.github.io/microbean-helm/apidocs/src-html/hapi/chart/ChartOuterClass.Chart.Builder.html#line.1680

You can instead get a sub-builder out from an existing builder, by calling
the getFieldBuilder() method, e.g., getDependenciesBuilder(int). This
sub-builder you get is "owned" by the parent builder and they are
cross-referenced: if you call build() on the parent builder, it will
recursively call build() on all its sub-builders; if you change the
sub-builder, its parent builder will get notified for the change.


>
> I also need to add a Chart.Builder only if it isn't present already.
>
> When I tried to implement this logic, I found that no Chart.Builder added
> is equal to any other Chart.Builder.  That is, there doesn't seem to be the
> ability to check for a Chart.Builder "in" another Chart.Builder's list of
> dependencies.
>
I think you can have some kind of id field in your Chart message and use
that to check whether a Chart is already added in dependencies?


> Finally at the end of all this, I call build() on the top-level
> Chart.Builder and expect it to produce a Chart with some subcharts in its
> dependencies.  Should I in fact expect this?
>
> Now I feel like I have the semantics about these generated methods all
> wrong.  Is there a better walkthrough than https://developers.
> google.com/protocol-buffers/docs/javatutorial?  Is there a standard way
> for assembling an object graph out of connected builders?
>
> Thanks,
> Best,
> Laird
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Need help with compilation of protobuf with a large enum for java

['Feng Xiao' via Protocol Buffers]

On Tue, Jul 11, 2017 at 1:06 PM, Minimol Thomas 
wrote:

> Compilation of java code generated using protobuf compiler fails giving
> the error, "code too large".
> We have an enumeration with 6000 values. And the code generated for this
> enum is causing the problem.
> I would not be able to make any changes to the enumeration as the
> protofile is owned/maintained by another team.
> The java code generated using protobuf compiler 3 for same protofile
> compiles without any issues.
> Do we have any option to continue using protobuf compiler version 2.6.1
> and solve "code too large" problem.
> Note: I am already using "option java_multiple_files = true;"
>

>
Please suggest
>
I remember we implemented something in protobuf to reduce the code size for
enums and that's probably why you don't get the error with newer version of
protocol compiler. If you have to use protobuf 2.6.1, I think you will have
to patch up protoc 2.6.1 yourself and adopt the same fix used in latest
version of protoc.


>
> Thanks,
> Minimol
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: MessageToJsonString output empty

['Feng Xiao' via Protocol Buffers]

On Sun, Jul 9, 2017 at 4:52 AM, hce h  wrote:

> Let me add more details, here is what I have a test program:
>
> test.proto:
>
> syntax = "proto3";
> package test;
>
> message TestMessage {
> string query = 1;
> int32 page_number = 2;  // Which page number do we want?
> int32 result_per_page = 3;  // Number of results to return per page.
> }
>
> test::TestMessage message;
>
If you don't set any value in the message, the JSON output is expected to
be empty.

If you want to print out values even not set, try the
always_print_primitive_fields flag:
https://github.com/google/protobuf/blob/master/src/google/protobuf/util/json_util.h#L63


> std::string output;
> google::protobuf::util::MessageToJsonString(message, );
> printf("Convert: %d %s\n", output.length(), output.c_str());
>
> Convert: 2 {}
>
>
>
>
>
>
> On Sunday, July 9, 2017 at 1:01:39 PM UTC+10, hce h wrote:
>>
>> Hi,
>>
>> I tried to call MessageToJsonString on Protobuf 3.3.0, but the output string 
>> was empty, output = "{}". Has anyone made it work? If so, could you please 
>> show me an example?
>>
>> Thank you.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

['Feng Xiao' via Protocol Buffers]

On Fri, Jul 7, 2017 at 1:28 PM, Brad Larson <bklar...@gmail.com> wrote:

>
>
> On Tuesday, July 19, 2016 at 10:15:29 AM UTC-5, Igor Gatis wrote:
>>
>> I'm using -DGOOGLE_PROTOBUF_NO_THREAD_SAFETY but I'm hitting the
>> following errors:
>>
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 43: Error:  #20:
>> identifier "AtomicWord" is undefined
>> AtomicWord GetNext() {
>> ^
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 47: Error:  #20:
>> identifier "AtomicWord" is undefined
>> AtomicWord word_;
>> ^
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 44: Error:  #20:
>> identifier "NoBarrier_AtomicIncrement" is undefined
>>   return NoBarrier_AtomicIncrement(_, 1) - 1;
>>
>>
>> Any chance there is a GOOGLE_PROTOBUF_NO_THREAD_SAFETY check missing?
>>
>
> I see these errors as well.
>
>
>
>>
>>
>> On Monday, October 21, 2013 at 2:16:20 PM UTC-3, Safi Ali wrote:
>>>
>>> Hi,
>>>
>>> Thanks a lot Feng for your quick answer. Well at the moment, we are only
>>> planning to use it in a single threaded application where we write/read
>>> messages sequentially, instead of in parallel. So I guess we are safe for
>>> now.
>>>
>>> Regards,
>>> Safi
>>>
>>> On Monday, October 21, 2013 7:32:19 PM UTC+3, Feng Xiao wrote:
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Oct 21, 2013 at 1:05 AM, Safi Ali <safi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have been trying to compile google protocol buffers 2.5.0 on solaris
>>>>> (sparc) environment. It seems I have to use the 
>>>>> GOOGLE_PROTOBUF_NO_THREAD_SAFETY
>>>>> macro in order to make it compile properly. So I follow these steps to
>>>>> compile protobuf:-
>>>>>
>>>>> ./configure CPPFLAGS="-DGOOGLE_PROTOBUF_NO_THREAD_SAFETY"
>>>>> make
>>>>> make check
>>>>>
>>>>> In 'make check', all tests pass.
>>>>> Can anyone shed some light on what are the caveats of using the
>>>>> no_thread_safety macro? What, if any, problems can I expect from protobufs
>>>>> with no thread safety. I have some apprehensions about it and it would be
>>>>> great if someone could clarify those for me:-
>>>>>
>>>>> 1. Is the thread safety only an issue during compilation of .proto
>>>>> files to java/c++ source files? or does protobuf also rely on thread 
>>>>> safety
>>>>> during execution of compiled code?
>>>>>
>>>> Protobuf uses mutex/locks at runtime to protect certain data structures
>>>> in multi-threading environment.
>>>>
>>>>
>>>>> 2. If I dont use thread safety, does protobuf gracefully fall back to
>>>>> single threaded model where needed, or still try to use threads but in
>>>>> somewhat "unsafe" fashion which can lead to bugs such as deadlocks if im
>>>>> unlucky?
>>>>>
>>>> Protobuf doesn't create threads, but with no_thread_safety macro, all
>>>> mutex/locks will be turned into nop. That means you can only use protobuf
>>>> in a single threaded binary. If you try to use messages in multiple
>>>> threads, the code may break unexpectedly.
>>>>
>>>
> Can anyone confirm, is it acceptable to read/write different messages in
> different threads?  Or can we only have one thread make any protobuf calls
> at all?
>
If you are using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, you cannot use protobuf
in multiple threads. Even using different message types in different
threads will be problematic because all message types share the same global
DescriptorPool/MessageFactory/etc.

If you are not using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, protobuf supports
the same thread-safety semantics as with standard types. Reading/writing
different messages in different threads are fine. Reading the same message
in multiple threads is also fine. To write a message in multiple threads a
lock must be used.


>
>
>>
>>>>
>>>>> 3. How is the performance affected while using thread unsafe code? if
>>>>> anyone has done some benchmarking, would be good to see the results.
>>>>>
>>>>> Regards,
>>>>> Safi
>>>&g

Re: [protobuf] Important!! please help, protoc command not working on linux x86_64

['Feng Xiao' via Protocol Buffers]

On Thu, Jul 6, 2017 at 6:57 AM,  wrote:

> Hi All,
> I downloaded protoc binary (protoc-3.2.0-linux-x86_64.zip
> )
> for linux from https://github.com/google/protobuf/releases
> My linux configuration is x86_64 GNU/Linux
>
> I placed the precompiled binary in a folder and try to execute the protoc
> command
>
> $:/opt/protocompiler/bin # ./protoc --version
> Floating point Exception
>
>
> I really dont understand why this problem is coming, I have to use this
> command in ant task to compile my proto files at runtime
>
> But it is always giving me the error message "Floating point Exception"
>
> Its blocking our production build, will be great help if someone can give
> some pointers on it
>
Can you try a different version and see if it has the same issue?

Also you can try to build the protoc binary from source.


>
> Thanks in advance
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto compiler creating Strong Type Enum

['Feng Xiao' via Protocol Buffers]

On Wed, Jul 5, 2017 at 9:07 PM, Abhi  wrote:

> Hi
>
> I am using proto compiler 3.1 and it's a C++ project.
>
> When I compile my proto Its generating myproto.struct.h file along with
> myproto.pb.cc and myproto.pb.h.
>
I don't think protoc generates .struct.h file. Is it coming from some
plugin you are using?


>
> So the issue is in my myproto.struct.h the enum is created as strong typed
> enum and when I compile this file with VS2010 I am getting
> compiler error as its not supported in VS2010.
>
> Could anyone please let me know how to solve this problem.
>
> Please do the needful.
>
>
>
> enum xyz {}
>
> enum class xyz{}  -->Strong typed enum
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf Json conversion

['Feng Xiao' via Protocol Buffers]

You can keep using proto2 (i.e., adding 'syntax = "proto2";' to your
existing files). There is a catch though: the JSON support is designed for
proto3 and won't work well (or won't work at all) for proto2 only features.
If feasible, it might be better to migrate to proto3 to avoid any confusion
when using JSON conversion.

On Sat, Jul 1, 2017 at 6:36 PM hce h  wrote:

> Hi,
>
> I m running system installed libprotobuf version 2.6 on Debian Jessie, it
> does not support json conversion. I need json conversion and I have build
> latest protobuf version 3 from source which supporting json conversion, but
> all our communications are using prottobuf version 2, do I need to add
> syntax = "proto2"; to all my legacy messages or not? I think the default is
> for proto2 I don't need to do it, but please correct me if I am wrong.
>
> Also, I think that json conversion is avilable for running on version 2 as
> well, correct?
>
> Thank you.
>
> - j
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] why not 150

['Feng Xiao' via Protocol Buffers]

"96 01" is written in hex. 0x96 = 150.

On Sun, Jul 2, 2017 at 6:18 AM Jing Lupeng  wrote:

>
> https://developers.google.com/protocol-buffers/docs/encoding#structure
>
>
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Unable to compile protobuf files under Ubuntu 16.04

['Feng Xiao' via Protocol Buffers]

It seems com.github.os72.protocjar only supports 64bit system.

On Thu, Jun 29, 2017 at 5:10 AM,  wrote:

> Hi guys,
>
> Unfortunate, I have met the following problem  - I cannot compile protos
> in my scala project.
> I use "Protocol Buffer Compiler for Scala - Scala PB". All work correctly
> under MacOS, but in Ubuntu 16.04 - Xenial Xerus (
> http://releases.ubuntu.com/16.04/ubuntu-16.04.2-desktop-amd64.iso) it
> doesn't work and returns the next stack trace:
>
> protoc-jar: protoc version: 310, detected platform: linux/i386
> java.lang.RuntimeException: error occurred while compiling protobuf files:
> Unsupported platform: linux/i386
> at sbtprotoc.ProtocPlugin$.executeProtoc(ProtocPlugin.scala:91)
> at sbtprotoc.ProtocPlugin$.sbtprotoc$ProtocPlugin$$
> compile(ProtocPlugin.scala:113)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$
> $anonfun$4.apply(ProtocPlugin.scala:142)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$
> $anonfun$4.apply(ProtocPlugin.scala:141)
> at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)
> at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$
> anonfun$apply$4.apply(Tracked.scala:267)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$
> anonfun$apply$4.apply(Tracked.scala:263)
> at sbt.Difference.apply(Tracked.scala:224)
> at sbt.Difference.apply(Tracked.scala:206)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.
> apply(Tracked.scala:263)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.
> apply(Tracked.scala:262)
> at sbt.Difference.apply(Tracked.scala:224)
> at sbt.Difference.apply(Tracked.scala:200)
> at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:262)
> at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:260)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1.
> apply(ProtocPlugin.scala:152)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1.
> apply(ProtocPlugin.scala:137)
> at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
> at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
> at sbt.std.Transform$$anon$4.work(System.scala:63)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
> at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
> at sbt.Execute.work(Execute.scala:237)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
> at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(
> ConcurrentRestrictions.scala:159)
> at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Unsupported platform: linux/i386
> at com.github.os72.protocjar.Protoc.extractProtoc(Protoc.java:151)
> at com.github.os72.protocjar.Protoc.extractProtoc(Protoc.java:169)
> at com.github.os72.protocjar.Protoc.runProtoc(Protoc.java:55)
> at sbtprotoc.ProtocPlugin$$anonfun$protobufGlobalSettings$8$$
> anonfun$apply$3.apply(ProtocPlugin.scala:54)
> at sbtprotoc.ProtocPlugin$$anonfun$protobufGlobalSettings$8$$
> anonfun$apply$3.apply(ProtocPlugin.scala:53)
> at protocbridge.ProtocBridge$.runWithGenerators(ProtocBridge.scala:79)
> at protocbridge.ProtocBridge$.run(ProtocBridge.scala:61)
> at sbtprotoc.ProtocPlugin$.executeProtoc(ProtocPlugin.scala:87)
> at sbtprotoc.ProtocPlugin$.sbtprotoc$ProtocPlugin$$
> compile(ProtocPlugin.scala:113)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$
> $anonfun$4.apply(ProtocPlugin.scala:142)
> at sbtprotoc.ProtocPlugin$$anonfun$sourceGeneratorTask$1$
> $anonfun$4.apply(ProtocPlugin.scala:141)
> at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)
> at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:253)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$
> anonfun$apply$4.apply(Tracked.scala:267)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$
> anonfun$apply$4.apply(Tracked.scala:263)
> at sbt.Difference.apply(Tracked.scala:224)
> at sbt.Difference.apply(Tracked.scala:206)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.
> apply(Tracked.scala:263)
> at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.
> apply(Tracked.scala:262)
> at sbt.Difference.apply(Tracked.scala:224)
> at sbt.Difference.apply(Tracked.scala:200)
> at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:262)
> at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:260)
> at 

Re: [protobuf] [golang] API provides no way to access unknown extensions

['Feng Xiao' via Protocol Buffers]

I'm not sure if the go team monitors this mailing group. You might get a
better chance to reach them by posting an issue to the golang/protobuf
github repo.

On Mon, Jun 26, 2017 at 7:49 PM, Josh Humphries 
wrote:

> For messages generates from syntax="proto2" files, there is a way to
> access unrecognized fields (as the stream of encoded bytes) via the
> XXX_unrecognized field. But there is no similar way to access unrecognized
> fields that happened to be in an extension range.
>
> If I were to add an issue to the github project to remedy this (and maybe
> even send a pull request), would it get serious consideration? Or is it
> intentional to keep such details completely inaccessible?
>
I know very little about Go, but in other languages like C++/Java, unknown
extensions are put in the same UnknownFieldSet as other fields. It probably
makes more sense to put unknown extensions in XXX_unrecognized?


>
>
> *More background:*
>
> I have a library, protoreflect
> , that supports
> runtime-dynamic messages, using descriptors to define the schemas. One
> thing I want to be able to do is to re-parse the descriptors using a custom
> registry of runtime-defined extensions (for custom options). But I don't
> seem to be able to dig these out of an already-parsed descriptor proto.
>
> *The use case*: I am building a dynamic GRPC stub that works by querying
> the server for its descriptors (using service reflection). I am providing a
> CLI that allows users to pass in JSON that gets parsed into the appropriate
> request message, serialized to the binary format, and sent in a GRPC
> request. Similar happens for de-serializing the responses and then dumping
> the message contents in a human-readable text form.
>
> In order to support custom options, I'd like to be able to efficiently
> extract the unrecognized extensions from descriptors and re-parse them
> using custom logic, that uses extension field descriptors that were also
> downloaded from the same server.
>
> For now, it looks like I need to marshal the options message to bytes, and
> then I could unmarshal it into a dynamic message (I already have support
> for unmarshalling with a custom registry of extensions). But I was hoping
> for something more efficient -- like being able to only do this for the
> unrecognized extensions. (I can already do this for other unrecognized
> fields thanks to the export XXX_unrecognized field that gets added to
> generated proto2 message structs).
>
> Thoughts?
>
> 
> *Josh Humphries*
> jh...@bluegosling.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is the extension registry address still monitored?

['Feng Xiao' via Protocol Buffers]

On Fri, Jun 16, 2017 at 11:42 AM, Marc Gravell 
wrote:

> In descriptor.proto, it asks public authors to request extension numbers
> (for the DSL) via protobuf-global-extension-regis...@google.com

It is monitored, but these emails (one request in about one to two months)
can easily be missed as they mix with all other emails from protobuf group
and github updates...


>
>
> I've done this, but without response. Is this address being monitored?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Timestamp

['Feng Xiao' via Protocol Buffers]

On Wed, May 17, 2017 at 8:10 AM, Leonardo Toledo 
wrote:

> Hi! I am using protobufs but i need to use the version 2 (syntax 2) of
> protobufs and at the same time i need to use the a timestamp, but the
> examples that i saw they use always the syntax 3 of protobufs. I already
> checked the protobufs documentation and i read that in syntax 2 also exist
> a timestamp, so my question is, how can i use your timestamp with syntax 2
> of protobufs?
>
> This is my code:
> import https://github.com/golang/protobuf/ptypes/timestamp/timestamp.proto
> (BUT THIS DOES EXIST A LINK TO THIS ALREADY)
> import https://github.com/google/protobuf/blob/master/
> src/google/protobuf/timestamp.proto (BUT THIS IS USING PROTO 3)
>
> repeated google.protobuf.Timestamp timestamp = 1;
>
> This does works, can anyone help me please and tell me how can i use the
> timestamp in protoc 2 ?
>
timestamp.proto uses proto3 syntax, but you can use it in a proto2 syntax
file.


>
> Thanks in advance! I am waiting for on response from you!
>
> Best regards,
> Leonardo Toledo
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Porting protobuf to iRTOS

['Feng Xiao' via Protocol Buffers]

On Tue, May 16, 2017 at 11:26 PM, Shiva Kumar 
wrote:

> I have even tried using static member class. But in protobuf, namespace
> are nested and distributed across files. Mapping them to class will not
> work out. Removing the all the namespace is another consideration. Please
> suggest is there a simple way to do this or if a complete C version is also
> will be very helpful.
>

You can find some third-party C implementations here:
https://github.com/google/protobuf/blob/master/docs/third_party.md


>
> -Shiva
>
>
> On Wednesday, May 17, 2017 at 10:49:37 AM UTC+5:30, Shiva Kumar wrote:
>>
>> Hi All,
>>
>> I am porting protobuf stack to iRTOS. My C++ compiler doesn't support
>> "namespace" (its kind of compliant to Embedded C++). I am finding it very
>> difficult to port it. I there a complete C alternative or any other easy
>> way to port it.
>>
>> -Regards,
>> Shiva
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Crosscompiling for AARCH64

['Feng Xiao' via Protocol Buffers]

On Wed, May 10, 2017 at 1:21 AM, jebina elsie  wrote:

> Hi All,
>
> I'm trying to cross compile the Protobuf 3.3.0 to the 64 bit ARM processor.
>
> with the configure command
> ./configure CC="aarch64-oe-linux-gcc" CXX="aarch64-oe-linux-g++"
> CFLAGS="--sysroot=/opt/oecore-x86_64/sysroots/aarch64-oe-linux -O2
> -fexpensive-optimizations -frename-registers -fomit-frame-pointer
> -ftree-vectorize -Wno-error=maybe-uninitialized -finline-functions
> -finline-limit=64" CXXFLAGS=" 
> --sysroot=/opt/oecore-x86_64/sysroots/aarch64-oe-linux
> -O2 -fexpensive-optimizations -frename-registers -fomit-frame-pointer
> -ftree-vectorize -Wno-error=maybe-uninitialized -finline-functions
> -finline-limit=64" --host=x86_64 --with-protoc=/home/ubuntu/
> proto3/bin/protoc
>
> when i run make: i get these errors as below.
>
> Any suggestions / fix?
>
I think Gerben is right. Somehow Atomic32 and Atomic64 are defined to the
same type in your environment. Can you check the value of sizeof(Atomic32)
and sizeof(Atomic64)?


>
> In file included from ./google/protobuf/stubs/atomicops.h:208:0,
>  from google/protobuf/stubs/atomicops_internals_x86_gcc.
> cc:36:
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h: In function
> 'google::protobuf::internal::Atomic64 google::protobuf::internal::
> NoBarrier_CompareAndSwap(volatile Atomic64*, 
> google::protobuf::internal::Atomic64,
> google::protobuf::internal::Atomic64)':
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:188:17: error:
> redefinition of 'google::protobuf::internal::Atomic64
> google::protobuf::internal::NoBarrier_CompareAndSwap(volatile Atomic64*,
> google::protobuf::internal::Atomic64, google::protobuf::internal::
> Atomic64)'
>  inline Atomic64 NoBarrier_CompareAndSwap(volatile Atomic64* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:52:17: note:
> 'google::protobuf::internal::Atomic32 google::protobuf::internal::
> NoBarrier_CompareAndSwap(volatile Atomic32*, 
> google::protobuf::internal::Atomic32,
> google::protobuf::internal::Atomic32)' previously defined here
>  inline Atomic32 NoBarrier_CompareAndSwap(volatile Atomic32* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h: In function
> 'google::protobuf::internal::Atomic64 google::protobuf::internal::
> NoBarrier_AtomicExchange(volatile Atomic64*, google::protobuf::internal::
> Atomic64)':
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:213:17: error:
> redefinition of 'google::protobuf::internal::Atomic64
> google::protobuf::internal::NoBarrier_AtomicExchange(volatile Atomic64*,
> google::protobuf::internal::Atomic64)'
>  inline Atomic64 NoBarrier_AtomicExchange(volatile Atomic64* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:77:17: note:
> 'google::protobuf::internal::Atomic32 google::protobuf::internal::
> NoBarrier_AtomicExchange(volatile Atomic32*, 
> google::protobuf::internal::Atomic32)'
> previously defined here
>  inline Atomic32 NoBarrier_AtomicExchange(volatile Atomic32* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h: In function
> 'google::protobuf::internal::Atomic64 google::protobuf::internal::
> NoBarrier_AtomicIncrement(volatile Atomic64*, google::protobuf::internal::
> Atomic64)':
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:233:17: error:
> redefinition of 'google::protobuf::internal::Atomic64
> google::protobuf::internal::NoBarrier_AtomicIncrement(volatile Atomic64*,
> google::protobuf::internal::Atomic64)'
>  inline Atomic64 NoBarrier_AtomicIncrement(volatile Atomic64* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:97:17: note:
> 'google::protobuf::internal::Atomic32 google::protobuf::internal::
> NoBarrier_AtomicIncrement(volatile Atomic32*, 
> google::protobuf::internal::Atomic32)'
> previously defined here
>  inline Atomic32 NoBarrier_AtomicIncrement(volatile Atomic32* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h: In function
> 'google::protobuf::internal::Atomic64 google::protobuf::internal::
> Barrier_AtomicIncrement(volatile Atomic64*, google::protobuf::internal::
> Atomic64)':
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:254:17: error:
> redefinition of 'google::protobuf::internal::Atomic64
> google::protobuf::internal::Barrier_AtomicIncrement(volatile Atomic64*,
> google::protobuf::internal::Atomic64)'
>  inline Atomic64 Barrier_AtomicIncrement(volatile Atomic64* ptr,
>  ^
> ./google/protobuf/stubs/atomicops_internals_arm64_gcc.h:118:17: note:
> 'google::protobuf::internal::Atomic32 google::protobuf::internal::
> Barrier_AtomicIncrement(volatile Atomic32*, 
> google::protobuf::internal::Atomic32)'
> previously defined here
>  inline Atomic32 Barrier_AtomicIncrement(volatile Atomic32* ptr,
>  ^
> 

Re: [protobuf] zigzag encode and zigzag decode invoking undefined behavior

['Feng Xiao' via Protocol Buffers]

Right now protobuf implementation assumes 2s complement and won't work on
any other environments. That's undefined behavior according to C++ standard
but in practice it's very unlikely to be a problem.

On Tue, Apr 25, 2017 at 5:42 AM, Wolfgang Brehm 
wrote:

> right shifting negative integers is tecnically undefined, this means that
> the implementation used for encoding integers:
> (n<<1)^(n>>(digits-1))
> is tecnically undefined according to:
>
>> The value of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits
>> are zero-filled. If E1 has an unsigned type, the value of the result is E1
>> × 2 E2, reduced modulo one more than the maximum value representable in the
>> result type. Otherwise, if E1 has a signed type and non-negative value, and
>> E1 × 2 E2 is representable in the corresponding unsigned type of the result
>> type, then that value, converted to the result type, is the resulting
>> value; otherwise, the behavior is undefined.
>
>
> and the implementation for decoding:
> (n>>1)^(-(n&1))
> depends on the 2s complement as well and is probably tecnically undefined
> (but I cant find a quote from the standard).
>
> This is why I propose the following solution:
>
> template::type>>
> U constexpr zigzag_encode(const S ){
>   return (U(n)<<1)^(n<0?~U(0):U(0));
> }
>
> template::type>>
> S constexpr zigzag_decode(const U ){
>   return (n&1)?-1-S(n>>1):(n>>1);
> }
>
> This does not look as cool, but modern compilers (llvm and gcc were
> tested) will compile to bascially the same instructions, gcc will introduce
> 1 additional instruction for decode.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] binary compatibility between versions 2.x and 3.x

['Feng Xiao' via Protocol Buffers]

On Thu, Apr 20, 2017 at 6:01 PM, Scott Sue <scott@celer-tech.com> wrote:

> Hi Feng,
>
> Apologies, what I was meaning was that with the introduction of protobuf
> 3.2.0 now, is 3.2.0 binary compatible with protobuf 2.5.0?  Or does you
> original message reply still hold true?
>
No, 3.2.0 runtime is not binary compatible with 2.5.0 generated code. It is
only binary compatible with older 3.x version generated code, such as 3.0,
or 3.1.


>
>
> Regards,
> Scott
>
> SCOTT SUE
> CHIEF TECHNOLOGY OFFICER
>
> Support Line : +44(0) 2031 371 603 <+44%2020%203137%201603>
> Mobile : +852 9611 3969 <+852%209611%203969>
>
> 9/F, 33 Lockhart Road, Wanchai, Hong Kong
> www.celer-tech.com
>
>
>
>
>
>
>
> On 21 Apr 2017, at 2:08 AM, Feng Xiao <xiaof...@google.com> wrote:
>
>
>
> On Thu, Apr 20, 2017 at 3:55 AM, Scott Sue <scott@celer-tech.com>
> wrote:
>
>> Hi Feng,
>>
>> Is this still the case considering that 3.2.0 is out that allows the
>> version number to be used by protoc?
>>
> Not sure what you are asking, but 3.0.0 generated code should work with
> 3.2.0 runtime and 3.2.0 protoc will work with any old version plugins.
>
>
>>
>> General
>> * Added protoc version number to protoc plugin protocol. It can be used by
>> protoc plugin to detect which version of protoc is used with the plugin
>> and
>> mitigate known problems in certain version of protoc.
>> Regards,
>> Scott
>>
>>
>> On Saturday, August 13, 2016 at 8:00:23 AM UTC+8, Feng Xiao wrote:
>>>
>>> On Thu, Aug 11, 2016 at 12:21 PM, Jeff Olson <jeff.d...@gmail.com>
>>> wrote:
>>>
>>>> I've read through the release documents
>>>> <https://github.com/google/protobuf/releases/tag/v3.0.0> for Protocol
>>>> Buffers v3.0.0 but I can't find any statements regrading binary
>>>> compatibility with version 2.6.1 (or earlier). There are a couple of things
>>>> I'd like to know:
>>>>
>>>>
>>>>1. If I dump a 3.0.0 version of the 
>>>> com.google.protobuf/protobuf-java.jar
>>>>in a project the depends on java classes generated by protoc v2.6.1 (or
>>>>v2.5.0) will everything still work? That is, is the java runtime 
>>>> backwards
>>>>compatible?
>>>>
>>>> Unfortunately no. The 3.0.0 version protobuf-java.jar will not work
>>> with java classes generated by protoc v2.6.1 (or v2.5.0). For those who can
>>> not upgrade to 3.0 directly, we are considering releasing a 2.7 version
>>> which provides the following compatibility guarantees:
>>> 1. 2.7 protobuf-java.jar will be compatible with java classes generated
>>> by protoc 2.6.1 and protoc 2.5.0.
>>> 2. 3.0 protobuf-java.jar will be compatible with java classes generated
>>> by protoc 2.7
>>>
>>> I.e., it allows you to upgrade to 3.0 in two steps: upgrade to 2.7, and
>>> after all your dependencies are upgraded to either 2.7 or 3.0, you can
>>> upgrade to 3.0 as well. It's not an ideal solution, but we hope it can help
>>> ease the migration pain for some users. I'm testing this 2.7 version in a
>>> github branch right now: https://github.com/google/protobuf/tree/2.7.0.
>>> If you are interested, feel free to test it and any feedback is welcome.
>>>
>>>
>>>>
>>>>1. Is the binary wire format the same for purely proto2 messages?
>>>>If I serialize messages generated from proto2 sources using the 3.0.0 
>>>> java
>>>>runtime will clients using protobuf v2.6.1 be able to decode the 
>>>> messages
>>>>correctly?
>>>>
>>>> Yes. The wire format stays the same as long as you are using the proto2
>>> syntax version.
>>>
>>> Thanks,
>>>> Jeff
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+u...@googlegroups.com.
>>>> To post to this group, send email to prot...@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>> *This message, including any attachments, may include private, privileged
>> and confidential informat

Re: [protobuf] binary compatibility between versions 2.x and 3.x

['Feng Xiao' via Protocol Buffers]

On Thu, Apr 20, 2017 at 3:55 AM, Scott Sue <scott@celer-tech.com> wrote:

> Hi Feng,
>
> Is this still the case considering that 3.2.0 is out that allows the
> version number to be used by protoc?
>
Not sure what you are asking, but 3.0.0 generated code should work with
3.2.0 runtime and 3.2.0 protoc will work with any old version plugins.


>
> General
> * Added protoc version number to protoc plugin protocol. It can be used by
> protoc plugin to detect which version of protoc is used with the plugin and
> mitigate known problems in certain version of protoc.
> Regards,
> Scott
>
>
> On Saturday, August 13, 2016 at 8:00:23 AM UTC+8, Feng Xiao wrote:
>>
>> On Thu, Aug 11, 2016 at 12:21 PM, Jeff Olson <jeff.d...@gmail.com> wrote:
>>
>>> I've read through the release documents
>>> <https://github.com/google/protobuf/releases/tag/v3.0.0> for Protocol
>>> Buffers v3.0.0 but I can't find any statements regrading binary
>>> compatibility with version 2.6.1 (or earlier). There are a couple of things
>>> I'd like to know:
>>>
>>>
>>>1. If I dump a 3.0.0 version of the com.google.protobuf/protobuf-java.jar
>>>in a project the depends on java classes generated by protoc v2.6.1 (or
>>>v2.5.0) will everything still work? That is, is the java runtime 
>>> backwards
>>>compatible?
>>>
>>> Unfortunately no. The 3.0.0 version protobuf-java.jar will not work with
>> java classes generated by protoc v2.6.1 (or v2.5.0). For those who can not
>> upgrade to 3.0 directly, we are considering releasing a 2.7 version which
>> provides the following compatibility guarantees:
>> 1. 2.7 protobuf-java.jar will be compatible with java classes generated
>> by protoc 2.6.1 and protoc 2.5.0.
>> 2. 3.0 protobuf-java.jar will be compatible with java classes generated
>> by protoc 2.7
>>
>> I.e., it allows you to upgrade to 3.0 in two steps: upgrade to 2.7, and
>> after all your dependencies are upgraded to either 2.7 or 3.0, you can
>> upgrade to 3.0 as well. It's not an ideal solution, but we hope it can help
>> ease the migration pain for some users. I'm testing this 2.7 version in a
>> github branch right now: https://github.com/google/protobuf/tree/2.7.0.
>> If you are interested, feel free to test it and any feedback is welcome.
>>
>>
>>>
>>>1. Is the binary wire format the same for purely proto2 messages? If
>>>I serialize messages generated from proto2 sources using the 3.0.0 java
>>>runtime will clients using protobuf v2.6.1 be able to decode the messages
>>>correctly?
>>>
>>> Yes. The wire format stays the same as long as you are using the proto2
>> syntax version.
>>
>> Thanks,
>>> Jeff
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> *This message, including any attachments, may include private, privileged
> and confidential information and is intended only for the personal and
> confidential use of the intended recipient(s). If the reader of this
> message is not an intended recipient, you are hereby notified that any
> review, use, dissemination, distribution, printing or copying of this
> message or its contents is strictly prohibited and may be unlawful. If you
> are not an intended recipient or have received this communication in error,
> please immediately notify the sender by telephone and/or a reply email and
> permanently delete the original message, including any attachments, without
> making a copy.*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Type URLs and Any types

['Feng Xiao' via Protocol Buffers]

On Wed, Apr 19, 2017 at 5:25 PM, Josh Humphries 
wrote:

> The protobuf docs for the Any type
> 
> talk about the URL being a real URL, where an HTTP GET request will reply
> with encoded google.protobuf.Type message. The doc further states that
> URLs with no scheme assume HTTPS (and runtime support that I've seen looks
> like it intentionally leaves out the scheme when generating type URLs).
>
> However, the given URL examples for all well-known types don't work. Any
> GET request for https://type.googleapis.com/ fails with a 404.
> The actual example in the Any doc is for google.protobuf.Duration, and
> https://type.googleapis.com/google.protobuf.Duration simply does not work.
>
> I don't necessarily need these to work, but I have written stuff for
> resolving types per the doc and have no out-of-the-box URLs I can play with
> to test it.. It seems like an issue that none of the documented examples
> actually work.
>
Can you create an github issue and give more context about how you plan to
use it? Right now the type service doesn't exist because we find no use
case of it for our API service stack.


>
> 
> *Josh Humphries*
> jh...@bluegosling.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Parse/Decode raw protobuf msg

['Feng Xiao' via Protocol Buffers]

On Wed, Apr 19, 2017 at 1:26 PM, Rohan Suri <rohan.a.s...@gmail.com> wrote:

> You create a descriptor by using --descriptor_set_out right?
>
That's right. You can make that part of your build process and embed the
generated data file into your binary however you want.


> Followed by creating FileDescriptorProto by using
> parseFrom(FileInputStream)?
>

> And moving forward as stated here https://groups.google.com/d/
> msg/protobuf/bWlJaaMmx4Q/kIYJ1-5YRb0J
>
>
> On Wednesday, April 19, 2017 at 11:43:42 PM UTC+5:30, Feng Xiao wrote:
>>
>> CodedInputStream is the right one to use for 1.
>>
>> Invoking protoc compiler in production sounds problematic to me, though
>> I'm not aware of any document listing it explicitly as bad practice. I
>> would suggest avoid that and use other alternatives. For example, in Google
>> we build a protodb which contains FileDescriptorProto of all transitively
>> depended .proto files and use that instead of actual .proto files.
>>
>> On Wed, Apr 19, 2017 at 10:39 AM Rohan Suri <rohan@gmail.com> wrote:
>>
>>> Hey Feng, thanks for your input.
>>>
>>> Any suggestions for 1) ? Other than using CodingInputStream?
>>>
>>> Also is it bad practise to invoke protoc compiler in a production
>>> environment?
>>>
>>> On Wednesday, April 19, 2017 at 11:00:17 PM UTC+5:30, Feng Xiao wrote:
>>>>
>>>>
>>>>
>>>> On Wed, Apr 19, 2017 at 4:20 AM, Rohan Suri <rohan@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello everyone,
>>>>>
>>>>> I'm trying to find the right Java APIs to do what protoc --decode_raw
>>>>> and protoc --decode do.
>>>>> Essentially handling two cases:
>>>>> 1) When I don't have the proto, and only the raw protobuf msg.
>>>>> eg protoc --decode_raw < rawmsg.bin
>>>>>
>>>>> 1: "rohan"
>>>>> 2 {
>>>>> 1: "13"
>>>>> 2: "08"
>>>>> 3: "94"
>>>>> }
>>>>>
>>>>> 2) When I have the .proto(not the generated classes, neither the
>>>>> parsed .proto)
>>>>> eg protoc --decode some.pkg.name.Person $proto_file_path < rawmsg.bin
>>>>>
>>>>> name: "rohan"
>>>>> dob {
>>>>>  dd: "13"
>>>>>  mm: "08"
>>>>>  yy: "94"
>>>>> }
>>>>>
>>>>> I did dig google groups.
>>>>>
>>>>> From those the hints I've got is
>>>>> For case 1) Using CodedInputStream write my own parser? parsing bytes
>>>>> by making informed guesses.
>>>>>
>>>>> https://groups.google.com/forum/#!searchin/protobuf/decode$
>>>>> 20without$20proto|sort:relevance/protobuf/7s4YIwrQLgk/r2Yc_36OYGAJ
>>>>> http://stackoverflow.com/questions/19647387/parsing-a-raw-
>>>>> protocol-buffer-byte-stream-in-c-sharp/19652735#19652735
>>>>> http://stackoverflow.com/questions/13937882/parsing-protocol
>>>>> -buffers-without-knowing-the-proto
>>>>>
>>>>> For case 2) well nothing. Everything I got assumed I have
>>>>> FileDescriptorProto for my .proto or instance of my message type.
>>>>>
>>>>
>>>> 2) requires parsing .proto files and that parser is in C++ only.
>>>>
>>>
>>>>> Are there any add-on libraries present to handle these scenarios?
>>>>> Or any logic to go about solving them?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Protocol Buffers" group.
>>>>>
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to protobuf+u...@googlegroups.com.
>>>>> To post to this group, send email to prot...@googlegroups.com.
>>>>
>>>>
>>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Parse/Decode raw protobuf msg

['Feng Xiao' via Protocol Buffers]

CodedInputStream is the right one to use for 1.

Invoking protoc compiler in production sounds problematic to me, though I'm
not aware of any document listing it explicitly as bad practice. I would
suggest avoid that and use other alternatives. For example, in Google we
build a protodb which contains FileDescriptorProto of all transitively
depended .proto files and use that instead of actual .proto files.

On Wed, Apr 19, 2017 at 10:39 AM Rohan Suri <rohan.a.s...@gmail.com> wrote:

> Hey Feng, thanks for your input.
>
> Any suggestions for 1) ? Other than using CodingInputStream?
>
> Also is it bad practise to invoke protoc compiler in a production
> environment?
>
> On Wednesday, April 19, 2017 at 11:00:17 PM UTC+5:30, Feng Xiao wrote:
>>
>>
>>
>> On Wed, Apr 19, 2017 at 4:20 AM, Rohan Suri <rohan@gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> I'm trying to find the right Java APIs to do what protoc --decode_raw
>>> and protoc --decode do.
>>> Essentially handling two cases:
>>> 1) When I don't have the proto, and only the raw protobuf msg.
>>> eg protoc --decode_raw < rawmsg.bin
>>>
>>> 1: "rohan"
>>> 2 {
>>> 1: "13"
>>> 2: "08"
>>> 3: "94"
>>> }
>>>
>>> 2) When I have the .proto(not the generated classes, neither the parsed
>>> .proto)
>>> eg protoc --decode some.pkg.name.Person $proto_file_path < rawmsg.bin
>>>
>>> name: "rohan"
>>> dob {
>>>  dd: "13"
>>>  mm: "08"
>>>  yy: "94"
>>> }
>>>
>>> I did dig google groups.
>>>
>>> From those the hints I've got is
>>> For case 1) Using CodedInputStream write my own parser? parsing bytes by
>>> making informed guesses.
>>>
>>> https://groups.google.com/forum/#!searchin/protobuf/
>>> decode$20without$20proto|sort:relevance/protobuf/
>>> 7s4YIwrQLgk/r2Yc_36OYGAJ
>>> http://stackoverflow.com/questions/19647387/parsing-a-
>>> raw-protocol-buffer-byte-stream-in-c-sharp/19652735#19652735
>>> http://stackoverflow.com/questions/13937882/parsing-
>>> protocol-buffers-without-knowing-the-proto
>>>
>>> For case 2) well nothing. Everything I got assumed I have
>>> FileDescriptorProto for my .proto or instance of my message type.
>>>
>>
>> 2) requires parsing .proto files and that parser is in C++ only.
>>
>
>>> Are there any add-on libraries present to handle these scenarios?
>>> Or any logic to go about solving them?
>>>
>>> Thanks.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>
>>
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Parse/Decode raw protobuf msg

['Feng Xiao' via Protocol Buffers]

On Wed, Apr 19, 2017 at 4:20 AM, Rohan Suri  wrote:

> Hello everyone,
>
> I'm trying to find the right Java APIs to do what protoc --decode_raw and
> protoc --decode do.
> Essentially handling two cases:
> 1) When I don't have the proto, and only the raw protobuf msg.
> eg protoc --decode_raw < rawmsg.bin
>
> 1: "rohan"
> 2 {
> 1: "13"
> 2: "08"
> 3: "94"
> }
>
> 2) When I have the .proto(not the generated classes, neither the parsed
> .proto)
> eg protoc --decode some.pkg.name.Person $proto_file_path < rawmsg.bin
>
> name: "rohan"
> dob {
>  dd: "13"
>  mm: "08"
>  yy: "94"
> }
>
> I did dig google groups.
>
> From those the hints I've got is
> For case 1) Using CodedInputStream write my own parser? parsing bytes by
> making informed guesses.
>
> https://groups.google.com/forum/#!searchin/protobuf/
> decode$20without$20proto|sort:relevance/protobuf/7s4YIwrQLgk/r2Yc_36OYGAJ
> http://stackoverflow.com/questions/19647387/parsing-a-
> raw-protocol-buffer-byte-stream-in-c-sharp/19652735#19652735
> http://stackoverflow.com/questions/13937882/parsing-
> protocol-buffers-without-knowing-the-proto
>
> For case 2) well nothing. Everything I got assumed I have
> FileDescriptorProto for my .proto or instance of my message type.
>

2) requires parsing .proto files and that parser is in C++ only.

>
> Are there any add-on libraries present to handle these scenarios?
> Or any logic to go about solving them?
>
> Thanks.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Define pubsub like rpc in protobuf file

['Feng Xiao' via Protocol Buffers]

On Tue, Apr 11, 2017 at 3:59 PM, 'Alex Barlow' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Ah yes,
>
> So I've done something similar, however the problem is that the pub sub
> extension would be a library. So using the 'Broadcasts' enum wouldn't scale
> as I couldn't foresee all the events that a particular service might
> implement and I'd want the developer to define their own enum. So I guess
> my question is, how can I take an enum or string constant (which an enum is
> generated to in some libraries).
>
> Essentially, since I can't define RPC methods as they're async and don't
> reference other services, I'm looking for a way to reference other messages
> and constants in my options. Perhaps there's another way of going about it?
>
Maybe not a good idea, but you can get something like the below with
extensions:
syntax = "proto2";

import "google/protobuf/descriptor.proto";

package xiaofeng;

message MyOptions {
  extensions 1000 to max;
}

extend google.protobuf.ServiceOptions {
  optional MyOptions any_option = ;
}

message MyMessage {
  extend MyOptions {
optional MyMessage extended_option = 1000;
  }
  optional int32 value = 1;
}

service MyService {
  option (any_option) = {
[xiaofeng.MyMessage.extended_option] {
  value: 10
}
  };
}

Basically you define your custom option with extensions, and let users
extend your option to add custom data into your option. This only works
with proto2 though as it uses extensions.


> I could just define strings, but then you would have to make sure they're
> the same between protobufs and using a code generator, I would have to find
> that other protobuf and link them together
>
>
> On Tuesday, 11 April 2017 22:29:29 UTC+1, Feng Xiao wrote:
>
>> Not sure if this is what you are asking for, but the following proto
>> compiles for me:
>> syntax = "proto3";
>> import "google/protobuf/descriptor.proto";
>> enum Broadcasts {
>>   HELLO = 0;
>>   CREATED = 1;
>> }
>> enum ReturnTypes {
>>   HELLO_REPLY = 0;
>> }
>> message Publisher {
>>   Broadcasts name = 1;
>>   ReturnTypes returns = 2;
>> }
>> message Subscriber {
>>   Broadcasts name = 1;
>> }
>> message Subscribers {
>>   repeated Publisher publisher = 1;
>>   repeated Subscriber subscriber = 2;
>> }
>>
>> extend google.protobuf.ServiceOptions {
>>   Subscribers subscribers = ;
>> }
>>
>> service Greeter {
>>   option (subscribers) = {
>> publisher {
>>   name: HELLO
>>   returns: HELLO_REPLY
>> }
>> subscriber {
>>   name: HELLO
>> }
>> subscriber {
>>   name: CREATED
>> }
>>   };
>> }
>>
>> On Tue, Apr 11, 2017 at 4:14 AM, 'Alex Barlow' via Protocol Buffers <
>> prot...@googlegroups.com> wrote:
>>
>>> Hi all,
>>> I was hoping for some advice for designing my publish subscribe system
>>> on top of protobuf and rpc.
>>>
>>> The idea is that a developer may have defined set of messages, a normal
>>> service and some rpc methods which they then augment with some publishers
>>> and subscribers.
>>>
>>> For example, given this ordinary service..
>>>
>>> syntax = "proto3";
>>> package helloworld;
>>>
>>> service Greeter {
>>>   rpc SayHello (HelloRequest) returns (HelloReply) {}
>>> }
>>>
>>> message HelloRequest {
>>>   string name = 1;
>>> }
>>>
>>> message HelloReply {
>>>   string message = 1;
>>> }
>>>
>>> I'm thinking of creating my own extensions and options so a developer
>>> can describe which messages it publishes and what messages it subscribes to
>>> but I'm struggling with how to implement it, here's a proposal of a
>>> annotated proto
>>>
>>> syntax = "proto3";
>>> package helloworld;
>>>
>>> enum Broadcasts {
>>>   HELLO = 0;
>>> }
>>>
>>> service Greeter {
>>>   rpc SayHello (HelloRequest) returns (HelloReply) {}
>>>
>>>   option (subscribers) = {
>>> publisher {
>>>   name: HELLO
>>>   returns: HelloReply
>>> }
>>>
>>> subscriber {
>>>   name: HELLO
>>> }
>>>
>>> subscriber {
>>>   name: some_service.CREATED
>>> }
>>>   };
>>>
>>> }
>>>
>>> message HelloRequest {
&g