Re: [protobuf] Proto3 Storing UUID

[Walter Schulze]

I wish there was just a 128 bit type, that could be used for uuid and ipv6, 
but for now we are stuck with the choice between bytes and two fixed64s, 
and the two groups of people that disagree about which is better.

On Saturday, 7 May 2016 02:00:02 UTC+2, Adam Cozzette wrote:
>
> The bytes type sounds like a good choice to me. I could also imagine using 
> a pair of fixed64 fields if you want to avoid having to validate the length 
> of the byte string.
>
> On Thu, Apr 28, 2016 at 10:39 PM, Osman Ali  > wrote:
>
>> Looking use a 128 bit uuid according to rfc4122. For Proto3 should use 
>> bytes type?
>>
>> So:
>>
>> message SendIdentifier {
>>  bytes user_uuid = 1;
>> }
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: protoc-gen-go and internal/grpc

[Walter Schulze]

Yes Please

On Sunday, 1 November 2015 21:43:56 UTC+2, zel...@squareup.com wrote:
>
> Any particular reason the grpc plugin for protoc-gen-go is in an internal/ 
> subdirectory? We import it because we want to include the grpc plugin along 
> with our custom plugins, but with the move to go 1.5, the import will break.
>
> I was considering adding a grpc or include_grpc package under 
> protoc-gen-go/ that could be imported just to link the grpc plugin, but 
> that seems silly: would it be reasonable to elide the /internal/ 
> subdirectory?
>
> Zellyn
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf3 deterministic binary serialization

[Walter Schulze]

Go also sorts the map keys.
I heard it was, because that is what the C++ implementation is doing.

So do all implementations sort their fields by field number?
What languages do "all" include?

On Friday, 23 October 2015 23:26:06 UTC+2, Feng Xiao wrote:
>
> On Wed, Oct 21, 2015 at 10:31 AM, Brandon Philips  > wrote:
>
>> Hello-
>>
>> The Open Container Initiative is investigating use of protobuf for use 
>> for filesystem and container metadata. Part of the goal is to make this 
>> metadata signable. However, we learned that protobuf binary serialization 
>> are not deterministic between implementations: 
>> https://groups.google.com/a/opencontainers.org/d/msg/dev/xo4SQ92aWJ8/ad1-xm9qCAAJ
>>
>> Is there any hope of that being tackled in protobuf3
>>
> Unlikely.
>  
>
>> or there being at least a mode to say "I really want the deterministic 
>> serialization"?
>>
> Maybe. There has been requests about adding such serialization support, 
> though we haven't put much effort in it because it's a low priority 
> comparing to many other stuff we are working on.
>  
>
>>
>> It isn't a huge blocker for us but it does mean the same spec serialized 
>> in a python tool and java tool MAY be different which is a bit annoying.
>>
> The undeterministic comes from unknown fields and a new feature protobuf 
> maps. If you can guarantee there are no such fields in your proto, the 
> protobuf library will always serialize other fields ordered by field number 
> and thus should output the same bytes. This is what most people are relying 
> on when they need to fingerprint or use protobuf serialized data as keys.
>  
>
>>
>> Thank You,
>>
>> Brandon
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Protobuf Buffers v3.0.0-alpha-1

[Walter Schulze]

In other words C# and those new languages won't be able to serialize the
descriptor?

On 9 October 2015 at 19:44, 'Feng Xiao' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> The decision is not to support proto2 in C# (and probably also for all
> other languages that are new in v3.0.0+).
>
> On Fri, Oct 9, 2015 at 10:42 AM, Teddy Zhang  wrote:
>
>> Will the C# implementation support proto2 message as well?
>> What is the compatibility story between proto2 and proto 3? I assume the
>> wire format is compatible as long as no proto 3 exclusive features are used.
>>
>>
>> On Tuesday, August 4, 2015 at 5:43:36 AM UTC-7, Jon Skeet wrote:
>>
>>> That looks like you're expecting a protobuf.net-style approach - to
>>> which the answer is "no" and will continue to be "no".
>>>
>>> The C# support will continue to be based on generated code, but there's
>>> a new code generator and runtime now in the master branch. The main changes
>>> from the previous code are:
>>>
>>> - proto3-only support (no proto2 at all)
>>> - mutable generated types rather than the Java-style builders and
>>> immutable messages
>>>
>>> Jon
>>>
>>> On Monday, 3 August 2015 22:50:06 UTC+1, The Nguyen Xuan wrote:

 Does this version support object type in C# ?

 ex:

 [ProtoMember(1)]
 public object A {get;set;}

 thank.

 Vào 11:51:01 UTC+7 Thứ Năm, ngày 11 tháng 12 năm 2014, Feng Xiao đã
 viết:
>
> Hi all,
>
> I just published protobuf v3.0.0-alpha-1 on our github site:
> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1
>
> This is the first alpha release of protobuf v3.0.0. In protobuf
> v3.0.0, we will add a new protobuf language version (aka proto3) and
> support a wider range of programming languages (to name a few: ruby, php,
> node.js, objective-c). This alpha version contains C++ and Java
> implementation with partial proto3 support (see below for details). In
> future releases we will add support for more programming languages and
> implement the full proto3 feature set. Besides proto3, this alpha version
> also includes two other new features: map fields and arena allocation. 
> They
> are implemented for both proto3 and the old protobuf language version (aka
> proto2).
>
> We are currently working on the documentation of these new features
> and when it's ready it will be updated to our protobuf developer guide
> . For
> the time being if you have any questions regarding proto3 or other new
> features, please post your question in the discussion group.
>
> CHANGS
> ===
> Version 3.0.0-alpha-1 (C++/Java):
>
>   General
>   * Introduced Protocol Buffers language version 3 (aka proto3).
>
> When protobuf was initially opensourced it implemented Protocol
> Buffers
> language version 2 (aka proto2), which is why the version number
> started from v2.0.0. From v3.0.0, a new language version (proto3)
> is
> introduced while the old version (proto2) will continue to be
> supported.
>
> The main intent of introducing proto3 is to clean up protobuf
> before
> pushing the language as the foundation of Google's new API
> platform.
> In proto3, the language is simplified, both for ease of use and  to
> make it available in a wider range of programming languages. At the
> same time a few features are added to better support common idioms
> found in APIs.
>
> The following are the main new features in language version 3:
>
>   1. Removal of field presence logic for primitive value fields,
> removal
>  of required fields, and removal of default values. This makes
> proto3
>  significantly easier to implement with open struct
> representations,
>  as in languages like Android Java, Objective C, or Go.
>   2. Removal of unknown fields.
>   3. Removal of extensions, which are instead replaced by a new
> standard
>  type called Any.
>   4. Fix semantics for unknown enum values.
>   5. Addition of maps.
>   6. Addition of a small set of standard types for representation
> of time,
>  dynamic data, etc.
>   7. A well-defined encoding in JSON as an alternative to binary
> proto
>  encoding.
>
> This release (v3.0.0-alpha-1) includes partial proto3 support for
> C++ and
> Java. Items 6 (well-known types) and 7 (JSON format) in the above
> feature
> list are not implemented.
>
> A new notion "syntax" is introduced to specify whether a .proto
> file
> uses proto2 or proto3:
>
>   // foo.proto
>   syntax = "proto3";
>   message Bar {...}
>
> 

[protobuf] Re: golang protoc compiler - additional golang tags

[Walter Schulze]

gogoprotobuf http://gogo.github.io/doc/ supports adding tags for more 
serialization formats.
See the section on More Serialization Formats.

On Tuesday, 21 July 2015 23:22:00 UTC+2, Ben Davies wrote:
>
> Hi!
>
> Not sure if this is the right forum, if not please let me know and I will 
> direct my question there.
>
> I've been experimenting with protobufs and grpc in Go and love it! 
> however, I have a question: is it possible to add additional golang tags to 
> the output golang code via the proto file? Maybe as a extension? The reason 
> I ask is that I'd like to add some additional tags that will make it easier 
> to export the members as XML (much like the JSON tags do) or to add an 
> additional layer of validation (to ensure that a string is a valid URL or 
> ISO-3166-A2 2 letter country code).
>
> I'd like to do this on the proto file side that way I can continue to 
> generate the go code without having to keep adding back my additional tags.
>
> Any thoughts?
>
> Cheers!
> Ben
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Reading protobuf dynamically in go at run time

[Walter Schulze]

Currently I can read some string defined paths using my fieldpath package
http://godoc.org/github.com/gogo/protobuf/fieldpath#pkg-examples
fieldpath requires a FileDescriptorSet (the parsed proto files).

I am working on something better, but fieldpath might be good enough for 
your use case, depending on what it is.
What do you actually want to do?

On Saturday, 4 April 2015 01:57:20 UTC+2, Huzefa Mehta wrote:
>
> Hello,
>
> I am trying to find out how to read a protobuf in go during run time 
> (rather than in a BUILD). Any examples or pointers would be appreciated.
>
> Thanks, 
> Huzefa
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Protobuf Buffers v3.0.0-alpha-1

[Walter Schulze]

I have declared quite a few file, message, field, etc. extensions on the
descriptor.
https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto
These extensions are used to modify the code that is generated.

This results in a user being able to create a proto file like this
https://github.com/gogo/protobuf/blob/master/test/example/example.proto
There are file, field and message extensions used in the proto, which
protoc can parse.
Can I use this style in proto3 syntax?

I guess I should be able to if the descriptor does not offer me any other
way to do this, like Any?

On 27 January 2015 at 20:33, Feng Xiao  wrote:

>
>
> On Tue Jan 27 2015 at 5:42:34 AM Walter Schulze 
> wrote:
>
>> Will the extensions in the descriptor.proto also be changed to Any types?
>>
> No. That will continue to be supported. descriptor.proto is the only proto
> that will allow extensions in proto3.
>
>
>>
>> On Thursday, 11 December 2014 06:51:01 UTC+2, Feng Xiao wrote:
>>>
>>> Hi all,
>>>
>>> I just published protobuf v3.0.0-alpha-1 on our github site:
>>> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1
>>>
>>> This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0,
>>> we will add a new protobuf language version (aka proto3) and support a
>>> wider range of programming languages (to name a few: ruby, php, node.js,
>>> objective-c). This alpha version contains C++ and Java implementation with
>>> partial proto3 support (see below for details). In future releases we will
>>> add support for more programming languages and implement the full proto3
>>> feature set. Besides proto3, this alpha version also includes two other new
>>> features: map fields and arena allocation. They are implemented for both
>>> proto3 and the old protobuf language version (aka proto2).
>>>
>>> We are currently working on the documentation of these new features and
>>> when it's ready it will be updated to our protobuf developer guide
>>> <https://developers.google.com/protocol-buffers/docs/overview>. For the
>>> time being if you have any questions regarding proto3 or other new
>>> features, please post your question in the discussion group.
>>>
>>> CHANGS
>>> ===
>>> Version 3.0.0-alpha-1 (C++/Java):
>>>
>>>   General
>>>   * Introduced Protocol Buffers language version 3 (aka proto3).
>>>
>>> When protobuf was initially opensourced it implemented Protocol
>>> Buffers
>>> language version 2 (aka proto2), which is why the version number
>>> started from v2.0.0. From v3.0.0, a new language version (proto3) is
>>> introduced while the old version (proto2) will continue to be
>>> supported.
>>>
>>> The main intent of introducing proto3 is to clean up protobuf before
>>> pushing the language as the foundation of Google's new API platform.
>>> In proto3, the language is simplified, both for ease of use and  to
>>> make it available in a wider range of programming languages. At the
>>> same time a few features are added to better support common idioms
>>> found in APIs.
>>>
>>> The following are the main new features in language version 3:
>>>
>>>   1. Removal of field presence logic for primitive value fields,
>>> removal
>>>  of required fields, and removal of default values. This makes
>>> proto3
>>>  significantly easier to implement with open struct
>>> representations,
>>>  as in languages like Android Java, Objective C, or Go.
>>>   2. Removal of unknown fields.
>>>   3. Removal of extensions, which are instead replaced by a new
>>> standard
>>>  type called Any.
>>>   4. Fix semantics for unknown enum values.
>>>   5. Addition of maps.
>>>   6. Addition of a small set of standard types for representation of
>>> time,
>>>  dynamic data, etc.
>>>   7. A well-defined encoding in JSON as an alternative to binary
>>> proto
>>>  encoding.
>>>
>>> This release (v3.0.0-alpha-1) includes partial proto3 support for
>>> C++ and
>>> Java. Items 6 (well-known types) and 7 (JSON format) in the above
>>> feature
>>> list are not implemented.
>>>
>>> A new notion "syntax" is introduced to specify whether a .proto file
>>> uses proto2 or proto3:
&

[protobuf] Re: Protobuf Buffers v3.0.0-alpha-1

[Walter Schulze]

Will the extensions in the descriptor.proto also be changed to Any types?

On Thursday, 11 December 2014 06:51:01 UTC+2, Feng Xiao wrote:
>
> Hi all,
>
> I just published protobuf v3.0.0-alpha-1 on our github site:
> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1
>
> This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we 
> will add a new protobuf language version (aka proto3) and support a wider 
> range of programming languages (to name a few: ruby, php, node.js, 
> objective-c). This alpha version contains C++ and Java implementation with 
> partial proto3 support (see below for details). In future releases we will 
> add support for more programming languages and implement the full proto3 
> feature set. Besides proto3, this alpha version also includes two other new 
> features: map fields and arena allocation. They are implemented for both 
> proto3 and the old protobuf language version (aka proto2).
>
> We are currently working on the documentation of these new features and 
> when it's ready it will be updated to our protobuf developer guide 
> . For the 
> time being if you have any questions regarding proto3 or other new 
> features, please post your question in the discussion group.
>
> CHANGS
> ===
> Version 3.0.0-alpha-1 (C++/Java):
>
>   General
>   * Introduced Protocol Buffers language version 3 (aka proto3).
>
> When protobuf was initially opensourced it implemented Protocol Buffers
> language version 2 (aka proto2), which is why the version number
> started from v2.0.0. From v3.0.0, a new language version (proto3) is
> introduced while the old version (proto2) will continue to be 
> supported.
>
> The main intent of introducing proto3 is to clean up protobuf before
> pushing the language as the foundation of Google's new API platform.
> In proto3, the language is simplified, both for ease of use and  to
> make it available in a wider range of programming languages. At the
> same time a few features are added to better support common idioms
> found in APIs.
>
> The following are the main new features in language version 3:
>
>   1. Removal of field presence logic for primitive value fields, 
> removal
>  of required fields, and removal of default values. This makes 
> proto3
>  significantly easier to implement with open struct 
> representations,
>  as in languages like Android Java, Objective C, or Go.
>   2. Removal of unknown fields.
>   3. Removal of extensions, which are instead replaced by a new 
> standard
>  type called Any.
>   4. Fix semantics for unknown enum values.
>   5. Addition of maps.
>   6. Addition of a small set of standard types for representation of 
> time,
>  dynamic data, etc.
>   7. A well-defined encoding in JSON as an alternative to binary proto
>  encoding.
>
> This release (v3.0.0-alpha-1) includes partial proto3 support for C++ 
> and
> Java. Items 6 (well-known types) and 7 (JSON format) in the above 
> feature
> list are not implemented.
>
> A new notion "syntax" is introduced to specify whether a .proto file
> uses proto2 or proto3:
>
>   // foo.proto
>   syntax = "proto3";
>   message Bar {...}
>
> If omitted, the protocol compiler will generate a warning and "proto2" 
> will
> be used as the default. This warning will be turned into an error in a
> future release.
>
> We recommend that new Protocol Buffers users use proto3. However, we 
> do not
> generally recommend that existing users migrate from proto2 from 
> proto3 due
> to API incompatibility, and we will continue to support proto2 for a 
> long
> time.
>
>   * Added support for map fields (implemented in C++/Java for both proto2 
> and
> proto3).
>
> Map fields can be declared using the following syntax:
>
>   message Foo {
> map values = 1;
>   }
>
> Data of a map field will be stored in memory as an unordered map and it
> can be accessed through generated accessors.
>
>   C++
>   * Added arena allocation support (for both proto2 and proto3).
>
> Profiling shows memory allocation and deallocation constitutes a 
> significant
> fraction of CPU-time spent in protobuf code and arena allocation is a
> technique introduced to reduce this cost. With arena allocation, new
> objects will be allocated from a large piece of preallocated memory and
> deallocation of these objects is almost free. Early adoption shows 20% 
> to
> 50% improvement in some Google binaries.
>
> To enable arena support, add the following option to your .proto file:
>
>   option cc_enable_arenas = true;
>
> Protocol compiler will generate additional code to make the generated
> message classes work with arenas. This does not change the existing API
> of protobuf messages and doe

[protobuf] Re: EBNF grammar for .proto files

[Walter Schulze]

Is this grammar now officially available and supported somewhere?

On Sunday, 13 July 2008 09:20:16 UTC+2, Alek Storm wrote:
>
> Okay, here is the revised grammar.  Kenton, can we make this official 
> somehow?  Not just because it's mostly my work (okay, kinda), but because 
> it's the *reference* grammar for the language.  It can't just stay in a 
> Groups thread; nobody will find it.  Perhaps a link to it in the 'Language 
> Guide' page?  Or posted there in full?  Developers need something official 
> so they can be confident that their implementations match.  And if the spec 
> changes in the future, it's a lot easier to just check the new grammar.
>
> Because line breaks got mangled in my first post, I've included this one 
> as an attachment.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.