Re: [protobuf] Re: What are well-known types?
The well-known types all have different purposes. The wrapper types allow for optional primitive fields, Any is a replacement for proto2 extensions, and Struct is a way of representing JSON in a protobuf form. I'm not sure what the plan is for the types like Api and Method defined in api.proto, though. On Wed, Mar 6, 2019 at 11:32 AM Dzmitry Lazerka wrote: > Hi guys, > > Is there any documentation now on the purpose of well-known types? Maybe > things have changed in 3 years. What problem they are intended to solve? > > Basically, bumping this... > > On Tuesday, February 23, 2016 at 12:55:21 PM UTC-8, Zellyn wrote: >> >> There are increasing numbers of references to "well-known" types in >> protos. For instance, I see changes in the Go implementation to support >> them. >> There were passing references in release notes in this group. >> >> However, the main protobuf site includes no narrative explanation that I >> can find. >> >> The idea of a few well-known types to represent "Boxed" values since >> proto3 removes the ability to null out fields makes sense, but the only >> documentation I could find, in the reference section of the protobuf site >> at >> https://developers.google.com/protocol-buffers/docs/reference/google.protobuf, >> includes all sorts of things like Struct, Method, Mixin, etc. that are >> entirely unclear. >> >> Is there a conversation happening somewhere that I'm missing, or is it >> Google-internal but not documented outside yet? >> >> Thanks, >> >> Zellyn >> >> ps - the reintroduction of message types for primitives rather undermines >> my belief in the arguments for removing optional fields in proto3 in the >> first place. I'd like to give the benefit of the doubt to the folks >> designing proto3: is the thinking articulated clearly somewhere? >> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Guidance for proto linkage
On Mon, Mar 11, 2019 at 12:24 PM Josh Humphries wrote: > > Since I've implemented this before, I have a fairly lengthy list. Wow, this is pretty comprehensive, much appreciated. I started in on the Protocol Buffer Parser code as a possible stop gap measure, but I'm not sure that this sort of thing, validation, verification, linkage, etc, is that knowable from examination of the source code? I could be wrong or persuaded otherwise, however. I'll review this in more depth, thanks for the feedback, again. > Most of these constraints are in the docs, at least in some way/shape/form. > But some are enforced by protoc but never actually mentioned in the docs > (such as disallowing the use of "map_entry" and "uninterpreted_options" > options): > > Tags for all fields of a given message must be valid: > > Must be in the range 1 to 536,870,911 (2^29-1) inclusive. > Must not be in the reserved range 19,000 to 19,999 (inclusive). > All fields must have a unique tag (i.e. no tag re-use allowed). > Tags from reserved ranges defined in the message are not allowed. > Tags from extension ranges defined in the message are not allowed for normal > fields. Similarly, extension fields must use a tag in one of the message's > extension ranges. > > Other message properties must be valid: > > No field may be named using a reserved name. > Any given reserved range or extension range must not overlap with any other > reserved range or extension range defined in the message. > Reserved names may not contain duplicates. > No message is allowed to use the "map_entry" option. (This option is used in > representing a message as a descriptor, in which a message descriptor is > synthesized for every map field. Only those synthetic messages may have this > option.) > A field whose type refers to a message must not have a "default" option. > Map fields and repeated fields also must not have "default" options. > > Numeric values for all enum values must be valid: > > Numeric values must be in range for signed 32-bit integers (-4,294,967,296 to > 4,294,967,295). > Numeric values for a single enum must be unique and may not be reused unless > the enum includes the option allow_alias set to true. > Numeric values from reserved ranges defined in the enum are not allowed. > > Other enum properties must be valid: > > No value may be named using a reserved name. > Any given reserved range must not overlap with any other reserved range > defined in the enum. > Reserved names may not contain duplicates. > > Must be able to resolve all relative references using protobuf scoping rules: > > Types in field definitions must resolve to an element that is a message or > enum. > Targets of "extends" blocks must resolve to a message. > Request/response types in methods must resolve to a message. > Names in custom options must resolve to an extension. That resolved extension > must extend the appropriate option type (e.g. > "google.protobuf.MessageOptions" for options that are scoped to a message). > > All options must have a value of the correct type: > > The "correct type" of the value is determined by resolving the option name to > a field (or extension) of the appropriate option type. > > Options in file scope must resolve to fields or extensions of > "google.protobuf.FileOptions"; options in a message scope must resolve to > fields or extensions of "google.protobuf.MessageOptions"; etc. > > Normal options (as opposed to custom options) will refer to field names on > the option type itself. > > There are two exceptions: "default" and "json_name" in field options: there > are no fields with these names on "google.protobuf.FieldOptions". (These > instead correspond to fields named "default_value" and "json_name" on the > "google.protobuf.FieldDescriptorProto".) > The "json_name" option value must be a string. > The "default" option value must be the same type as the field itself. So it > must be a literal integer for fields with an integer type, a literal string > for fields with a string or bytes type, etc. (Default options are not allowed > on repeated fields, map fields, and fields whose type is a message.) > > Custom options (those that use parentheses around the first portion of the > option name; e.g. "(custom.option)") will refer to extension fields. > Any two option statements in the same scope may not attempt to set the same > option field(s). > If any path of the option name refers to a message type, there can be > additional path elements that refer to fields of that message type. > For example, in a service option that has name "(custom.option).foo.bar": > > "(custom.option)" refers to an extension that extends > "google.protobuf.ServiceOptions" whose type is a message named MessageOne. > "(custom.option).foo" refers to a field named "foo" in MessageOne. This > field's type is a message named MessageTwo. > "(custom.option).foo.bar" refers to a field named "bar" in MessageTwo. > > References to repeated fields
Re: [protobuf] Guidance for proto linkage
Since I've implemented this before, I have a fairly lengthy list. Most of these constraints are in the docs, at least in some way/shape/form. But some are enforced by protoc but never actually mentioned in the docs (such as disallowing the use of "map_entry" and "uninterpreted_options" options): - Tags for all fields of a given message must be valid: - Must be in the range 1 to 536,870,911 (2^29-1) inclusive. - Must *not* be in the reserved range 19,000 to 19,999 (inclusive). - All fields must have a unique tag (i.e. no tag re-use allowed). - Tags from reserved ranges defined in the message are not allowed. - Tags from extension ranges defined in the message are not allowed for normal fields. Similarly, extension fields *must* use a tag in one of the message's extension ranges. - Other message properties must be valid: - No field may be named using a reserved name. - Any given reserved range or extension range must not overlap with any other reserved range or extension range defined in the message. - Reserved names may not contain duplicates. - No message is allowed to use the "map_entry" option. (This option is used in representing a message as a descriptor, in which a message descriptor is synthesized for every map field. Only those synthetic messages may have this option.) - A field whose type refers to a message must not have a "default" option. - Map fields and repeated fields also must not have "default" options. - Numeric values for all enum values must be valid: - Numeric values must be in range for signed 32-bit integers (-4,294,967,296 to 4,294,967,295). - Numeric values for a single enum must be unique and may not be reused *unless* the enum includes the option allow_alias set to true. - Numeric values from reserved ranges defined in the enum are not allowed. - Other enum properties must be valid: - No value may be named using a reserved name. - Any given reserved range must not overlap with any other reserved range defined in the enum. - Reserved names may not contain duplicates. - Must be able to resolve all relative references using protobuf scoping rules: - Types in field definitions must resolve to an element that is a message or enum. - Targets of "extends" blocks must resolve to a message. - Request/response types in methods must resolve to a message. - Names in custom options must resolve to an extension. That resolved extension must extend the appropriate option type (e.g. "google.protobuf.MessageOptions" for options that are scoped to a message). - All options must have a value of the correct type: - The "correct type" of the value is determined by resolving the option name to a field (or extension) of the appropriate option type. - Options in file scope must resolve to fields or extensions of "google.protobuf.FileOptions"; options in a message scope must resolve to fields or extensions of "google.protobuf.MessageOptions"; etc. - Normal options (as opposed to custom options) will refer to field names on the option type itself. - There are two exceptions: "default" and "json_name" in field options: there are no fields with these names on "google.protobuf.FieldOptions". (These instead correspond to fields named "default_value" and "json_name" on the "google.protobuf.FieldDescriptorProto".) - The "json_name" option value must be a string. - The "default" option value must be the same type as the field itself. So it must be a literal integer for fields with an integer type, a literal string for fields with a string or bytes type, etc. (Default options are not allowed on repeated fields, map fields, and fields whose type is a message.) - Custom options (those that use parentheses around the first portion of the option name; e.g. "(custom.option)") will refer to extension fields. - Any two option statements in the same scope may not attempt to set the same option field(s). - If any path of the option name refers to a message type, there can be additional path elements that refer to fields of that message type. For example, in a service option that has name "(custom.option).foo.bar": - "(custom.option)" refers to an extension that extends "google.protobuf.ServiceOptions" whose type is a message named MessageOne. - "(custom.option).foo" refers to a field named "foo" in MessageOne. This field's type is a message named MessageTwo. - "(custom.option).foo.bar" refers to a field named "bar" in MessageTwo. - References to repeated fields may not have values that appear to be list literals. Instead, a single
