Re: [protobuf] can proto 3 message's bytes parsed by proto2?
On Wednesday, March 20, 2019 at 10:41:43 AM UTC-4, Adam Cozzette wrote: > > That is true, in proto2 the default value of an enum field is the first > value. But the behavior is more confusing than one might expect. Let's say > the only enum values declared are 0, 1, and 2 but you parse a 3 from the > wire. If you examine that field it will appear to be empty and you can read > it to get the default value (0 in this case assuming that was the first > value defined). But the 3 is still there, hidden in the unknown field set. > Once you reserialize the proto, the 3 will be serialized again. Worse > still, since unknown fields are serialized after known fields (that's not a > requirement but typically happens in practice), the unknown enum value can > overwrite a change that your code tried to make. This is why proto3 changed > things so that unknown enum values are stored normally instead of in the > unknown field set, and this makes it much easier to reason about. > You may need to do some vetting of your actual domain data before/after parsing. That's the extent of my knowledge, at least as far as the docs describe it. Perhaps there are also posts, blogs, etc, concerning how to manage versioning of assets; is a problem not exclusive to protocol buffers, ZeroC, ZeroMQ, WCF from back in the day, even nanomsg, I would imagine. On Tue, Mar 19, 2019 at 6:07 PM Michael Powell > wrote: > >> >> >> On Tuesday, March 19, 2019 at 8:25:38 PM UTC-4, Adam Cozzette wrote: >>> >>> Oh, I am talking about the binary format in particular. In that scenario >>> it's important for unknown enum values to be handled in some way, since you >>> might want to add a new enum value but it should still be parseable by >>> older binaries. >>> >> >> From reading the language guide, etc, I take it that the default to the >> first value. That can be anything, 0, 1, 99, whatever. I could be wrong >> there, however; my current interest in Protobuf is in the v2 descriptors >> only, at the moment. >> >> >> >>> On Tue, Mar 19, 2019 at 4:14 PM Michael Powell >>> wrote: >>> >>>> >>>> >>>> On Tuesday, March 19, 2019 at 6:50:37 PM UTC-4, Adam Cozzette wrote: >>>>> >>>>> Not exactly, by unknown enum value I mean an enum value that doesn't >>>>> appear in the enum definition. For example let's say your enum has only >>>>> values 0, 1, and 2 but you parse a 3. This could happen if the message >>>>> was >>>>> serialized by another binary using a newer version of the schema. Proto2 >>>>> will store unknown enum values in the unknown field set whereas proto3 >>>>> will >>>>> just store them normally in the field. >>>>> >>>> >>>> I think in the above scenario, that would likely break whether v2 or >>>> v3, but I could be wrong. >>>> >>>> I would have to re-read the language guide. You can parse an ordinal >>>> value where the Enumeration was expected? >>>> >>>> i.e. UNKNOWN = 0; >>>> >>>> Would accept 0 or UNKNOWN? >>>> >>>> Are we talking descriptor / protobuf specification level? Or binary >>>> level? >>>> >>>> I do not read any biases where unexpected ordinal values are concerned, >>>> but I would expect that it fail any sort of verification. >>>> >>>> That being said, specification versioning is a concern regardless of >>>> whether v2 or v3, I think, and not just with Protocol Buffers. It's a >>>> concern for this type of framework, regardless. >>>> >>>> On Thu, Mar 14, 2019 at 7:29 PM Jason Huang >>>>> wrote: >>>>> >>>>>> thanks for your reply , as for the `unknown enum values` , do you >>>>>> mean the case of a unseted enum field ? >>>>>> >>>>>> On Friday, 15 March 2019 02:12:31 UTC+8, Adam Cozzette wrote: >>>>>>> >>>>>>> Going from proto3 to proto2 should be fine. There are some slight >>>>>>> differences but I can't think of any major problems. The only thing >>>>>>> that >>>>>>> comes to mind is that proto2 handles unknown enum values a little bit >>>>>>> differently from proto3. I doubt that would be a problem but if you >>>>>>> want to >>>>>>> be extra cautious you could double-chec
Re: [protobuf] can proto 3 message's bytes parsed by proto2?
On Tuesday, March 19, 2019 at 8:25:38 PM UTC-4, Adam Cozzette wrote: > > Oh, I am talking about the binary format in particular. In that scenario > it's important for unknown enum values to be handled in some way, since you > might want to add a new enum value but it should still be parseable by > older binaries. > >From reading the language guide, etc, I take it that the default to the first value. That can be anything, 0, 1, 99, whatever. I could be wrong there, however; my current interest in Protobuf is in the v2 descriptors only, at the moment. > On Tue, Mar 19, 2019 at 4:14 PM Michael Powell > wrote: > >> >> >> On Tuesday, March 19, 2019 at 6:50:37 PM UTC-4, Adam Cozzette wrote: >>> >>> Not exactly, by unknown enum value I mean an enum value that doesn't >>> appear in the enum definition. For example let's say your enum has only >>> values 0, 1, and 2 but you parse a 3. This could happen if the message was >>> serialized by another binary using a newer version of the schema. Proto2 >>> will store unknown enum values in the unknown field set whereas proto3 will >>> just store them normally in the field. >>> >> >> I think in the above scenario, that would likely break whether v2 or v3, >> but I could be wrong. >> >> I would have to re-read the language guide. You can parse an ordinal >> value where the Enumeration was expected? >> >> i.e. UNKNOWN = 0; >> >> Would accept 0 or UNKNOWN? >> >> Are we talking descriptor / protobuf specification level? Or binary >> level? >> >> I do not read any biases where unexpected ordinal values are concerned, >> but I would expect that it fail any sort of verification. >> >> That being said, specification versioning is a concern regardless of >> whether v2 or v3, I think, and not just with Protocol Buffers. It's a >> concern for this type of framework, regardless. >> >> On Thu, Mar 14, 2019 at 7:29 PM Jason Huang wrote: >>> >>>> thanks for your reply , as for the `unknown enum values` , do you mean >>>> the case of a unseted enum field ? >>>> >>>> On Friday, 15 March 2019 02:12:31 UTC+8, Adam Cozzette wrote: >>>>> >>>>> Going from proto3 to proto2 should be fine. There are some slight >>>>> differences but I can't think of any major problems. The only thing that >>>>> comes to mind is that proto2 handles unknown enum values a little bit >>>>> differently from proto3. I doubt that would be a problem but if you want >>>>> to >>>>> be extra cautious you could double-check that you're not storing any >>>>> unknown enum values. >>>>> >>>>> However, there is no need to downgrade to version 2.6.1 and if >>>>> anything that would only introduce bugs and make the code slower. The >>>>> proto2 semantics are still fully supported in all versions going forward, >>>>> so all you have to do is put syntax = "proto2"; at the top of your >>>>> .proto files. You can stick with 3.6.1 or even upgrade to any newer >>>>> version. >>>>> >>>>> On Thu, Mar 14, 2019 at 10:21 AM Jason Huang >>>>> wrote: >>>>> >>>>>> i chose proto3 for cache （serialize） for my application , and it have >>>>>> run for several month . but now i wanna change my mind for proto2 , >>>>>> because >>>>>> the hasField is really needed . >>>>>> >>>>>> the problem is , there're still lots of data in the cache which are >>>>>> serialized with proto3 , if i can't deserialize with them with proto2 . >>>>>> it >>>>>> will be unacceptable . >>>>>> >>>>>> my question is , is it safe to switch from proto3 to proto2 ? i made >>>>>> some test i could work in some case . but i'm afraid this is not full >>>>>> coverage test . >>>>>> >>>>>> my current proto3 version is 3.6.1 , and the proto2 version i want to >>>>>> use is 2.6.1 >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Protocol Buffers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to protobuf+u...@googlegrou
Re: [protobuf] can proto 3 message's bytes parsed by proto2?
On Tuesday, March 19, 2019 at 6:50:37 PM UTC-4, Adam Cozzette wrote: > > Not exactly, by unknown enum value I mean an enum value that doesn't > appear in the enum definition. For example let's say your enum has only > values 0, 1, and 2 but you parse a 3. This could happen if the message was > serialized by another binary using a newer version of the schema. Proto2 > will store unknown enum values in the unknown field set whereas proto3 will > just store them normally in the field. > I think in the above scenario, that would likely break whether v2 or v3, but I could be wrong. I would have to re-read the language guide. You can parse an ordinal value where the Enumeration was expected? i.e. UNKNOWN = 0; Would accept 0 or UNKNOWN? Are we talking descriptor / protobuf specification level? Or binary level? I do not read any biases where unexpected ordinal values are concerned, but I would expect that it fail any sort of verification. That being said, specification versioning is a concern regardless of whether v2 or v3, I think, and not just with Protocol Buffers. It's a concern for this type of framework, regardless. On Thu, Mar 14, 2019 at 7:29 PM Jason Huang > wrote: > >> thanks for your reply , as for the `unknown enum values` , do you mean >> the case of a unseted enum field ? >> >> On Friday, 15 March 2019 02:12:31 UTC+8, Adam Cozzette wrote: >>> >>> Going from proto3 to proto2 should be fine. There are some slight >>> differences but I can't think of any major problems. The only thing that >>> comes to mind is that proto2 handles unknown enum values a little bit >>> differently from proto3. I doubt that would be a problem but if you want to >>> be extra cautious you could double-check that you're not storing any >>> unknown enum values. >>> >>> However, there is no need to downgrade to version 2.6.1 and if anything >>> that would only introduce bugs and make the code slower. The proto2 >>> semantics are still fully supported in all versions going forward, so all >>> you have to do is put syntax = "proto2"; at the top of your .proto >>> files. You can stick with 3.6.1 or even upgrade to any newer version. >>> >>> On Thu, Mar 14, 2019 at 10:21 AM Jason Huang >>> wrote: >>> i chose proto3 for cache （serialize） for my application , and it have run for several month . but now i wanna change my mind for proto2 , because the hasField is really needed . the problem is , there're still lots of data in the cache which are serialized with proto3 , if i can't deserialize with them with proto2 . it will be unacceptable . my question is , is it safe to switch from proto3 to proto2 ? i made some test i could work in some case . but i'm afraid this is not full coverage test . my current proto3 version is 3.6.1 , and the proto2 version i want to use is 2.6.1 -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com. To post to this group, send email to prot...@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. >>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com . >> To post to this group, send email to prot...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Q: v2 field numbering scope
On Thursday, March 14, 2019 at 8:48:54 PM UTC-4, Michael Powell wrote: > > Hello, > > I am scanning through the v2 language guide, and the reserved field > numbers as well as the allowed field numbers makes sense. > > What does not make sense is the scope. Is the scope contained with in a > given message? That is, MessageA has one set of field numbers, whereas > MessageB has an entirely different, distinct set of field numbers? > Anyone at all? I'm thinking Message is the scope of a unique set of field numbers, but I could be wrong there. Just looking for some clarification. Or is the scope broader than that? Say, an entire Package, Proto file, etc? > > Thanks! > > Best regards, > > Michael W Powell > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] can proto 3 message's bytes parsed by proto2?
On Thursday, March 14, 2019 at 2:12:31 PM UTC-4, Adam Cozzette wrote: > > Going from proto3 to proto2 should be fine. There are some slight > differences but I can't think of any major problems. The only thing that > comes to mind is that proto2 handles unknown enum values a little bit > differently from proto3. I doubt that would be a problem but if you want to > be extra cautious you could double-check that you're not storing any > unknown enum values. > > However, there is no need to downgrade to version 2.6.1 and if anything > that would only introduce bugs and make the code slower. The proto2 > semantics are still fully supported in all versions going forward, so all > you have to do is put syntax = "proto2"; at the top of your .proto files. > You can stick with 3.6.1 or even upgrade to any newer version. > > On Thu, Mar 14, 2019 at 10:21 AM Jason Huang > wrote: > >> i chose proto3 for cache （serialize） for my application , and it have run >> for several month . but now i wanna change my mind for proto2 , because the >> hasField is really needed . >> >> the problem is , there're still lots of data in the cache which are >> serialized with proto3 , if i can't deserialize with them with proto2 . it >> will be unacceptable . >> > There is "some" backwards compatibility if you need to import proto3 into proto2. Groups is the chief one that the docs talk about that I know of. https://developers.google.com/protocol-buffers/docs/proto#groups my question is , is it safe to switch from proto3 to proto2 ? i made some >> test i could work in some case . but i'm afraid this is not full coverage >> test . >> >> my current proto3 version is 3.6.1 , and the proto2 version i want to use >> is 2.6.1 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com . >> To post to this group, send email to prot...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Q: v2 field numbering scope
Hello, I am scanning through the v2 language guide, and the reserved field numbers as well as the allowed field numbers makes sense. What does not make sense is the scope. Is the scope contained with in a given message? That is, MessageA has one set of field numbers, whereas MessageB has an entirely different, distinct set of field numbers? Or is the scope broader than that? Say, an entire Package, Proto file, etc? Thanks! Best regards, Michael W Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Typo in v2 spec
Hello,
I think there's a typo in the v2 spec, re: Field Names Reserved Statement.
Consider:
ident = letter { letter | decimalDigit | "_" }
fieldName = ident
reserved = "reserved" ( ranges | fieldNames ) ";"
fieldNames = fieldName { "," fieldName }
In the case of a list of Field Names, I do not see any reference to any
enclosed quotes. However, in the examples in both the specification as well
as the language guide, there are quotes.
message Foo {
reserved 2, 15, 9 to 11;
reserved "foo", "bar";
}
reserved "foo", "bar";
Thoughts? I just wanted to mention that, I think this is a typo. It makes
sense to me that these are literally Identifiers. But if there are
different thoughts?
Cheers,
Michael W Powell
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#reserved
https://developers.google.com/protocol-buffers/docs/proto#reserved
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec
https://developers.google.com/protocol-buffers/docs/proto
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Guidance for proto linkage
On Tuesday, March 12, 2019 at 9:43:38 PM UTC-4, Josh Humphries wrote: > > > On Tue, Mar 12, 2019 at 7:31 PM Michael Powell > wrote: > >> >> >> On Monday, March 11, 2019 at 12:24:57 PM UTC-4, Josh Humphries wrote: >>> >>> Since I've implemented this before, I have a fairly lengthy list. >>> >>> Most of these constraints are in the docs, at least in some >>> way/shape/form. But some are enforced by protoc but never actually >>> mentioned in the docs (such as disallowing the use of "map_entry" and >>> "uninterpreted_options" options): >>> >> >> So, this may sound like a stupid question, but where do you even go to >> discover heuristics such as these? Well, besides this super comprehensive >> list, that is. >> > > I go to the source: in this case protoc (and sometimes its C source code). > > Many I discovered because I was writing a parser/compiler in Go. I > realized that these were areas where the spec was light and the comments in > descriptor.proto weren't crystal clear. So I just tried a few things out to > observe the behavior of protoc. > I see, I see. Thanks for pointing that out. For example, it just seems like these are so arbitrary. I'm sure there is a >> reason why, but in and of itself, "just because" does not seem like an >> adequate response: >> >> >>- If the file indicates syntax = "proto2": >> - Fields and extensions must specify a label: "optional", >> "repeated", or "required" (excluding those in oneof declarations and >> excluding map fields, which must *not* have a label). >> - Extension fields must *not* use the "required" label. >> >> >> Are these heuristics internal bits? Or are they really coming from Google >> Protocol Buffers? >> > > I'm not sure I understand the question. I think the answer is that they > come from Google Protocol Buffers. > > Regarding labels, this is a difference in the actual language > <https://developers.google.com/protocol-buffers/docs/reference/proto2-spec> > specs > <https://developers.google.com/protocol-buffers/docs/reference/proto3-spec>. > I was writing a single parser that could support either, so instead of > implementing two different specs, I merged the two. And the label thing > shook out as a difference between the two specs. > > For extensions not being required, it's not explicitly documented. But > given that extensions are not meant to even be known by all clients, it's > intuitive that they can't be required. It's also hinted at in the docs, where > it is stated > <https://developers.google.com/protocol-buffers/docs/proto#updating> that > non-required fields can safely be converted to extensions and vice versa. > > >> >> >>> >>>- Tags for all fields of a given message must be valid: >>> - Must be in the range 1 to 536,870,911 (2^29-1) inclusive. >>> - Must *not* be in the reserved range 19,000 to 19,999 >>> (inclusive). >>> - All fields must have a unique tag (i.e. no tag re-use allowed). >>> - Tags from reserved ranges defined in the message are not >>> allowed. >>> - Tags from extension ranges defined in the message are not >>> allowed for normal fields. Similarly, extension fields *must* use >>> a tag in one of the message's extension ranges. >>>- Other message properties must be valid: >>> - No field may be named using a reserved name. >>> - Any given reserved range or extension range must not overlap >>> with any other reserved range or extension range defined in the >>> message. >>> - Reserved names may not contain duplicates. >>> - No message is allowed to use the "map_entry" option. (This >>> option is used in representing a message as a descriptor, in which a >>> message descriptor is synthesized for every map field. Only those >>> synthetic >>> messages may have this option.) >>> - A field whose type refers to a message must not have a >>> "default" option. >>> - Map fields and repeated fields also must not have "default" >>> options. >>>- Numeric values for all enum values must be valid: >>> - Numeric values must be in range for signed 32-bit integers >>> (-4,294,967,296 to 4,294,967,295). >>> - N
Re: [protobuf] Guidance for proto linkage
Proto".)
> - The "json_name" option value must be a string.
> - The "default" option value must be the same type as the field
> itself. So it must be a literal integer for fields with an integer
> type, a
> literal string for fields with a string or bytes type, etc. (Default
> options are not allowed on repeated fields, map fields, and fields
> whose
> type is a message.)
> - Custom options (those that use parentheses around the first
> portion of the option name; e.g. "(custom.option)") will refer to
> extension
> fields.
> - Any two option statements in the same scope may not attempt to
> set the same option field(s).
> - If any path of the option name refers to a message type, there
> can be additional path elements that refer to fields of that message
> type.
> For example, in a service option that has name
> "(custom.option).foo.bar":
> - "(custom.option)" refers to an extension that extends
> "google.protobuf.ServiceOptions" whose type is a message named
> MessageOne.
> - "(custom.option).foo" refers to a field named "foo" in
> MessageOne. This field's type is a message named MessageTwo.
> - "(custom.option).foo.bar" refers to a field named "bar" in
> MessageTwo.
> - References to repeated fields may not have values that appear
> to be list literals. Instead, a single option that references a
> repeated
> field defines a single element of the repeated option. Subsequent
> options
> that reference the same repeated field define subsequent elements.
> - Only the *leaf* field named can be a repeated field. If earlier
> path elements name repeated fields, the option is invalid. Instead,
> the
> option statement must refer only to the *first* repeated field
> in the path and then use an aggregate value (which then includes all
> values
> for nested repeated fields).
> - References to message fields may use an aggregate value: an
> aggregate value is enclosed in curly braces "{ }" and uses the protobuf
> text format therein to define the message value.
> - Option values whose type is an enum must use unqualified
> identifiers. The identifier must be one of the values in the enum.
>- No option statement may refer to an option named
>"uninterpreted_option". (This field of the various option types is used
>internally by protoc and other parsers to represent
>unresolved/uninterpreted option statements.)
>- If the file indicates syntax = "proto3":
> - No "optional" or "required" labels are allowed on any field
> definition. (Fields are always optional in proto3, unless they have the
> "repeated" label or are map fields.)
> - No messages may define extension ranges.
> - No messages may define groups.
> - No messages may include a field whose type is an enum defined in
> a file with syntax = "proto2".
> - The first value for an enum *must* have a numeric value of zero.
> (Similarly: every enum must have a value whose numeric value is zero.)
> - Field definitions may not use the "default" option (default field
> values in proto3 are always the zero value for the field's type).
>- If the file indicates syntax = "proto2":
> - Fields and extensions must specify a label: "optional",
> "repeated", or "required" (excluding those in oneof declarations and
> excluding map fields, which must *not* have a label).
> - Extension fields must *not* use the "required" label.
>
>
>
>
> *Josh Humphries*
> jh...@bluegosling.com
>
>
> On Sat, Mar 9, 2019 at 2:03 PM Michael Powell > wrote:
>
>> Hello,
>>
>> I am looking for guidance as far as what steps one must take in order to
>> verify that a Proto is valid, link the names, i.e. considering ambiguities
>> of Element (i.e. Message or Enum) Type Names during parsing, etc.
>>
>> Some things seem obvious, such as the Field Numbers must be unique, that
>> sort of thing, but a more comprehensive set of guidelines would be helpful.
>>
>> Cheers, thank you.
>>
>> Michael W Powell
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Guidance for proto linkage
> For example, in a service option that has name "(custom.option).foo.bar":
>
> "(custom.option)" refers to an extension that extends
> "google.protobuf.ServiceOptions" whose type is a message named MessageOne.
> "(custom.option).foo" refers to a field named "foo" in MessageOne. This
> field's type is a message named MessageTwo.
> "(custom.option).foo.bar" refers to a field named "bar" in MessageTwo.
>
> References to repeated fields may not have values that appear to be list
> literals. Instead, a single option that references a repeated field defines a
> single element of the repeated option. Subsequent options that reference the
> same repeated field define subsequent elements.
>
> Only the leaf field named can be a repeated field. If earlier path elements
> name repeated fields, the option is invalid. Instead, the option statement
> must refer only to the first repeated field in the path and then use an
> aggregate value (which then includes all values for nested repeated fields).
>
> References to message fields may use an aggregate value: an aggregate value
> is enclosed in curly braces "{ }" and uses the protobuf text format therein
> to define the message value.
> Option values whose type is an enum must use unqualified identifiers. The
> identifier must be one of the values in the enum.
>
> No option statement may refer to an option named "uninterpreted_option".
> (This field of the various option types is used internally by protoc and
> other parsers to represent unresolved/uninterpreted option statements.)
> If the file indicates syntax = "proto3":
>
> No "optional" or "required" labels are allowed on any field definition.
> (Fields are always optional in proto3, unless they have the "repeated" label
> or are map fields.)
> No messages may define extension ranges.
> No messages may define groups.
> No messages may include a field whose type is an enum defined in a file with
> syntax = "proto2".
> The first value for an enum must have a numeric value of zero. (Similarly:
> every enum must have a value whose numeric value is zero.)
> Field definitions may not use the "default" option (default field values in
> proto3 are always the zero value for the field's type).
>
> If the file indicates syntax = "proto2":
>
> Fields and extensions must specify a label: "optional", "repeated", or
> "required" (excluding those in oneof declarations and excluding map fields,
> which must not have a label).
> Extension fields must not use the "required" label.
>
>
>
>
> Josh Humphries
> jh...@bluegosling.com
>
>
> On Sat, Mar 9, 2019 at 2:03 PM Michael Powell wrote:
>>
>> Hello,
>>
>> I am looking for guidance as far as what steps one must take in order to
>> verify that a Proto is valid, link the names, i.e. considering ambiguities
>> of Element (i.e. Message or Enum) Type Names during parsing, etc.
>>
>> Some things seem obvious, such as the Field Numbers must be unique, that
>> sort of thing, but a more comprehensive set of guidelines would be helpful.
>>
>> Cheers, thank you.
>>
>> Michael W Powell
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Guidance for proto linkage
Hello, I am looking for guidance as far as what steps one must take in order to verify that a Proto is valid, link the names, i.e. considering ambiguities of Element (i.e. Message or Enum) Type Names during parsing, etc. Some things seem obvious, such as the Field Numbers must be unique, that sort of thing, but a more comprehensive set of guidelines would be helpful. Cheers, thank you. Michael W Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Resolving a Protobuf descriptor identification strategy
Hello, I'd like to get a handle on an identification strategy for elements in a Protocol Buffer descriptor hierarchy and just trying to get a sense of what is involved. My parser is working well. I decided to take an approach through Antlr4 for dotnet and CSharp after all. ... Where is some combination of items, such as messages, enumerations, and is a terminal item, such as options, enumerated fields, and so on. I am focused on the v2 specification, however, this may be applicable to v3 as well, though the specific protos I am interested in are currently presented in the v2 version. http://developers.google.com/protocol-buffers/docs/reference/proto2-spec I'm thinking that the whole Identification strategy need to be stronger than language level String, for instance, in order to best align with a path through an AST tree, but I'd be curious what insights others are willing to share. Cheers, and thanks in advance for any insights, Michael W Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Language mappings for Protobuf Types
I found the same for proto2, thanks! On Wed, Jan 30, 2019 at 8:12 PM Michael Powell wrote: > > On Wed, Jan 30, 2019 at 8:05 PM Mohammed Nisamudheen S > wrote: > > > > Following table can be the one you are looking for. > > > > https://developers.google.com/protocol-buffers/docs/proto3#scalar > > Sorry, I should be more specific: proto2. But this is a start, thank you. > > > On Thu, Jan 31, 2019 at 3:35 AM Michael Powell > > wrote: > >> > >> Hello, > >> > >> I am looking for a table or some cross reference for target language > >> to Protocol Buffer types and key types. I can make some informed > >> guesses, of course, but if there was a generally recognized table, > >> cross reference, something like that, that would be a useful > >> reference. > >> > >> Thanks! > >> > >> Best regards, > >> > >> Michael W Powell > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Protocol Buffers" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to protobuf+unsubscr...@googlegroups.com. > >> To post to this group, send email to protobuf@googlegroups.com. > >> Visit this group at https://groups.google.com/group/protobuf. > >> For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Language mappings for Protobuf Types
On Wed, Jan 30, 2019 at 8:05 PM Mohammed Nisamudheen S wrote: > > Following table can be the one you are looking for. > > https://developers.google.com/protocol-buffers/docs/proto3#scalar Sorry, I should be more specific: proto2. But this is a start, thank you. > On Thu, Jan 31, 2019 at 3:35 AM Michael Powell wrote: >> >> Hello, >> >> I am looking for a table or some cross reference for target language >> to Protocol Buffer types and key types. I can make some informed >> guesses, of course, but if there was a generally recognized table, >> cross reference, something like that, that would be a useful >> reference. >> >> Thanks! >> >> Best regards, >> >> Michael W Powell >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+unsubscr...@googlegroups.com. >> To post to this group, send email to protobuf@googlegroups.com. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Language mappings for Protobuf Types
Hello, I am looking for a table or some cross reference for target language to Protocol Buffer types and key types. I can make some informed guesses, of course, but if there was a generally recognized table, cross reference, something like that, that would be a useful reference. Thanks! Best regards, Michael W Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] v2 disposition towards nodes like "package", and then some
On Fri, Dec 28, 2018 at 1:08 PM Josh Humphries wrote:
>
> On Fri, Dec 28, 2018 at 11:57 AM Michael Powell wrote:
>>
>> Hello,
>>
>> At present I've "compiled" the .proto (v2) to its AST; analogous, I
>> think, to the protoc "descriptor". I've got parent to child
>> relationships captured intrinsic to the AST being synthesized. I've
>> developed a visitor that connects the parent to the children,
>> subsequent to the parsing to AST, which captures the "having parent"
>> aspect. So at this point I am confident I can navigate the AST
>> downwards as well as upwards, to siblings, etc.
>>
>> Now, when deciding how to prefix and otherwise "build" the path to
>> element names, I assume it is not unlike Java, starting with the
>> package name, extending through the messages, etc. Or C++ and/or C#,
>> starting from a namespace.
>>
>> I have scanned over the C++ and Java and so forth Proto v2 specs,
>> language guides, etc, and at first blush, prima facie, it makes sense.
>>
>> However, how would one handle cases where more than one package exists
>> in a single .proto file? Say, I had:
>>
>> package "foo.baz";
>> message A { /* ... */ }
>> package "baz.fiz";
>> message B { /* ... */ }
>>
>> I gather the A fully qualified name would be ".foo.baz.A", correct?
>> Whereas the fully qualified B name would be ".baz.fiz.B"?
>
> That is not a valid proto file. Proto files can have only one package
> declaration. If you try to compile this file with protoc, you'll get an error
> pointing to the second package declaration.
Well, I wouldn't think it is, either, however in the v2 spec, per se,
there is nothing there explicitly forbidding it.
>> Reading a bit deeper into the docs, there is one area however that
>> does not make much sense to me:
>>
>> https://developers.google.com/protocol-buffers/docs/proto#customoptions
>>
>> In the MyService, I'm not sure how one can make the leap from MethodOptions
>> to:
>>
>> // ...
>> extend google.protobuf.MethodOptions {
>> optional MyMessage my_method_option = 50006;
>> }
>> // ...
>> // Note: my_method_option has type MyMessage. We can set each field
>> // within it using a separate "option" line.
>> option (my_method_option).foo = 567;
>> option (my_method_option).bar = "Some string";
>> // ...
>>
>> Yes, my_method_option is of type MyMessage. That's a given. However,
>> there does not appear to be anything connecting that to any element
>> the Service would be dealing with.
>
>
> Sorry: I do not understand what you are asking.
>
>>
>>
>> Even diving into the Google "descriptor.proto", yes, there are
>> "messages" defined, sort of meta-metadata describing what a service
>> should look like. Apart from that, I think it is possible the proto
>> compiler itself that is binding the pieces together? Because from what
>> I can tell, there does not appear to be anything that would suggest
>> that a "MethodOptions" should appear anywhere accessible to a service
>> RPC method. Certainly not without an "(option)" prefix, I think?
>
>
> I'm having great difficulty following along. When you say a "a MethodOptions
> should appear anywhere accessible to a service RPC method", I really don't
> follow.
Really, I'm just trying to gain some comprehension over how the
elements connect. From the v2 examples, which extend out from the
Protocol Buffer .protos themselves, it seems, I'm not sure that I can
readily identify the names, etc, that would lend itself to the example
as it has been illustrated.
> Is this question how to use custom options or how to support them when
> writing a custom parser and linker for proto sources? The MethodOptions
> message represents all options defined in an RPC method declaration. Each
> field represents a possible "known" option (no parentheses around name). And
> it has an extension range in which custom options are defined (whose names
> must be referenced in source surrounded by parentheses).
>>
>> Seems very ad-hoc to me. Am I missing something there?
>
> Well, it's general in that all options and custom options can be parsed the
> same way (well, slight variance for enum value and field options, just due to
> their compact syntax -- but semantics are the same).
>
> Do you mean "ad hoc" because these options are defined in the language spec?
> If so, that is intentional, mainly due to the fact that they can refe
[protobuf] v2 disposition towards nodes like "package", and then some
Hello,
At present I've "compiled" the .proto (v2) to its AST; analogous, I
think, to the protoc "descriptor". I've got parent to child
relationships captured intrinsic to the AST being synthesized. I've
developed a visitor that connects the parent to the children,
subsequent to the parsing to AST, which captures the "having parent"
aspect. So at this point I am confident I can navigate the AST
downwards as well as upwards, to siblings, etc.
Now, when deciding how to prefix and otherwise "build" the path to
element names, I assume it is not unlike Java, starting with the
package name, extending through the messages, etc. Or C++ and/or C#,
starting from a namespace.
I have scanned over the C++ and Java and so forth Proto v2 specs,
language guides, etc, and at first blush, prima facie, it makes sense.
However, how would one handle cases where more than one package exists
in a single .proto file? Say, I had:
package "foo.baz";
message A { /* ... */ }
package "baz.fiz";
message B { /* ... */ }
I gather the A fully qualified name would be ".foo.baz.A", correct?
Whereas the fully qualified B name would be ".baz.fiz.B"?
Reading a bit deeper into the docs, there is one area however that
does not make much sense to me:
https://developers.google.com/protocol-buffers/docs/proto#customoptions
In the MyService, I'm not sure how one can make the leap from MethodOptions to:
// ...
extend google.protobuf.MethodOptions {
optional MyMessage my_method_option = 50006;
}
// ...
// Note: my_method_option has type MyMessage. We can set each field
// within it using a separate "option" line.
option (my_method_option).foo = 567;
option (my_method_option).bar = "Some string";
// ...
Yes, my_method_option is of type MyMessage. That's a given. However,
there does not appear to be anything connecting that to any element
the Service would be dealing with.
Even diving into the Google "descriptor.proto", yes, there are
"messages" defined, sort of meta-metadata describing what a service
should look like. Apart from that, I think it is possible the proto
compiler itself that is binding the pieces together? Because from what
I can tell, there does not appear to be anything that would suggest
that a "MethodOptions" should appear anywhere accessible to a service
RPC method. Certainly not without an "(option)" prefix, I think?
Seems very ad-hoc to me. Am I missing something there?
...
Not to mention the "ad hoc" appearance of "supported" options:
https://developers.google.com/protocol-buffers/docs/proto#options
Just looking for a little clarification here, if you please.
Cheers,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Looking for guidelines for v2 transitive linkage
Hello, I am looking for a little guidance, guidelines, etc, concerning the transitive linking process. In basic terms, it seems to me that much of this has to do with ensuring that the field numbers, etc, are all aligned properly, there are no collisions, the numbers identified for reserved, extensions, etc, are all accounted for, and so on. Then there's the matter of, I suppose, specifying import directories. First of all either testing whether the import path exists, whether it is absolute or relative is immaterial, I think; or scan some user specified import directories for the same. Another significant aspect is how to treat the imports via vis-a-vis the import modifier. I'd received some feedback along these lines, but any clarification along these lines would also be appreciated. Best regards, Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Import statement semantics
On Wed, Nov 7, 2018 at 4:28 PM Michael Powell wrote: > > Hello, > > Is there some documentation concerning import semantics? > > For starters, I assume that you specify the path. This may be further > enhanced by specification of an "import path", not unlike in C++ > wherein you may specify include, library, etc, paths. I have not received any clarification for this. Can someone please clarify? I want to think in terms of C++ access modifiers, private, protected, public, etc, but I'm not positive this is analogous, or what the ramifications ought to be when "linking" across .proto files. http://developers.google.com/protocol-buffers/docs/reference/proto2-spec#import_statement > Second, what is the impact of "weak"? "public"? Or an unspecified modifier? > > Thanks! > > Best regards, > > Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Proto v2 ambiguities in parsed identifiers
On Thu, Nov 29, 2018 at 1:08 PM Josh Humphries wrote:
>
> You know it's a message vs. enum (or invalid identifier) only after linking.
Thanks very much for the clarification. I sort of figured that was the case.
> To link, you need the file's entire transitive closure (e.g. its imports and
> their imports and so on) already parsed (and, preferably, already linked).
> You can then resolve the type name, basically using C++ scoping rules. The
> leading dot tells it that the identifier is fully-qualified -- so you don't
> need to worry about scoping rules (which allows for partially-qualified or
> unqualified names) but instead search for the symbol by full name. After you
> have resolved the name to an element, you then know the element's type --
> whether it was a message or an enum (or if it was invalid, such as no
> matching element or referring to a service or method instead of a type).
>
> An element's full name is the file package (if any), all enclosing element
> names (note that in an "extend" block, the extendee is not considered an
> element name: extend blocks have no name), and then item itself.
>
> Example:
>
> syntax = "proto3";
> package foo.bar;
> import "google/protobuf/descriptor.proto";
>
> extend google.protobuf.MessageOptions {
> string some_custom_option = ; // fqn = foo.bar.some_custom_option
> }
>
> enum SomeEnum { // fqn = foo.bar.SomeEnum
> VAL0 = 0; // fqn = foo.bar.SomeEnum.VAL0
> }
>
> message SomeMsg { // fqn = foo.bar.SomeMsg
> string name = 1; // fqn = foo.bar.SomeMsg.name
>
> // refers to top-level SomeEnum above
> .foo.bar.SomeEnum enum2 = 3; // fqn = foo.bar.SomeMsg.enum2
>
> // refers to nested SomeEnum below
> SomeEnum enum = 2; // fqn = foo.bar.SomeMsg.enum
>
> enum SomeEnum { // fqn = foo.bar.SomeMsg.SomeEnum
> VAL0 = 0; // fqn = foo.bar.SomeMsg.SomeEnum.VAL0
> }
>
> // In this scope, can refer to SomeEnum w/out qualifier, as SomeEnum.
> // Could also use partial-qualifiers, like SomeMsg.SomeEnum,
> bar.SomeMsg.SomeEnum,
> // or foo.bar.SomeMsg.SomeEnum. That last one is actually fully-qualified,
> but protoc
> // doesn't know that, so it must use same scope rules to resolve symbol
> name. However,
> // using .foo.bar.SomeMsg.SomeEnum (leading dot) means the current lexical
> scope
> // doesn't matter, so symbol resolution can take "short cut".
> //
> // As you can see, compound package names work like nested C++ namespaces,
> not like
> // Java or .NET package names.
>
> message NestedMsg { // fqn = foo.bar.SomeMsg.NestedMsg
> extends google.protobuf.FieldOption {
> uint64 some_other_option = ; // fqn =
> foo.bar.SomeMsg.NestedMsg.some_other_option
> }
> }
> }
>
>
> Josh Humphries
> jh...@bluegosling.com
>
>
> On Thu, Nov 29, 2018 at 9:14 AM Michael Powell wrote:
>>
>> Hello,
>>
>> In the Protocol Buffer v2 Field specification there is a notion of
>> messageType and enumType. Casually reading the specification, it is
>> easy to miss this, but there is nothing there to tell anyone whether a
>> messageType or an enumType was encountered, except perhaps a second
>> pass verification.
>>
>> Which brings me to the second question, how does one interpret the
>> identifiers involved in the type specification? Also, what's with the
>> leading dot? Please explain. Thanks!
>>
>> Also, minor question, is the trailing semi-colon a typo? i.e.
>> "fieldNumber = intLit;". I gather that perhaps it is a typo but I
>> wanted to clarify that as well.
>>
>> // Identifiers
>> ident = letter { letter | decimalDigit | "_" }
>> // ...
>> messageName = ident
>> enumName = ident
>> // ...
>> messageType = [ "." ] { ident "." } messageName
>> enumType = [ "." ] { ident "." } enumName
>> // ...
>> // Field
>> label = "required" | "optional" | "repeated"
>> type = "double" | "float" | "int32" | "int64" | "uint32" | "uint64" |
>> "sint32" | "sint64" | "fixed32" | "fixed64" | "sfixed32" | "sfixed64"
>> | "bool" | "string" | "bytes" | messageType | enumType
>> fieldNumber = intLit; // <- is the trailing semicolon here a typo?
>>
>> http://developers.google.com/protocol-buffers/docs/reference/proto2-spec
>>
>> Best,
>>
>> Michael Powell
>>
>> --
>> You received this message b
[protobuf] Proto v2 ambiguities in parsed identifiers
Hello,
In the Protocol Buffer v2 Field specification there is a notion of
messageType and enumType. Casually reading the specification, it is
easy to miss this, but there is nothing there to tell anyone whether a
messageType or an enumType was encountered, except perhaps a second
pass verification.
Which brings me to the second question, how does one interpret the
identifiers involved in the type specification? Also, what's with the
leading dot? Please explain. Thanks!
Also, minor question, is the trailing semi-colon a typo? i.e.
"fieldNumber = intLit;". I gather that perhaps it is a typo but I
wanted to clarify that as well.
// Identifiers
ident = letter { letter | decimalDigit | "_" }
// ...
messageName = ident
enumName = ident
// ...
messageType = [ "." ] { ident "." } messageName
enumType = [ "." ] { ident "." } enumName
// ...
// Field
label = "required" | "optional" | "repeated"
type = "double" | "float" | "int32" | "int64" | "uint32" | "uint64" |
"sint32" | "sint64" | "fixed32" | "fixed64" | "sfixed32" | "sfixed64"
| "bool" | "string" | "bytes" | messageType | enumType
fieldNumber = intLit; // <- is the trailing semicolon here a typo?
http://developers.google.com/protocol-buffers/docs/reference/proto2-spec
Best,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Getting started developing a protoc plugin
On Wed, Nov 28, 2018 at 1:03 AM Nadav Samet wrote: > > In Java, the dependencies you mentioned are provided by Gradle. We pull > protobuf-java in here. I see... Been a little while since I dove that deep in any Java stacks, so that's a new one to me, but good to know. > In C++, the equivalent files would come from descriptor.h which is shipped > with protobuf sources, and the implementation is provided by libprotobuf, > which you can build yourself of the sources. > > The plugin is going to be an executable (.EXE on Windows) file. Thanks, I appreciate the clarification. I'm not sure which download I saw, but it did not include sources, or I downloaded the minimal one. If memory serves there was a full source one, so I'll have a look there. Thanks! > On Tue, Nov 27, 2018 at 9:46 PM Michael Powell wrote: >> >> So... >> >> You've got a magic set of imports (in Java). >> >> import com.google.protobuf.DescriptorProtos; >> import com.google.protobuf.Descriptors; >> import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse; >> import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest; >> >> Where do you get these from? >> >> Or Assemblies or NuGet Package References in .NET / C#, or >> source/libraries, ostensibly, for C/C++. >> >> This is the question I have in getting started. >> >> The nearest assumption I could derive is that I start with protoc.exe >> (Windows) and the bundled .proto files, "compile" those to target >> plugin language, and build them with the project? >> >> Again, as a what? .DLL? .EXE? >> >> On Wed, Nov 28, 2018 at 12:31 AM Nadav Samet wrote: >> > >> > Hi Rahul and Michael, >> > >> > I put together a basic example of a protoc plugin in Java that generates >> > text files: https://github.com/thesamet/protoc-plugin-in-java >> > >> > The plugin itself is an executable that gets executed by protoc, it can be >> > written in any language. In this instance, the plugin is written in Java, >> > so we use gradle to generate a shell script (or bat file for windows) that >> > protoc will execute to run our plugin inside a JVM. The exact instructions >> > to run it are in the provided README.md >> > >> > >> > On Tue, Nov 27, 2018 at 6:01 AM Michael Powell >> > wrote: >> >> >> >> >> >> >> >> On Saturday, November 10, 2018 at 5:33:11 PM UTC-5, Nadav Samet wrote: >> >>> >> >>> Michael, >> >>> >> >>> A protoc plugin is a program that receives a serialized >> >>> CodeGeneratorRequest message via its stdin, and writes a >> >>> CodeGeneratorResponse to its stdout. >> >>> >> >>> You asked where the CodeGeneratorRequest comes from. The answer is that >> >>> the message is defined here. Assuming your plugin is named >> >>> protoc-gen-boilerplate, then when you invoke protoc like this: >> >>> protoc --boilerplate_out=outdir file1.proto [...] >> >>> >> >>> Then protoc would parse the proto files, prepare a CodeGeneratorRequest >> >>> and pipe it into your plugin. That request object contains a >> >>> representation of all the files you passed in the command line as well >> >>> as the files their transitively import. Your plugin should parse the >> >>> request and print out a serialized CodeGeneratorResponse. Unless you set >> >>> an error message, protoc would create the files you specified in your >> >>> response. The plugin can be written in any language, but if you use some >> >>> of the supported languages (at least C++, Python and Java, but probably >> >>> others), you'll find a method named something like >> >>> DescriptorPool.BuildFile which would convert a FileDescriptorProto to an >> >>> higher-level object (FileDescriptor) which is easier to work with. >> >>> >> >>> Plugins are pretty simple to write (look online for examples). I would >> >>> advise not writing a parser for proto files manually, since you'll be >> >>> chasing a moving target - over time language features get added albeit >> >>> not very often. It would also be hard to match the language spec >> >>> perfectly. >> >> >> >> >> >> I guess I am not sure quite where to begin. I've downloaded the protoc >> >> compiler, which itself has a set of proto f
Re: [protobuf] Re: Getting started developing a protoc plugin
So... You've got a magic set of imports (in Java). import com.google.protobuf.DescriptorProtos; import com.google.protobuf.Descriptors; import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse; import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest; Where do you get these from? Or Assemblies or NuGet Package References in .NET / C#, or source/libraries, ostensibly, for C/C++. This is the question I have in getting started. The nearest assumption I could derive is that I start with protoc.exe (Windows) and the bundled .proto files, "compile" those to target plugin language, and build them with the project? Again, as a what? .DLL? .EXE? On Wed, Nov 28, 2018 at 12:31 AM Nadav Samet wrote: > > Hi Rahul and Michael, > > I put together a basic example of a protoc plugin in Java that generates text > files: https://github.com/thesamet/protoc-plugin-in-java > > The plugin itself is an executable that gets executed by protoc, it can be > written in any language. In this instance, the plugin is written in Java, so > we use gradle to generate a shell script (or bat file for windows) that > protoc will execute to run our plugin inside a JVM. The exact instructions to > run it are in the provided README.md > > > On Tue, Nov 27, 2018 at 6:01 AM Michael Powell wrote: >> >> >> >> On Saturday, November 10, 2018 at 5:33:11 PM UTC-5, Nadav Samet wrote: >>> >>> Michael, >>> >>> A protoc plugin is a program that receives a serialized >>> CodeGeneratorRequest message via its stdin, and writes a >>> CodeGeneratorResponse to its stdout. >>> >>> You asked where the CodeGeneratorRequest comes from. The answer is that the >>> message is defined here. Assuming your plugin is named >>> protoc-gen-boilerplate, then when you invoke protoc like this: >>> protoc --boilerplate_out=outdir file1.proto [...] >>> >>> Then protoc would parse the proto files, prepare a CodeGeneratorRequest and >>> pipe it into your plugin. That request object contains a representation of >>> all the files you passed in the command line as well as the files their >>> transitively import. Your plugin should parse the request and print out a >>> serialized CodeGeneratorResponse. Unless you set an error message, protoc >>> would create the files you specified in your response. The plugin can be >>> written in any language, but if you use some of the supported languages (at >>> least C++, Python and Java, but probably others), you'll find a method >>> named something like DescriptorPool.BuildFile which would convert a >>> FileDescriptorProto to an higher-level object (FileDescriptor) which is >>> easier to work with. >>> >>> Plugins are pretty simple to write (look online for examples). I would >>> advise not writing a parser for proto files manually, since you'll be >>> chasing a moving target - over time language features get added albeit not >>> very often. It would also be hard to match the language spec perfectly. >> >> >> I guess I am not sure quite where to begin. I've downloaded the protoc >> compiler, which itself has a set of proto files, but no target language >> level generated code. >> >> Am I accurate in assuming I run those through protoc itself to my target >> language, in this case I suppose C++? >> >> On Windows, what form does the plugin need to take? DLL? Executable file? >> >> These are the sort of nibblets I think that would make it more evident where >> to pick up and run with it. >> >> Examples of this appear to be few and far between, if at all. >> >> Absent examples, I am working on a Boost Spirit Qi parser in the meantime, >> which the learning curve is not a plateau, I can agree there. It's not >> terrible to work through, and I end up with an AST that I know forwards and >> backwards without too much extra effort once the rules have done their >> synthetic magic. >> >> Cheers, thank you! >> >>> On Thursday, November 1, 2018 at 6:14:54 PM UTC-7, Michael Powell wrote: >>>> >>>> Hello, >>>> >>>> I'm a bit confused. I thought I was grasping it earlier, then I >>>> thought I wasn't ... What is the starting point developing a protoc >>>> plugin? >>>> >>>> What I want to accomplish is processing a .proto (v2) and generating >>>> some C# boilerplate code for it. I'm not generating a proto adapter >>>> itself, per se, but C# DSL that would be used to support it. >>>> >>
[protobuf] Re: Getting started developing a protoc plugin
On Saturday, November 10, 2018 at 5:33:11 PM UTC-5, Nadav Samet wrote: > > Michael, > > A protoc plugin is a program that receives a serialized > CodeGeneratorRequest message via its stdin, and writes a > CodeGeneratorResponse to its stdout. > > You asked where the CodeGeneratorRequest comes from. The answer is that > the message is defined here > <https://github.com/protocolbuffers/protobuf/blob/446e34ffc71ad458c21c60529cbcdc1de74e4c3d/src/google/protobuf/compiler/plugin.proto#L67>. > > Assuming your plugin is named protoc-gen-boilerplate, then when you invoke > protoc like this: > protoc --boilerplate_out=outdir file1.proto [...] > > Then protoc would parse the proto files, prepare a CodeGeneratorRequest > and pipe it into your plugin. That request object contains a representation > of all the files you passed in the command line as well as the files their > transitively import. Your plugin should parse the request and print out a > serialized CodeGeneratorResponse. Unless you set an error message, protoc > would create the files you specified in your response. The plugin can be > written in any language, but if you use some of the supported languages (at > least C++, Python and Java, but probably others), you'll find a method > named something like DescriptorPool.BuildFile which would convert a > FileDescriptorProto to an higher-level object (FileDescriptor) which is > easier to work with. > > Plugins are pretty simple to write (look online for examples). I would > advise not writing a parser for proto files manually, since you'll be > chasing a moving target - over time language features get added albeit not > very often. It would also be hard to match the language spec perfectly. > I guess I am not sure quite where to begin. I've downloaded the protoc compiler, which itself has a set of proto files, but no target language level generated code. Am I accurate in assuming I run those through protoc itself to my target language, in this case I suppose C++? On Windows, what form does the plugin need to take? DLL? Executable file? These are the sort of nibblets I think that would make it more evident where to pick up and run with it. Examples of this appear to be few and far between, if at all. Absent examples, I am working on a Boost Spirit Qi parser in the meantime, which the learning curve is not a plateau, I can agree there. It's not terrible to work through, and I end up with an AST that I know forwards and backwards without too much extra effort once the rules have done their synthetic magic. Cheers, thank you! On Thursday, November 1, 2018 at 6:14:54 PM UTC-7, Michael Powell wrote: > >> Hello, >> >> I'm a bit confused. I thought I was grasping it earlier, then I >> thought I wasn't ... What is the starting point developing a protoc >> plugin? >> >> What I want to accomplish is processing a .proto (v2) and generating >> some C# boilerplate code for it. I'm not generating a proto adapter >> itself, per se, but C# DSL that would be used to support it. >> >> The starting point for me would seem to be receiving the compiler >> request, but that itself is a .proto, which leaves me a bit confused. >> Where do you obtain that from? Or are you starting by compiling those >> .protos from the bits bundled with the protoc.exe compiler? >> >> What I want is to just have the descriptors drawn from the target >> .proto(s) themselves. That's it. A full-on plugin seems like a little >> bit of overkill. >> >> Suggestions? >> >> Thanks so much! >> >> Best regards, >> >> Michael Powell >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] String literals escaped content
Hello,
Maybe a stupid question on my part... But, is the thing being encoded
always a character? In C++ terms, char, or unsigned char? That is to
say, the range of valid values is more or less equal to
numeric_limits::min() ::max()? Or the same for
numeric_limits?
strLit = ( "'" { charValue } "'" ) | ( '"' { charValue } '"' )
charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/
hexEscape = '\' ( "x" | "X" ) hexDigit hexDigit
octEscape = '\' octalDigit octalDigit octalDigit
charEscape = '\' ( "a" | "b" | "f" | "n" | "r" | "t" | "v" | '\' | "'" | '"' )
quote = "'" | '"'
I think Octal can support through 512, whereas Hex supports through
256. But apart from that, basically the char or unsigned char limits?
Thanks!
Cheers,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Spec v2 int-lit snafu?
On Mon, Nov 12, 2018 at 12:46 PM Michael Powell wrote:
>
> On Mon, Nov 12, 2018 at 10:06 AM Michael Powell wrote:
> >
> > Hello,
> >
> > Another question following up, how about the sign character for hex
> > and oct integers? Is it necessary, should it be discarded?
> >
> > intLit = decimalLit | octalLit | hexLit
> > decimalLit = ( "1" … "9" ) { decimalDigit }
> > octalLit = "0" { octalDigit }
> > hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
> >
> > constant = fullIdent | ( [ "-" | "+" ] intLit ) | ( [ "-" | "+" ]
> > floatLit ) | strLit | boolLit
> >
> > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#integer_literals
> > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> >
> > For instance, I am fairly certain the sign character is encoded in a
> > hex encoded integer. Not sure about octal, but I imagine that it is
> > fairly consistent.
Got it sorted out I believe. Actually, it's quite nice the parser
support Spirit provides, aligns pretty much perfectly with the grammar
specification. There's a bit of gymnastics involved juggling whether
the AST has a sign or not and so forth, but other than that, it flows
well enough.
> > Case in point, the value 107026150751750362 gets encoded as
> > 0X17C3BB7913C48DA (upper-case). Whereas it's negative counterpart,
> > -107026150751750362, really does get encoded as 0xFE83C4486EC3B726.
> > Signage included, if memory serves.
> >
> > In these cases, I think the sign bit falls in the "optional" category?
>
> So... As far as I can determine, there are a couple of ways to
> interpret this, semantically speaking. But this potentially informs
> whatever parsing stack you are using as well.
>
> I'm using Boost Spirit Qi, for instance, which supports radix-based
> integer parsing well enough, but has its own set of issues when
> dealing with signage. That being said...
>
> 1. Treat the value itself as positive one way or another, with an
> optional sign attribute (i.e. '+' or '-'). This would potentially
> work, especially when there is base 16 (hex) or base 8 (octal)
> involved.
>
> 2. Otherwise, open to suggestions, but for Qi constraints; that I know
> of, fails to parse negative signed hexadecimal/octal encoded values.
>
> Again, kind of a symptom of an imprecise grammar specification. I can
> get a sense for how to handle it, but does it truly capture "intent".
>
> Thanks in advance for any light that can be shed.
>
> > Cheers, thanks,
> >
> > Michael
> > On Sun, Nov 11, 2018 at 10:56 AM Josh Humphries
> > wrote:
> > >
> > > For the case of zero by itself, per the spec, it will be parsed as an
> > > octal literal with value zero -- so functionally equivalent to a decimal
> > > literal with value zero. And for values with multiple digits, a leading
> > > zero means it is an octal literal. Decimal values will not have a leading
> > > zero.
> > >
> > >
> > > Josh Humphries
> > > jh...@bluegosling.com
> > >
> > >
> > > On Sat, Nov 10, 2018 at 10:16 PM Michael Powell
> > > wrote:
> > >>
> > >> Hello,
> > >>
> > >> I think 0 can be a decimal-lit, don't you think? However, the spec
> > >> reads as follows:
> > >>
> > >> intLit = decimalLit | octalLit | hexLit
> > >> decimalLit = ( "1" … "9" ) { decimalDigit }
> > >> octalLit = "0" { octalDigit }
> > >> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
> > >>
> > >> Is there a reason, semantically speaking, why decimal must be greater
> > >> than 0? And that's not including a plus/minus sign when you factor in
> > >> constants.
> > >>
> > >> Of course, parsing, order matters, similar as with the escape
> > >> character phrases in the string-literal:
> > >>
> > >> hex-lit | oct-lit | dec-lit
> > >>
> > >> And so on, since you have to rule out 0x\d+ for hex, followed by 0\d* ...
> > >>
> > >> Actually, now that I look at it "0" (really, "decimal" 0) is lurking
> > >> in the oct-lit phrase.
> > >>
> > >> Kind of a grammatical nit-pick, I know, but I just wanted to be clear
> > >> here. Seems like a possible source of confusion if you aren't
Re: [protobuf] Spec v2 int-lit snafu?
On Mon, Nov 12, 2018 at 10:06 AM Michael Powell wrote:
>
> Hello,
>
> Another question following up, how about the sign character for hex
> and oct integers? Is it necessary, should it be discarded?
>
> intLit = decimalLit | octalLit | hexLit
> decimalLit = ( "1" … "9" ) { decimalDigit }
> octalLit = "0" { octalDigit }
> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
>
> constant = fullIdent | ( [ "-" | "+" ] intLit ) | ( [ "-" | "+" ]
> floatLit ) | strLit | boolLit
>
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#integer_literals
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
>
> For instance, I am fairly certain the sign character is encoded in a
> hex encoded integer. Not sure about octal, but I imagine that it is
> fairly consistent.
>
> Case in point, the value 107026150751750362 gets encoded as
> 0X17C3BB7913C48DA (upper-case). Whereas it's negative counterpart,
> -107026150751750362, really does get encoded as 0xFE83C4486EC3B726.
> Signage included, if memory serves.
>
> In these cases, I think the sign bit falls in the "optional" category?
So... As far as I can determine, there are a couple of ways to
interpret this, semantically speaking. But this potentially informs
whatever parsing stack you are using as well.
I'm using Boost Spirit Qi, for instance, which supports radix-based
integer parsing well enough, but has its own set of issues when
dealing with signage. That being said...
1. Treat the value itself as positive one way or another, with an
optional sign attribute (i.e. '+' or '-'). This would potentially
work, especially when there is base 16 (hex) or base 8 (octal)
involved.
2. Otherwise, open to suggestions, but for Qi constraints; that I know
of, fails to parse negative signed hexadecimal/octal encoded values.
Again, kind of a symptom of an imprecise grammar specification. I can
get a sense for how to handle it, but does it truly capture "intent".
Thanks in advance for any light that can be shed.
> Cheers, thanks,
>
> Michael
> On Sun, Nov 11, 2018 at 10:56 AM Josh Humphries wrote:
> >
> > For the case of zero by itself, per the spec, it will be parsed as an octal
> > literal with value zero -- so functionally equivalent to a decimal literal
> > with value zero. And for values with multiple digits, a leading zero means
> > it is an octal literal. Decimal values will not have a leading zero.
> >
> >
> > Josh Humphries
> > jh...@bluegosling.com
> >
> >
> > On Sat, Nov 10, 2018 at 10:16 PM Michael Powell
> > wrote:
> >>
> >> Hello,
> >>
> >> I think 0 can be a decimal-lit, don't you think? However, the spec
> >> reads as follows:
> >>
> >> intLit = decimalLit | octalLit | hexLit
> >> decimalLit = ( "1" … "9" ) { decimalDigit }
> >> octalLit = "0" { octalDigit }
> >> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
> >>
> >> Is there a reason, semantically speaking, why decimal must be greater
> >> than 0? And that's not including a plus/minus sign when you factor in
> >> constants.
> >>
> >> Of course, parsing, order matters, similar as with the escape
> >> character phrases in the string-literal:
> >>
> >> hex-lit | oct-lit | dec-lit
> >>
> >> And so on, since you have to rule out 0x\d+ for hex, followed by 0\d* ...
> >>
> >> Actually, now that I look at it "0" (really, "decimal" 0) is lurking
> >> in the oct-lit phrase.
> >>
> >> Kind of a grammatical nit-pick, I know, but I just wanted to be clear
> >> here. Seems like a possible source of confusion if you aren't paying
> >> careful attention.
> >>
> >> Thoughts?
> >>
> >> Best regards,
> >>
> >> Michael Powell
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Protocol Buffers" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to protobuf+unsubscr...@googlegroups.com.
> >> To post to this group, send email to protobuf@googlegroups.com.
> >> Visit this group at https://groups.google.com/group/protobuf.
> >> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Spec v2 int-lit snafu?
Hello,
Another question following up, how about the sign character for hex
and oct integers? Is it necessary, should it be discarded?
intLit = decimalLit | octalLit | hexLit
decimalLit = ( "1" … "9" ) { decimalDigit }
octalLit = "0" { octalDigit }
hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
constant = fullIdent | ( [ "-" | "+" ] intLit ) | ( [ "-" | "+" ]
floatLit ) | strLit | boolLit
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#integer_literals
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
For instance, I am fairly certain the sign character is encoded in a
hex encoded integer. Not sure about octal, but I imagine that it is
fairly consistent.
Case in point, the value 107026150751750362 gets encoded as
0X17C3BB7913C48DA (upper-case). Whereas it's negative counterpart,
-107026150751750362, really does get encoded as 0xFE83C4486EC3B726.
Signage included, if memory serves.
In these cases, I think the sign bit falls in the "optional" category?
Cheers, thanks,
Michael
On Sun, Nov 11, 2018 at 10:56 AM Josh Humphries wrote:
>
> For the case of zero by itself, per the spec, it will be parsed as an octal
> literal with value zero -- so functionally equivalent to a decimal literal
> with value zero. And for values with multiple digits, a leading zero means it
> is an octal literal. Decimal values will not have a leading zero.
>
>
> Josh Humphries
> jh...@bluegosling.com
>
>
> On Sat, Nov 10, 2018 at 10:16 PM Michael Powell wrote:
>>
>> Hello,
>>
>> I think 0 can be a decimal-lit, don't you think? However, the spec
>> reads as follows:
>>
>> intLit = decimalLit | octalLit | hexLit
>> decimalLit = ( "1" … "9" ) { decimalDigit }
>> octalLit = "0" { octalDigit }
>> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
>>
>> Is there a reason, semantically speaking, why decimal must be greater
>> than 0? And that's not including a plus/minus sign when you factor in
>> constants.
>>
>> Of course, parsing, order matters, similar as with the escape
>> character phrases in the string-literal:
>>
>> hex-lit | oct-lit | dec-lit
>>
>> And so on, since you have to rule out 0x\d+ for hex, followed by 0\d* ...
>>
>> Actually, now that I look at it "0" (really, "decimal" 0) is lurking
>> in the oct-lit phrase.
>>
>> Kind of a grammatical nit-pick, I know, but I just wanted to be clear
>> here. Seems like a possible source of confusion if you aren't paying
>> careful attention.
>>
>> Thoughts?
>>
>> Best regards,
>>
>> Michael Powell
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Spec v2 int-lit snafu?
Hello,
I think 0 can be a decimal-lit, don't you think? However, the spec
reads as follows:
intLit = decimalLit | octalLit | hexLit
decimalLit = ( "1" … "9" ) { decimalDigit }
octalLit = "0" { octalDigit }
hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
Is there a reason, semantically speaking, why decimal must be greater
than 0? And that's not including a plus/minus sign when you factor in
constants.
Of course, parsing, order matters, similar as with the escape
character phrases in the string-literal:
hex-lit | oct-lit | dec-lit
And so on, since you have to rule out 0x\d+ for hex, followed by 0\d* ...
Actually, now that I look at it "0" (really, "decimal" 0) is lurking
in the oct-lit phrase.
Kind of a grammatical nit-pick, I know, but I just wanted to be clear
here. Seems like a possible source of confusion if you aren't paying
careful attention.
Thoughts?
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: String literal character set
On Fri, Nov 9, 2018 at 11:37 AM Michael Powell wrote: > > Hello, > > When considering the String Literal character set, there is the > regular expression /[^\0\n\\]/, meaning virtually any other character > than those specified in the expression. That's in addition to the > several or so alphanumeric alternatives preceding that alternative. > > As far as I can determine, that's virtually any character, some of > which are escaped in the string itself. Does this sound about right? Escape characters themselves are pretty easy to add as a range of characters. The troublesome cases IMO are how to discern between a hexEscape and an octEscape, at least as a character. Makes this somewhat of a unique "character set" issue. Not sure it's possible as such, but I'm open to suggestions. > I'm just trying to get a handle on how best to construct the character > set when building my parser test case generators. > > http://developers.google.com/protocol-buffers/docs/reference/proto2-spec > > Thanks! > > Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] String literal character set
Hello, When considering the String Literal character set, there is the regular expression /[^\0\n\\]/, meaning virtually any other character than those specified in the expression. That's in addition to the several or so alphanumeric alternatives preceding that alternative. As far as I can determine, that's virtually any character, some of which are escaped in the string itself. Does this sound about right? I'm just trying to get a handle on how best to construct the character set when building my parser test case generators. http://developers.google.com/protocol-buffers/docs/reference/proto2-spec Thanks! Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Import statement semantics
Hello, Is there some documentation concerning import semantics? For starters, I assume that you specify the path. This may be further enhanced by specification of an "import path", not unlike in C++ wherein you may specify include, library, etc, paths. Second, what is the impact of "weak"? "public"? Or an unspecified modifier? Thanks! Best regards, Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: [Proto2] Language spec help
On Friday, November 2, 2018 at 6:44:27 PM UTC-4, Michael Powell wrote: > > > > On Wednesday, October 31, 2018 at 1:17:46 PM UTC-4, Adam Cozzette wrote: >> >> I think that specification has suffered a little bit of neglect (sorry >> about that), because in practice our C++ parser is really the de facto >> standard and we have not recently made an effort to go through and make >> sure the official spec matches it perfectly. My reading of that string >> (/[^\0\n\\]/) is that it's a regular expression saying "any character other >> than null, newline, or backslash." But in general I would say the best bet >> is to resolve ambiguities by looking at what the C++ parser does. >> > > I have a bit of an AST problem, I wonder if you have some C++ insight to > help sort it out. If I translate the grammar on its face to an AST via the > parser rules, I end up with a MessageBody that must be forward declared, in > C++ parlance, for use in both Group and Message. Does that sound about > right? There are probably ways around that, such as introducing AST > pointers, things of this nature; a bit beyond the scope of this discussion, > but, assuming Boost Spirit would be able to handle that? (TBD via > Boost/Spirit forums.) > FYI, https://wandbox.org/permlink/WeRqkmDR93Wqu8BI > Provided my internal AST works, that's really all I need, to navigate it > and generate code given that. > > By the way, have you considered just reusing the C++ parser that's >> included in protoc? You can call protoc with the --descriptor_set_out flag >> to have it parse your .proto file and produce a serialized >> FileDescriptorSet proto as output. Then at that point it's easy to parse >> the descriptors using just about any language we support, and that should >> give you all the information you need, without the need for a new .proto >> file parser. >> > > I'm considering this as an approach, but I am a bit confused as to the > starting point. It seems like it all starts "compiling" the proto .protos > into the plugin implementation language, i.e. C++? Or perhaps I am missing > something? Also, do I need to have the source aligned with the protoc > version also readily available? i.e. with requisite libs, includes, etc? > Seems like possibly yes. > > Getting past the sort of "getting started" baby steps, it seems > straightforward enough to receive the compiler request, navigate the protoc > metadata API, and return a response. It's the getting past part that I'm > not exactly certain what the starting point really is. > > On Wed, Oct 31, 2018 at 9:23 AM Michael Powell wrote: >> >>> On Wed, Oct 31, 2018 at 12:22 PM Michael Powell >>> wrote: >>> > >>> > Concerning Constant, literally from the v2 spec: >>> >>> Rather, Syntax section, excuse me... >>> >>> > syntax = "syntax" "=" quote "proto2" quote ";" >>> > >>> > Do I read that correctly you can expect either 'proto2' or "proto2", >>> > but never 'proto2" nor "proto2' ? >>> > >>> > If accurate, that just seems to me to be lazy spec authorship... >>> > >>> > Thanks! >>> > On Wed, Oct 31, 2018 at 12:07 PM Michael Powell >>> wrote: >>> > > >>> > > Hello, >>> > > >>> > > I am writing a parser for the Proto language specification starting >>> > > with v2. I need a little help interpreting one of the lines if you >>> > > please: >>> > > >>> > > In the "String literals" section, what does this mean: >>> > > >>> > > charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/ >>> > > >>> > > Specifically, the trailing list of character soup? I want to say that >>> > > there are escaped characters in the sequence? Or am I to take that >>> > > string literally? Or notwithstanding the enclosing forward slashes? >>> > > >>> > > Thanks much in advance! >>> > > >>> > > Best regards, >>> > > >>> > > Michael Powell >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to protobuf+u...@googlegroups.com. >>> To post to this group, send email to prot...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Getting started developing a protoc plugin
Hello, I'm a bit confused. I thought I was grasping it earlier, then I thought I wasn't ... What is the starting point developing a protoc plugin? What I want to accomplish is processing a .proto (v2) and generating some C# boilerplate code for it. I'm not generating a proto adapter itself, per se, but C# DSL that would be used to support it. The starting point for me would seem to be receiving the compiler request, but that itself is a .proto, which leaves me a bit confused. Where do you obtain that from? Or are you starting by compiling those .protos from the bits bundled with the protoc.exe compiler? What I want is to just have the descriptors drawn from the target .proto(s) themselves. That's it. A full-on plugin seems like a little bit of overkill. Suggestions? Thanks so much! Best regards, Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Proto2 possible field options
On Thu, Nov 1, 2018 at 11:08 AM Josh Humphries wrote:
>
> Michael,
> They map to field names in the options messages defined in descriptor.proto.
This is pretty interesting. So they've used the Proto spec to describe
itself, ey?
> Also, if you define custom options (by extending those options messages), you
> refer to them using qualified name enclosed in parentheses. That proto file
> above also serves as a sort of AST for proto source files, where
> FileDescriptorProto is the root of the AST. You can then look at
> UninterpretedOption for seeing the AST structure for parsing options.
So, for instance, my field options could include something like this:
... [default_value = 3, oneof_index = 1, json_name = "myField", ...]
And, so on, ad nauseam for the possible field options?
https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L136
> Josh Humphries
> jh...@bluegosling.com
>
>
> On Wed, Oct 31, 2018 at 7:30 PM Michael Powell wrote:
>>
>> Hello,
>>
>> In the Proto2 specification, "field options" are sort of meta-tucked
>> away. I don't think the specification really names them. Is there a
>> list of them somewhere in the docs?
>>
>> i.e. field = label type fieldName "=" fieldNumber [ "[" fieldOptions "]" ]
>> ";"
>> fieldOptions = fieldOption { "," fieldOption }
>> fieldOption = optionName "=" constant
>>
>> Specifically, "optionName" ? If memory serves from examples, things
>> like "default" ?
>>
>> Thanks!
>>
>> Best regards,
>>
>> Michael Powell
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Proto2 possible field options
Hello,
In the Proto2 specification, "field options" are sort of meta-tucked
away. I don't think the specification really names them. Is there a
list of them somewhere in the docs?
i.e. field = label type fieldName "=" fieldNumber [ "[" fieldOptions "]" ] ";"
fieldOptions = fieldOption { "," fieldOption }
fieldOption = optionName "=" constant
Specifically, "optionName" ? If memory serves from examples, things
like "default" ?
Thanks!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: [Proto2] Language spec help
On Wed, Oct 31, 2018 at 1:17 PM Adam Cozzette wrote: > > I think that specification has suffered a little bit of neglect (sorry about > that), because in practice our C++ parser is really the de facto standard and > we have not recently made an effort to go through and make sure the official > spec matches it perfectly. My reading of that string (/[^\0\n\\]/) is that > it's a regular expression saying "any character other than null, newline, or > backslash." But in general I would say the best bet is to resolve ambiguities > by looking at what the C++ parser does. Thanks for that bit of clarification. > By the way, have you considered just reusing the C++ parser that's included > in protoc? You can call protoc with the --descriptor_set_out flag to have it > parse your .proto file and produce a serialized FileDescriptorSet proto as > output. Then at that point it's easy to parse the descriptors using just > about any language we support, and that should give you all the information > you need, without the need for a new .proto file parser. Good point, it's a possibility? Does it protoc to C#? It would be less difficult, I think, for me to Reflect through that and generate the boilerplate that I want, probably, than spinning up a full on parser replete with its AST. > On Wed, Oct 31, 2018 at 9:23 AM Michael Powell wrote: >> >> On Wed, Oct 31, 2018 at 12:22 PM Michael Powell >> wrote: >> > >> > Concerning Constant, literally from the v2 spec: >> >> Rather, Syntax section, excuse me... >> >> > syntax = "syntax" "=" quote "proto2" quote ";" >> > >> > Do I read that correctly you can expect either 'proto2' or "proto2", >> > but never 'proto2" nor "proto2' ? >> > >> > If accurate, that just seems to me to be lazy spec authorship... >> > >> > Thanks! >> > On Wed, Oct 31, 2018 at 12:07 PM Michael Powell >> > wrote: >> > > >> > > Hello, >> > > >> > > I am writing a parser for the Proto language specification starting >> > > with v2. I need a little help interpreting one of the lines if you >> > > please: >> > > >> > > In the "String literals" section, what does this mean: >> > > >> > > charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/ >> > > >> > > Specifically, the trailing list of character soup? I want to say that >> > > there are escaped characters in the sequence? Or am I to take that >> > > string literally? Or notwithstanding the enclosing forward slashes? >> > > >> > > Thanks much in advance! >> > > >> > > Best regards, >> > > >> > > Michael Powell >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+unsubscr...@googlegroups.com. >> To post to this group, send email to protobuf@googlegroups.com. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: [Proto2] Language spec help
On Wed, Oct 31, 2018 at 12:22 PM Michael Powell wrote: > > Concerning Constant, literally from the v2 spec: Rather, Syntax section, excuse me... > syntax = "syntax" "=" quote "proto2" quote ";" > > Do I read that correctly you can expect either 'proto2' or "proto2", > but never 'proto2" nor "proto2' ? > > If accurate, that just seems to me to be lazy spec authorship... > > Thanks! > On Wed, Oct 31, 2018 at 12:07 PM Michael Powell wrote: > > > > Hello, > > > > I am writing a parser for the Proto language specification starting > > with v2. I need a little help interpreting one of the lines if you > > please: > > > > In the "String literals" section, what does this mean: > > > > charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/ > > > > Specifically, the trailing list of character soup? I want to say that > > there are escaped characters in the sequence? Or am I to take that > > string literally? Or notwithstanding the enclosing forward slashes? > > > > Thanks much in advance! > > > > Best regards, > > > > Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: [Proto2] Language spec help
Concerning Constant, literally from the v2 spec: syntax = "syntax" "=" quote "proto2" quote ";" Do I read that correctly you can expect either 'proto2' or "proto2", but never 'proto2" nor "proto2' ? If accurate, that just seems to me to be lazy spec authorship... Thanks! On Wed, Oct 31, 2018 at 12:07 PM Michael Powell wrote: > > Hello, > > I am writing a parser for the Proto language specification starting > with v2. I need a little help interpreting one of the lines if you > please: > > In the "String literals" section, what does this mean: > > charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/ > > Specifically, the trailing list of character soup? I want to say that > there are escaped characters in the sequence? Or am I to take that > string literally? Or notwithstanding the enclosing forward slashes? > > Thanks much in advance! > > Best regards, > > Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] [Proto2] Language spec help
Hello, I am writing a parser for the Proto language specification starting with v2. I need a little help interpreting one of the lines if you please: In the "String literals" section, what does this mean: charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/ Specifically, the trailing list of character soup? I want to say that there are escaped characters in the sequence? Or am I to take that string literally? Or notwithstanding the enclosing forward slashes? Thanks much in advance! Best regards, Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] [C# .NET] How to parse .proto file and generate some boilerplate code
On Mon, Oct 29, 2018 at 9:35 PM Michael Powell wrote: > > Hello, > > I'm a frequent participant in the OR-tools group, but here I have a Protobuf > specific question, I think. > > I'm looking for a way to parse a .proto file and generate some really > boilerplate code from it. I might use a JSON or Xml sister file to further > inform the generator, but the core bits of it are focused on the .proto file. > > From a C#, or any other language, perspective, am I accurate in saying the > Protobuf API is the way to do just this? > > If possibly, I'm also wanting to generate at compile time using either Roslyn > code generation, a custom build task, or what have you. The only rub there > would be whether it fits with the Roslyn/MSBuild work flow. Short of leveraging the Protobuf API, I'm thinking of a Boost Spirit, or similar, approach. But for this, I need to get my hands on the *.proto language grammar itself. > Thanks in advance! > > Best regards, > > Michael Powell > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Protocol Buffers" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/protobuf/pueWYTikzxY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] [C# .NET] How to parse .proto file and generate some boilerplate code
Hello, I'm a frequent participant in the OR-tools group, but here I have a Protobuf specific question, I think. I'm looking for a way to parse a .proto file and generate some really boilerplate code from it. I might use a JSON or Xml sister file to further inform the generator, but the core bits of it are focused on the .proto file. >From a C#, or any other language, perspective, am I accurate in saying the Protobuf API is the way to do just this? If possibly, I'm also wanting to generate at compile time using either Roslyn code generation, a custom build task, or what have you. The only rub there would be whether it fits with the Roslyn/MSBuild work flow. Thanks in advance! Best regards, Michael Powell -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
