Re: [protobuf] How to model {key:value, key:value, ...} object in the ProtoBuf

[Josh Humphries]

The map field didn't work because it would be an object *inside* the reply
-- like so:
{
  "balances": {
"somekey": "somevalue"
  }
}

To make the top-level object an arbitrary map, it would need to be a
google.protobuf.Struct
<https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/struct.proto#L43-L54>.
This type is basically analogous to a JSON object with arbitrary
keys/properties.

The downside to that type is that it is not type-safe: the values can be
*any* type. So you'd need to do something special in the consuming code to
check that all values are of an expected type (e.g. string).


*Josh Humphries*
jh...@bluegosling.com


On Thu, Nov 24, 2022 at 12:28 PM MantasS  wrote:

> I'm trying to model this JSON into ProtoBuff reply:
> {
>  "0xc02aaa39b223fe8d4a0e5c4f27ead9083c756cc2": "73185315951404654",
> "0x1117dc0aa78b770fa6a738034120c302": "90483392540197705773",
> "0xdac17f958d2ee523a2206206944597c13d831ec7": "2599292593",
>  }
>
> Tried to use map:
>
> message Reply { map balances = 1; }
> but ended up with this error message:
> Object literal may only specify known properties, and
> '0xc02aaa39b223fe8d4a0e5c4f27ead9083c756cc2' does not exist in type 'Reply'.
>
> Is there a way to return this type of JSON?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/efabc47a-0c6d-4bd6-a152-0fc29a7a60c6n%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/efabc47a-0c6d-4bd6-a152-0fc29a7a60c6n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAFGBEr25TQXDXPky4iktKJkud7ZqHU5iRyQVq%2B_3HQ30C11QGg%40mail.gmail.com.

Re: [protobuf] Re: How to: get the parent message

[Josh Humphries]

No official runtimes provide a way to do this. Messages do not have
backlinks to containers if they are nested inside another message.

To do what you are trying to do would require a recursive traversal that
starts at the top (a) and then accumulates the names into a path with each
recursive call. Something like so:

func handle(msg b, path []string) {
currentPath := append(path, msg.name)

// ... do something with msg and currentPath ...

if msg.b != nil {
handle(msg.b, currentPath)
}
}


*Josh Humphries*
jh...@bluegosling.com


On Tue, Nov 22, 2022 at 8:46 AM Fred Eisele 
wrote:

> Given a message (b of type B), nested in another (a of type A); how does
> one get ‘a’?
> I was hoping for something like ‘b.getParent()’.
>
> message B { optional string name = 1;  repeated B b = 2; }
> message A { optional string name = 1;  repeated B b = 1; }
>
> Here is an example instance of an ‘a’ with textual serialization.
>
> name: "a"
> b { name: “foo”
>   b { name: “fred”
> b { name: “flintstone” }}}
> b { name: “bar” }
> b { name: “baz” }
>
> The issue is that I am navigating the collection 'a' with a visitor and I
> need to be able to reconstruct a full name composed of names of all the
> ancestors.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/9ba69d16-218b-4cde-b0c9-faa651a46167n%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/9ba69d16-218b-4cde-b0c9-faa651a46167n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAFGBEr19p2S7G1_CO0uTL5UNiGK-B29xkZqdHvLagzLA%3DvLc-Q%40mail.gmail.com.

Re: [protobuf] proto3 pretty printer

[Josh Humphries]

At FullStory, we use clang-format to format protobuf code. And we have a
custom linter for catching some other style issues, outside of just
formatting.

The clang-format tool doesn't have a ton of options for protos, but it's
been good enough for our purposes. It can get confused when you have
complicated custom options -- like message options, where you put a long
message literal as the value -- which can lead to less-than-stellar
results, aesthetically speaking.

But having a standard with some warts seemed better than no standard...


*Josh Humphries*
jh...@bluegosling.com


On Fri, Dec 10, 2021 at 11:08 AM Jason Polis  wrote:

> Is there a pretty printer that will format proto3 consistently
> making it easier to compare diffs over the long term ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/a5807c8b-159a-41b9-a702-1c6398b9edbdn%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/a5807c8b-159a-41b9-a702-1c6398b9edbdn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAO78j%2BL%2BFZvPdRepMPMk2GRN5Tk3Ozfzkt0S9Nx%3DPACtRBefdg%40mail.gmail.com.

Re: [protobuf] Re: Google Cloud Security - Redact Keyword?

[Josh Humphries]

There is no built-in way to do this.

However, you can create your own custom options and use those to annotate
sensitive fields/messages whose contents should be redacted.

Related: https://github.com/protocolbuffers/protobuf/issues/1160


*Josh Humphries*
jh...@bluegosling.com


On Thu, Oct 28, 2021 at 3:27 PM 'Mike Vacirca' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Pinging again on this item - is this the right forum to ask this type of
> question?
>
> On Tue, Oct 19, 2021 at 10:20 AM Mike Vacirca  wrote:
>
>> Hello,
>>
>> I was curious if there was ever discussion on extending Protocol Buffers
>> to include coverage for redaction of a data field?
>>
>> Specifically, we are looking at the use case of a developer having a
>> keyword available to them that would allow data classification services,
>> logging services, et. al. to be able to read a particular data field as
>> requiring redaction due to core customer content (or other data
>> classification state) and filtering out the contents during
>> archival/transmission/etc.
>>
>> Or if this would be the wrong approach, are there areas that may be the
>> right technical approach to "shift left" the annotation of parameters
>> earlier in the cycle during coding?
>>
>> We are looking at ways to improve the current workflow in GCP which
>> requires us to  track and annotate data between RPC services in production
>> using manual annotation from engineers post-development.
>>
>
>
> --
>
> *Michael Vacirca*
>
> 718-938-9424
>
> Senior Engineering Manager, Regulated Cloud
>
> Google Cloud
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/CAPvXYq1eGgNpf2KF%3D68wO4mLMwwRHMGe-%3DzE4fq4yCzTLOkGXA%40mail.gmail.com
> <https://groups.google.com/d/msgid/protobuf/CAPvXYq1eGgNpf2KF%3D68wO4mLMwwRHMGe-%3DzE4fq4yCzTLOkGXA%40mail.gmail.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAO78j%2BLtWBNXRbbAdnRD_-mJHJeENk3-i%2B7-HEMXey-Y0zf%3D9g%40mail.gmail.com.

Re: [protobuf] Feature Request: make protocol buffer AST/parser SDK

[Josh Humphries]

If you are using Go, you can use this protoparse package to parse proto
source:
https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse

It provides functions for parsing and linking into descriptors. But it also
provides an option to parse into a richer AST. The descriptor format is not
a particularly good AST because it is lossy. So if you wanted to write a
formatter, for example, the descriptor loses a lot of token position and
comment information, which may prevent you from perfectly reconstructing
the original source text. The ast sub-package of protoparse is not lossy
and includes position information, including whitespace and comments, for
everything lexed token.
https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse/ast


*Josh Humphries*
jh...@bluegosling.com


On Tue, Feb 16, 2021 at 7:34 PM Gerardo Mora  wrote:

> I'm gonna use golang for my project, and will be open source, so
> eventually I will show you the software if you guys are interested!
> I would like that the parser will run in the same thread of the request,
> when a microservice registers itself.
>
> I'm gonna read all approaches and see, thanks guys!
>
> PS
> if there are more ideas all are welcomed.
>
>
> El mar, 16 de feb. de 2021 a la(s) 18:28, Derek Perez (p...@google.com)
> escribió:
>
>> It sounds like you may wanna build a protoc plugin like so:
>>
>> https://expobrain.net/2015/09/13/create-a-plugin-for-google-protocol-buffer/
>>
>> The AST for protobuf is actually in protobuf itself! It's very easy to
>> work with, and is the backbone of all the code generators.
>>
>> On Tue, Feb 16, 2021 at 4:24 PM Marc Gravell 
>> wrote:
>>
>>> You can use protoc to output the compiled schema (one of the
>>> command-line-options - something "file descriptor set"), and deserialize
>>> the resultant binary file as a FileDescriptorSet instance, deserializing
>>> via your choice of language via descriptor.proto - any use?
>>>
>>> As an aside, I also have a separate implementation of a .proto parser,
>>> implemented in C#/.NET (so it can be used in 100% managed .NET code). Not
>>> sure if that is any use to you, but if so: let me know.
>>>
>>> On Wed, 17 Feb 2021, 00:12 Gerardo Mora,  wrote:
>>>
>>>> Hello everyone, I would like to create an application that analyzes a
>>>> .proto file and dynamically creates GraphQL queries, I don't want to parse
>>>> the .proto files because it will outdated and probably buggy at some point,
>>>> but I was reading the source code and the only available thing is create a
>>>> plugin that ouputs a file after the processing which is not idea.
>>>>
>>>> Is there an API/SDK available for it?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/protobuf/f78681b8-47f2-44ac-955a-73b828f1b08dn%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/protobuf/f78681b8-47f2-44ac-955a-73b828f1b08dn%40googlegroups.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/CAF95VAz3nWGiqsHRQNaEBgWN5VvnXtvP-W6SsjekDHhzFt6Ctg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/protobuf/CAF95VAz3nWGiqsHRQNaEBgWN5VvnXtvP-W6SsjekDHhzFt6Ctg%40mail.gmail.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>
> --
> May the source be with you!!!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/CAP88TbchGtuKaek38TU5JOByJBm7rTX-fr9TLtf%3DM_WoDYMbAg%40mail.gmail.com
> <https://groups.google.com/d/msgid/protobuf/CAP88TbchGtuKaek38TU5JOByJBm7rTX-fr9TLtf%3DM_WoDYMbAg%40mail.gmail.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAO78j%2BL%3D%2BafvkuT_6630WPPPka2ikH3YDaVzUc7GBmsr0oCu5Q%40mail.gmail.com.

Re: [protobuf] [Proposal]: special case fieldmask behaviour for struct.proto types

[Josh Humphries]

+1, this feels like the only sane way to handle field masks for those two
types.

I wonder if it might make sense for this special-casing to also apply to
google.protobuf.Any. It is very similar to google.protobuf.Struct in its
JSON form. I know the runtime and binary forms make it less practical to
handle this way (since the embedded type must be known by the masker and
the embedded bytes must be unmarshaled, masked, then re-marshaled). But it
would be strictly more useful if it could work this way.


*Josh Humphries*
jh...@bluegosling.com


On Fri, Oct 23, 2020 at 6:42 AM Johan Brandhorst 
wrote:

>
> Hi!
>
> I'm one of the maintainers of the grpc-gateway
> <https://github.com/grpc-ecosystem/grpc-gateway>, a popular third-party
> generator for Protobuf. As part of our runtime implementation, we have an
> automatic JSON-body-to-fieldmask-generator
> <https://github.com/grpc-ecosystem/grpc-gateway/blob/68878db0032aae8413e99ba833fcaa8568a28182/runtime/fieldmask.go#L23>.
> It was written before the availability of the official Go WKT helpers
> <https://pkg.go.dev/google.golang.org/protobuf@v1.25.0/types/known/fieldmaskpb>
> for the google.protobuf.FieldMask message type, and so has a custom idea
> of what constitutes a valid field mask path, based on the input JSON
> structure. In a recent contribution
> <https://github.com/grpc-ecosystem/grpc-gateway/pull/1595>, we added
> support for creating a field mask from the dynamic struct.proto types.
> For example:
>
> Given a field named struct_field with a type google.protobuf.Struct and
> the incoming JSON:
>
> {"struct_field": {"name":{"first": "bob"}, "amount": 2}}
>
> We create the FieldMask with paths:
>
> ["struct_field.name.first", "struct_field.amount"]
>
> This is apparently not considered a valid field mask path according to the
> spec in field_mask.proto
> <https://github.com/protocolbuffers/protobuf/blob/de5d1b98c27428450b9a38ab5c2de479f59025af/src/google/protobuf/field_mask.proto#L43-L241>
> .
>
> My proposal is to extend what is considered a valid FieldMask to include
> this special case for the struct.proto types google.protobuf.Struct and
> google.protobuf.Value *only*.
>
> My rationale for this is that special casing well known types is a widely
> practiced in the Protobuf community, and I think this field mask
> representation of these types will make sense to users of these types,
> since they're for use with fundamentally dynamic data.
>
> For example, in JavaScript the google.protobuf.Struct type is represented
> by an Object, for which such a field mask would make sense. For other
> languages that don't have a special in-language representation for these
> types, field masks are generally useless already since they can't currently
> be used to reference anything consistently, because of the discrepancy of
> the implementation with other languages. We could generalise the meaning of
> a field mask for these types to be that which makes sense for the *intent*
> of these types.
>
> I originally raised this idea with the Go protobuf maintainers
> <https://github.com/golang/protobuf/issues/1227> but it was agreed that
> it requires a wider discussion.
>
> Thanks for reading,
>
> Johan Brandhorst
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/380bae2f-d41b-4545-a1f7-57eb89a7962fn%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/380bae2f-d41b-4545-a1f7-57eb89a7962fn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAO78j%2BKDhxkGfVW6DOALVXXHO-CBxNyCvW-Ae%3DJBiP7JjdkLpA%40mail.gmail.com.

Re: [protobuf] protoc generates value instead of pointer (Go)

[Josh Humphries]

The simplest answer is to use syntax="proto2" for that. In proto3, it is
intentional that you cannot distinguish between absence and zero value for
scalar fields.

However, a new "proto3 optional
<https://github.com/protocolbuffers/protobuf/blob/master/docs/implementing_proto3_presence.md>"
feature is now available that allows this. It allows you to add the
"optional" keyword to a field, where you might otherwise have specified
"repeated". The feature is in a kind of beta. You must be using 3.12 of
protoc or newer and you must enable the feature via a flag:
--experimental_allow_proto3_optional. You also need to be using a very
recent protoc-gen-go (v1.22 or newer from the google.golang.org/protobuf import
path, or v1.4.1 or newer from github.com/golang/protobuf).


*Josh Humphries*
jh...@bluegosling.com


On Thu, Jun 18, 2020 at 11:30 AM Klim Sidorov  wrote:

> Hello!
>
> *The question is:*
> How can I make protoc generate a pointer to the value?
>
> *The problem:*
> I have 2 files in the root directory.
> I use Makefile to generate Go code from .proto files.
>
> But the *language* field in the Video struct is a value not a pointer to
> the value.
> And the *subtitles* field in the Video struct is an array of values not
> an array of pointers to the value.
>
> *video.pb.go*
>
> type Video struct {
> state protoimpl.MessageState
> sizeCache protoimpl.SizeCache
> unknownFields protoimpl.UnknownFields
>
> Idstring  `protobuf:"bytes,1,opt,name=id,proto3" 
> json:"id,omitempty"`
> Title string  `protobuf:"bytes,2,opt,name=title,proto3" 
> json:"title,omitempty"`
> Languages language.ISO639_1   
> `protobuf:"varint,3,opt,name=languages,proto3,enum=language.ISO639_1" 
> json:"languages,omitempty"`
> Subtitles []language.ISO639_1 
> `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=language.ISO639_1" 
> json:"subtitles,omitempty"`
> }
>
>
>
> *Makefile*
>
> gen:
># Video
>protoc -I. --go_out=plugins=grpc,paths=source_relative:video video.proto
>
># Language
>protoc -I. --go_out=plugins=grpc,paths=source_relative:language 
> language.proto
>
>
>
> *language.proto*
>
> syntax = "proto3";
>
> package language;
>
> option go_package = "example.com/group/repo/language;language";
>
> enum ISO639_1 {
> UNKNOWN = 0;
> zh  = 1;
> }
>
>
>
> *video.proto*
>
>
> syntax = "proto3";
>
> package video;
>
> import "language.proto";
>
> option go_package = "example.com/group/repo/video;video";
>
> message Video {
>  stringid= 1;
>  stringtitle = 2;
>  language.ISO639_1 language  = 3;
> repeated language.ISO639_1 subtitles = 4;
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/eac045ed-aceb-401c-a6ba-1de876c7bfdbo%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/eac045ed-aceb-401c-a6ba-1de876c7bfdbo%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAO78j%2B%2B_jsp-x7LmkkoapKfYEa_C1x8SaY0LDxT8pvQ2ip_MFA%40mail.gmail.com.

Re: [protobuf] Proposal: a mechanism to deal with sensitive/redacted fields in string output

[Josh Humphries]

I don't think there has been any movement on this, but I'd like to ping the
thread again.

I am still a strong proponent of standard metadata in the proto source and
descriptor to indicate sensitive things. I also think it's a truly wise
idea to use that information to trigger auto-redaction of sensitive data
when serializing message in certain contexts (such as "stringification",
like when emitting the data into a debug log message).

At FullStory, we've built some interesting tools for using protobufs and
gRPC. Most recent is a web UI for gRPC:
https://github.com/fullstorydev/grpcui. I would *love* to be able to add
features to some of these tools that can be aware of sensitive data in
messages that they handle and even give the data special treatment in some
ways. For the web UI, there's a few interesting things that a client might
choose to do in the face of sensitive data. For one, if that page includes
an analytics/recording tool, it could make sure the elements in the DOM
that may contain sensitive data are excluded from analysis/recording.
Another: it could refuse to send an RPC over an insecure connection if any
of the message's fields are marked as sensitive.

Outside of the web UI, especially in a world with GDPR, there are all kinds
of tools that could be built that care deeply about whether data is
sensitive or not. For example, static analysis/linters that make sure
sensitive data is treated in a suitably sensitive manner. Without a
standard way to denote that in the proto, open-source tools in this vein
become more complicated to use. If open-source tools must provide their own
custom option (as an example), and then someone wants to use multiple such
tools on their codebase, they end up having to redundantly define multiple
options on each sensitive field. (Admittedly, this is a manufactured
hypothetical.)

Anyhow, where do things stand with this enhancement request? Is there
anything I can do to stoke the fire and get some attention on it?

----
*Josh Humphries*
jh...@bluegosling.com


On Wed, Aug 22, 2018 at 10:01 AM Zellyn  wrote:

> Apologies for the long delay, but I got radically reassigned at work, so I
> haven't had much time to work on this. But it keeps niggling at me, because
> I hate our internal protobuf forks so much.
>
> Here is the proposal: Proto Proposal: a “sensitive” field option
> <https://docs.google.com/document/d/18WI8zN7rk6R0jXW1iC8LDYz7LJ0OrUOTKMGD7nyEnFs>
> .
>
> Zellyn
>
> On Wednesday, February 22, 2017 at 12:10:14 PM UTC-5, Adam Cozzette wrote:
>>
>> Hi Zellyn, this sounds like a reasonable idea. As the next step could you
>> perhaps write up a short proposal with more details on what exactly it
>> would mean for a field to be redacted? To me it seems like the important
>> thing would be to make sure it's clear how redacted fields are supposed to
>> be behave in each situation (i.e. when they should be dropped or not), so
>> that there's no uncertainty about when they're dropped and when they're
>> preserved. (For example, we might say that they're never shown when a proto
>> is implicitly stringified but maybe preserved in all other situations?) We
>> might also need to be careful to get this right for all languages early;
>> even if there's some language where we don't care about redaction for now,
>> it will be hard to change later without making a breaking change.
>>
>> On Thu, Feb 16, 2017 at 1:45 PM, zellyn via Protocol Buffers <
>> prot...@googlegroups.com> wrote:
>>
>>> There are many ways that protocol buffers might be stringified into
>>> logs, accidentally or on purpose, printed in stack traces, etc. The
>>> built-in behavior stringifies the entire protobuf recursively, including
>>> all field data.
>>>
>>> At Square, we deal with payments, and often have data of varying
>>> sensitivity in protobuf fields, which we'd like to be elided from
>>> stringified output.
>>>
>>> We use an internal fork of protoc to handle a custom field option,
>>> "redacted", and have also patched the stringification code to print
>>> "[REDACTED]" for those fields. We do the same in Go, and in the C
>>> implementation (for Ruby).
>>>
>>> Last year, we chatted with the protobuf team, and they were sympathetic
>>> to our use case (in fact, they mentioned that the part of Google that deals
>>> with payments has something similar internally: I think that's where the
>>> "sensitive" name came from). I'd like to get that discussion rolling again.
>>>
>>> We'd like to see one of the following happen (in decreasing order of
>>> awesomeness for us):
>>>
>>>- upstreaming of the "redacted&q

Re: [protobuf] Guidance for proto linkage

[Josh Humphries]

On Tue, Mar 12, 2019 at 7:31 PM Michael Powell 
wrote:

>
>
> On Monday, March 11, 2019 at 12:24:57 PM UTC-4, Josh Humphries wrote:
>>
>> Since I've implemented this before, I have a fairly lengthy list.
>>
>> Most of these constraints are in the docs, at least in some
>> way/shape/form. But some are enforced by protoc but never actually
>> mentioned in the docs (such as disallowing the use of "map_entry" and
>> "uninterpreted_options" options):
>>
>
> So, this may sound like a stupid question, but where do you even go to
> discover heuristics such as these? Well, besides this super comprehensive
> list, that is.
>

I go to the source: in this case protoc (and sometimes its C source code).

Many I discovered because I was writing a parser/compiler in Go. I realized
that these were areas where the spec was light and the comments in
descriptor.proto weren't crystal clear. So I just tried a few things out to
observe the behavior of protoc.


>
> For example, it just seems like these are so arbitrary. I'm sure there is
> a reason why, but in and of itself, "just because" does not seem like an
> adequate response:
>
>
>- If the file indicates syntax = "proto2":
>   - Fields and extensions must specify a label: "optional",
>   "repeated", or "required" (excluding those in oneof declarations and
>   excluding map fields, which must *not* have a label).
>   - Extension fields must *not* use the "required" label.
>
>
> Are these heuristics internal bits? Or are they really coming from Google
> Protocol Buffers?
>

I'm not sure I understand the question. I think the answer is that they
come from Google Protocol Buffers.

Regarding labels, this is a difference in the actual language
<https://developers.google.com/protocol-buffers/docs/reference/proto2-spec>
specs
<https://developers.google.com/protocol-buffers/docs/reference/proto3-spec>.
I was writing a single parser that could support either, so instead of
implementing two different specs, I merged the two. And the label thing
shook out as a difference between the two specs.

For extensions not being required, it's not explicitly documented. But
given that extensions are not meant to even be known by all clients, it's
intuitive that they can't be required. It's also hinted at in the docs, where
it is stated
<https://developers.google.com/protocol-buffers/docs/proto#updating> that
non-required fields can safely be converted to extensions and vice versa.


>
>
>>
>>- Tags for all fields of a given message must be valid:
>>   - Must be in the range 1 to 536,870,911 (2^29-1) inclusive.
>>   - Must *not* be in the reserved range 19,000 to 19,999 (inclusive).
>>   - All fields must have a unique tag (i.e. no tag re-use allowed).
>>   - Tags from reserved ranges defined in the message are not allowed.
>>   - Tags from extension ranges defined in the message are not
>>   allowed for normal fields. Similarly, extension fields *must* use
>>   a tag in one of the message's extension ranges.
>>- Other message properties must be valid:
>>   - No field may be named using a reserved name.
>>   - Any given reserved range or extension range must not overlap
>>   with any other reserved range or extension range defined in the 
>> message.
>>   - Reserved names may not contain duplicates.
>>   - No message is allowed to use the "map_entry" option. (This
>>   option is used in representing a message as a descriptor, in which a
>>   message descriptor is synthesized for every map field. Only those 
>> synthetic
>>   messages may have this option.)
>>   - A field whose type refers to a message must not have a "default"
>>   option.
>>   - Map fields and repeated fields also must not have "default"
>>   options.
>>- Numeric values for all enum values must be valid:
>>   - Numeric values must be in range for signed 32-bit integers
>>   (-4,294,967,296 to 4,294,967,295).
>>   - Numeric values for a single enum must be unique and may not be
>>   reused *unless* the enum includes the option allow_alias set to
>>   true.
>>   - Numeric values from reserved ranges defined in the enum are not
>>   allowed.
>>- Other enum properties must be valid:
>>   - No value may be named using a reserved name.
>>   - Any given reserved range must not overlap with any other
>>   reserved range defined in the enum.
>>   - Reserved names may not contain duplicate

Re: [protobuf] Guidance for proto linkage

[Josh Humphries]

 is a message named MessageTwo.
 - "(custom.option).foo.bar" refers to a field named "bar" in
 MessageTwo.
 - References to repeated fields may not have values that appear to
  be list literals. Instead, a single option that references a
repeated field
  defines a single element of the repeated option. Subsequent options that
  reference the same repeated field define subsequent elements.
  - Only the *leaf* field named can be a repeated field. If earlier
 path elements name repeated fields, the option is invalid.
Instead, the
 option statement must refer only to the *first* repeated field in
 the path and then use an aggregate value (which then includes
all values
 for nested repeated fields).
  - References to message fields may use an aggregate value: an
  aggregate value is enclosed in curly braces "{ }" and uses the protobuf
  text format therein to define the message value.
  - Option values whose type is an enum must use unqualified
  identifiers. The identifier must be one of the values in the enum.
   - No option statement may refer to an option named
   "uninterpreted_option". (This field of the various option types is used
   internally by protoc and other parsers to represent
   unresolved/uninterpreted option statements.)
   - If the file indicates syntax = "proto3":
  - No "optional" or "required" labels are allowed on any field
  definition. (Fields are always optional in proto3, unless they have the
  "repeated" label or are map fields.)
  - No messages may define extension ranges.
  - No messages may define groups.
  - No messages may include a field whose type is an enum defined in a
  file with syntax = "proto2".
  - The first value for an enum *must* have a numeric value of zero.
  (Similarly: every enum must have a value whose numeric value is zero.)
  - Field definitions may not use the "default" option (default field
  values in proto3 are always the zero value for the field's type).
   - If the file indicates syntax = "proto2":
  - Fields and extensions must specify a label: "optional", "repeated",
  or "required" (excluding those in oneof declarations and excluding map
  fields, which must *not* have a label).
  - Extension fields must *not* use the "required" label.




*Josh Humphries*
jh...@bluegosling.com


On Sat, Mar 9, 2019 at 2:03 PM Michael Powell  wrote:

> Hello,
>
> I am looking for guidance as far as what steps one must take in order to
> verify that a Proto is valid, link the names, i.e. considering ambiguities
> of Element (i.e. Message or Enum) Type Names during parsing, etc.
>
> Some things seem obvious, such as the Field Numbers must be unique, that
> sort of thing, but a more comprehensive set of guidelines would be helpful.
>
> Cheers, thank you.
>
> Michael W Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Java Date object to be added in .proto file

[Josh Humphries]

There are the well-known types
<https://developers.google.com/protocol-buffers/docs/reference/google.protobuf>,
which include google.protobuf.Timestamp:

syntax = "proto3";
import "google/protobuf/timestamp.proto";

message Product {
google.protobuf.Timestamp last_ordered_date = 1;
}


Sadly, the documentation for each type (in the link above) does not tell
you the import path to use it. All of them are in the google.protobuf
package and the files are all imported from google/protobuf/*.proto. These
files are included with protoc, so you can find their sources in your
protobuf installation and also in the repo
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/timestamp.proto>
.



*Josh Humphries*
jh...@bluegosling.com


On Fri, Jan 18, 2019 at 12:27 PM Shilpa Vittal  wrote:

> HI,
>
> I have a Date java object that I want to add in .proto file. I use Proto3.
> Is there any exisitng proto Datatype that I can use?
> Please advise at the earliest.
>
> Ex:
>
> message Product{
>
> optional date last_ordered_date = 1;
> }
>
> where last_ordered_date='2019-01-15'
>
> Thanks,
> Shilpa
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Question about json_name and protoc

[Josh Humphries]

On Wed, Jan 16, 2019 at 12:56 PM Zellyn  wrote:

> Yeah, it's precisely the fact that protoc populates that field regardless
> of whether it was specified that I consider bad.
> https://github.com/protocolbuffers/protobuf/issues/5587
>

Doh, I apparently misread your original email, thinking you were talking
about the generated Go code only.

FWIW, I think most decoders (for the runtimes that support JSON encoding)
support either: they first look for a matching name per JSON name, and, if
no field is found, they fall back to looking for a matching proto name.
Perhaps that may help if you are trying to, for eaxmple, bridge a pre-3.0
custom JSON encoding with proto3 canonical JSON encoding (just kind of
guessing at your possible reason for needing this distinction). If that is
the motivation, you might also watch out for the custom JSON encoding for
well-known types and how one-ofs are handled (e.g. one-of names never
appear in JSON; their constituents are flattened into the enclosing
message).


>
> I'm probably way too late to get it fixed, though: it looks like 3.1.0
> introduced that behavior :-(
>
> Zellyn
>
> On Wednesday, January 16, 2019 at 12:23:17 PM UTC-5, Josh Humphries wrote:
>>
>> Hey, Zellyn,
>> I don't think it's possible for protoc-gen-go to do what you're asking.
>> IIRC, protoc itself always populates this field in the field descriptor. So
>> I don't think a plugin can actually make this distinction -- at least not
>> without a change to protoc. And the downside to a change to protoc is that
>> it would require all code-gen plugins (and/or runtime libraries that
>> implement the JSON encoding) to have redundant logic for computing the
>> canonical JSON name from the proto name.
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>>
>> On Wed, Jan 16, 2019 at 10:10 AM Zellyn  wrote:
>>
>>> A quick followup: a reason that including a json name for every field in
>>> the descriptor is bad: it makes it impossible to tell whether an explicit
>>> json_name was specified or not. So it would be impossible to give jsonpb a
>>> mode (which it needs, imho) where it uses the protobuf field names as-is,
>>> unless an explicit json_name was specified.
>>>
>>> Or am I understanding it wrong?
>>>
>>> Zellyn
>>>
>>>
>>> On Tuesday, January 15, 2019 at 12:47:31 PM UTC-5, Zellyn wrote:
>>>>
>>>> Hey folks,
>>>>
>>>> We're trying to update the version of protoc we use for compiling our
>>>> Go protobufs. It appears that at some point between 3.0.0 and 3.6.0, protoc
>>>> started populating the `json_name` field option for *every* field,
>>>> whether the option was specified or not. Is that expected? If so, could
>>>> anyone help me track down the commit that did it?
>>>>
>>>> Thanks,
>>>>
>>>> Zellyn
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Question about json_name and protoc

[Josh Humphries]

Hey, Zellyn,
I don't think it's possible for protoc-gen-go to do what you're asking.
IIRC, protoc itself always populates this field in the field descriptor. So
I don't think a plugin can actually make this distinction -- at least not
without a change to protoc. And the downside to a change to protoc is that
it would require all code-gen plugins (and/or runtime libraries that
implement the JSON encoding) to have redundant logic for computing the
canonical JSON name from the proto name.


*Josh Humphries*
jh...@bluegosling.com


On Wed, Jan 16, 2019 at 10:10 AM Zellyn  wrote:

> A quick followup: a reason that including a json name for every field in
> the descriptor is bad: it makes it impossible to tell whether an explicit
> json_name was specified or not. So it would be impossible to give jsonpb a
> mode (which it needs, imho) where it uses the protobuf field names as-is,
> unless an explicit json_name was specified.
>
> Or am I understanding it wrong?
>
> Zellyn
>
>
> On Tuesday, January 15, 2019 at 12:47:31 PM UTC-5, Zellyn wrote:
>>
>> Hey folks,
>>
>> We're trying to update the version of protoc we use for compiling our Go
>> protobufs. It appears that at some point between 3.0.0 and 3.6.0, protoc
>> started populating the `json_name` field option for *every* field,
>> whether the option was specified or not. Is that expected? If so, could
>> anyone help me track down the commit that did it?
>>
>> Thanks,
>>
>> Zellyn
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: packed = true encoding in protobuf

[Josh Humphries]

Looks like you create a new person in the loop, so each person has only a
single ID. As Thomas mentioned, if you don't actually have multiple values,
the framing is just overhead.

With a single value, the encoding basically looks like so:

   - *With packed = false*: varint encoded tag, varint encoded value
   - *With packed = true*: varint encoded tag, varint encoded length value
   == 1, varint encoded value

But with multiple IDs per person, it will be smaller. With 10 values, the
encoding basically looks like so:

   - *With packed = false*: (varint encoded tag, varint encoded value) x 10
   - *With packed = true*: varint encoded tag, varint encoded length value
   == 10, (varint encoded value) x 10

The break-even point is ~2 values in the repeated field, since both packed
and unpacked would use 4 varints to represent the pair of values (though
packed may possibly be 1 byte smaller since varint encoded length of 2 will
be only one byte, where as extra varint encoded tag could possibly be >1
byte, depending on the tag value).

----
*Josh Humphries*
jh...@bluegosling.com


On Mon, Dec 10, 2018 at 4:03 AM  wrote:

> The code i am using is below:
>
> #include 
> #include 
> #include 
> #include "addressbook.pb.h"
>
> #define FILENAME "ashu.pb"
>
> using namespace std;
>
> AddressBook address_book;
>
> int getFileSize()
> {
> ifstream file(FILENAME, ifstream::in | ifstream::binary);
>
> if(!file.is_open())
> {
> return -1;
> }
>
> file.seekg(0, ios::end);
> int fileSize = file.tellg();
> cout << "Size - " << fileSize << endl;
> file.close();
>
> return fileSize;
> }
>
> int main()
> {
> int x,size;
> Person *p = address_book.add_people();
>
> Person::PhoneNumber* phone_number = p->add_phones();
>
> fstream out(FILENAME, ios::out | ios::binary | ios::trunc);
>
> for(int j=0;j<5000;j++)
> {
> p = address_book.add_people();
>
> p->add_id(j);
> }
>
> x = address_book.SerializeToOstream();
> out.close();
>
> size = getFileSize();
> cout << "size " << size << endl;
>
> AddressBook ad1;
> fstream in(FILENAME, ios::in | ios::binary);
> ad1.ParseFromIstream();
>
> return 0;
> }
>
> and i am using only id field of message Person. Is that the reason why i
> end up getting more size with packed.
>
> On Friday, December 7, 2018 at 12:12:04 PM UTC+5:30, ajinsi...@gmail.com
> wrote:
>>
>> I am trying to find out difference in encoded size using packed equal to
>> true and without any packed encoding. I looped a int32 variable 5000 times.
>> With packed = true, i get size as 29876 and without packed encoding, i get
>> size as 24876. I am confused why i get opposite results. As per my
>> understanding, size with packed should be less as compared to non packed.
>> Can anyone please explain what the issue is?
>> My proto file is below for non packed and for packed i modify id filed
>> as; repeated int32 id = 2[packed=true]:
>>
>> syntax = "proto2";
>> message Person {
>>   optional string name = 1;
>>   repeated int32 id = 2;
>>   optional string email = 3;
>>
>>   enum PhoneType {
>> MOBILE = 0;
>> HOME = 1;
>> WORK = 2;
>>   }
>>
>>   message PhoneNumber {
>> optional string number = 1;
>> optional PhoneType type = 2 [default = HOME];
>>   }
>>
>>   repeated PhoneNumber phones = 4;
>> }
>>
>> message AddressBook {
>>   repeated Person people = 1;
>> }
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Import statement semantics

[Josh Humphries]

The "public" and "weak" import modifiers are in no way similar to anything
in C++.

A "public" import means that all imported symbols are "transitively"
imported into dependees. It's a little hard to describe, but easier to
understand with an example:

Let's say we have three files.

   1. foo.proto: This file defines message foo.Foo.
   2. bar.proto: This file defines message bar.Bar and imports *public*
"foo.proto".
   3. baz.proto: This imports "bar.proto".

Normally, the import in baz.proto (bullet 3) means that the contents of the
baz.proto file can freely refer to symbols defined in the imported file --
namely, bar.Bar.

However, because the imported file *has public imports (*namely,
foo.proto), it's as if baz.proto imported foo.proto, too. So not only can
baz.proto use the symbols defined in its imports (e.g. bar.proto), *but
also* all of the symbols that are *publicly* imported therein. So, in the
example, the contents of baz.proto (bullet 3) can refer to the symbol
foo.Foo *without * actually importing foo.proto, thanks to the public
import in bar.proto.

Does that make sense? It's like the "public" keyword takes all symbols from
the imported file and adds them to its own set of exported symbols.

There is a brief bit of doc on the main website for public imports. Perhaps
it explains it better than me:
https://developers.google.com/protocol-buffers/docs/proto#importing-definitions


A "weak" import is very different. A weak import is one whose contents are
only *weakly* required. I think the actual implementation varies from
language to language (IIRC, some languages do not even support weak
imports, treating them instead as normal imports). I believe the idea is
that generated code will not have a strong compile-time dependency on the
generated code/library for the imported file. IIUC, fields whose types are
defined in weakly imported files will be treated like unknown fields if the
runtime type is not available (e.g. not linked in).

Despite this being documented
<https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#DescriptorPool.EnforceWeakDependencies.details>
as if supported in the C++ library, I think it's intentionally
<https://github.com/protocolbuffers/protobuf/blob/b61dd9d9a229c5a82788fdeefa344aa65affae09/src/google/protobuf/descriptor.proto#L70>
 not
<https://github.com/protocolbuffers/protobuf/blob/b61dd9d9a229c5a82788fdeefa344aa65affae09/src/google/protobuf/descriptor.proto#L587>
supported
<https://github.com/protocolbuffers/protobuf/blob/2f28980f07eb0840854cb4338b2df6007a2c56bb/src/google/protobuf/compiler/cpp/cpp_file.cc#L1198>
in the open-source library (along with lazily-parsed
<https://github.com/protocolbuffers/protobuf/blob/b61dd9d9a229c5a82788fdeefa344aa65affae09/src/google/protobuf/descriptor.proto#L579>
fields
<https://github.com/protocolbuffers/protobuf/blob/2f28980f07eb0840854cb4338b2df6007a2c56bb/src/google/protobuf/compiler/cpp/cpp_file.cc#L1202>).
So it's part of the grammar, but perhaps disallowed in practice (or, more
likely, ignored and treated as a normal import).


*Josh Humphries*
jh...@bluegosling.com


On Tue, Dec 4, 2018 at 5:00 PM Michael Powell  wrote:

> On Wed, Nov 7, 2018 at 4:28 PM Michael Powell 
> wrote:
> >
> > Hello,
> >
> > Is there some documentation concerning import semantics?
> >
> > For starters, I assume that you specify the path. This may be further
> > enhanced by specification of an "import path", not unlike in C++
> > wherein you may specify include, library, etc, paths.
>
> I have not received any clarification for this. Can someone please
> clarify? I want to think in terms of C++ access modifiers, private,
> protected, public, etc, but I'm not positive this is analogous, or
> what the ramifications ought to be when "linking" across .proto files.
>
>
> http://developers.google.com/protocol-buffers/docs/reference/proto2-spec#import_statement
>
> > Second, what is the impact of "weak"? "public"? Or an unspecified
> modifier?
> >
> > Thanks!
> >
> > Best regards,
> >
> > Michael Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto v2 ambiguities in parsed identifiers

[Josh Humphries]

You know it's a message vs. enum (or invalid identifier) only after linking.

To link, you need the file's entire transitive closure (e.g. its imports
and their imports and so on) already parsed (and, preferably, already
linked). You can then *resolve* the type name, basically using C++ scoping
rules. The leading dot tells it that the identifier is fully-qualified --
so you don't need to worry about scoping rules (which allows for
partially-qualified or unqualified names) but instead search for the symbol
by full name. After you have resolved the name to an element, you then know
the element's type -- whether it was a message or an enum (or if it was
invalid, such as no matching element or referring to a service or method
instead of a type).

An element's full name is the file package (if any), all enclosing element
names (note that in an "extend" block, the extendee is *not* considered an
element name: extend blocks have no name), and then item itself.

Example:

syntax = "proto3";
package foo.bar;
import "google/protobuf/descriptor.proto";

extend google.protobuf.MessageOptions {
  string some_custom_option = ;* // fqn = foo.bar.some_custom_option*
}

enum SomeEnum {* // fqn = foo.bar.SomeEnum*
  VAL0 = 0;* // fqn = foo.bar.SomeEnum.VAL0*
}

message SomeMsg {* // fqn = foo.bar.SomeMsg*
  string name = 1;* // fqn = foo.bar.SomeMsg.name
<http://foo.bar.SomeMsg.name>*

  *// refers to top-level SomeEnum above*
  .foo.bar.SomeEnum enum2 = 3; *// fqn = foo.bar.SomeMsg.enum2*

  *// refers to nested SomeEnum below*
  SomeEnum enum = 2; *// fqn = foo.bar.SomeMsg.enum*

  enum SomeEnum { *// fqn = foo.bar.SomeMsg.SomeEnum*
VAL0 = 0; *// fqn = foo.bar.SomeMsg.SomeEnum.VAL0*
  }

*  // In this scope, can refer to SomeEnum w/out qualifier, as SomeEnum.*
*  // Could also use partial-qualifiers, like SomeMsg.SomeEnum,
bar.SomeMsg.SomeEnum,*
*  // or foo.bar.SomeMsg.SomeEnum. **That last one is actually
fully-qualified, but protoc*
*  // doesn't know that, so it must use same scope rules to resolve symbol
name. However,*
*  // using .foo.bar.SomeMsg.SomeEnum (leading dot) means the current
lexical scope*
*  // doesn't matter, so symbol resolution can take "short cut".*
*  //*
*  // As you can see, compound package names work like** nested C++
namespaces, not like*
*  // Java or .NET package names.*

  message NestedMsg { *// fqn = foo.bar.SomeMsg.NestedMsg*
extends google.protobuf.FieldOption {
  uint64 some_other_option = ; *// fqn =
foo.bar.SomeMsg.NestedMsg.some_other_option*
}
  }
}


*Josh Humphries*
jh...@bluegosling.com


On Thu, Nov 29, 2018 at 9:14 AM Michael Powell 
wrote:

> Hello,
>
> In the Protocol Buffer v2 Field specification there is a notion of
> messageType and enumType. Casually reading the specification, it is
> easy to miss this, but there is nothing there to tell anyone whether a
> messageType or an enumType was encountered, except perhaps a second
> pass verification.
>
> Which brings me to the second question, how does one interpret the
> identifiers involved in the type specification? Also, what's with the
> leading dot? Please explain. Thanks!
>
> Also, minor question, is the trailing semi-colon a typo? i.e.
> "fieldNumber = intLit;". I gather that perhaps it is a typo but I
> wanted to clarify that as well.
>
> // Identifiers
> ident = letter { letter | decimalDigit | "_" }
> // ...
> messageName = ident
> enumName = ident
> // ...
> messageType = [ "." ] { ident "." } messageName
> enumType = [ "." ] { ident "." } enumName
> // ...
> // Field
> label = "required" | "optional" | "repeated"
> type = "double" | "float" | "int32" | "int64" | "uint32" | "uint64" |
> "sint32" | "sint64" | "fixed32" | "fixed64" | "sfixed32" | "sfixed64"
> | "bool" | "string" | "bytes" | messageType | enumType
> fieldNumber = intLit; // <- is the trailing semicolon here a typo?
>
> http://developers.google.com/protocol-buffers/docs/reference/proto2-spec
>
> Best,
>
> Michael Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Spec v2 int-lit snafu?

[Josh Humphries]

On Mon, Nov 12, 2018 at 10:30 PM Michael Powell 
wrote:

> On Mon, Nov 12, 2018 at 12:46 PM Michael Powell 
> wrote:
> >
> > On Mon, Nov 12, 2018 at 10:06 AM Michael Powell 
> wrote:
> > >
> > > Hello,
> > >
> > > Another question following up, how about the sign character for hex
> > > and oct integers? Is it necessary, should it be discarded?
> > >
> > > intLit = decimalLit | octalLit | hexLit
> > > decimalLit = ( "1" … "9" ) { decimalDigit }
> > > octalLit   = "0" { octalDigit }
> > > hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
> > >
> > > constant = fullIdent | ( [ "-" | "+" ] intLit ) | ( [ "-" | "+" ]
> > > floatLit ) | strLit | boolLit
> > >
> > >
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#integer_literals
> > >
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> > >
> > > For instance, I am fairly certain the sign character is encoded in a
> > > hex encoded integer. Not sure about octal, but I imagine that it is
> > > fairly consistent.
>
> Got it sorted out I believe. Actually, it's quite nice the parser
> support Spirit provides, aligns pretty much perfectly with the grammar
> specification. There's a bit of gymnastics involved juggling whether
> the AST has a sign or not and so forth, but other than that, it flows
> well enough.
>

If you haven't already, take a look at descriptor.proto
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto>
-- FileDescriptorProto
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L61>
therein is basically like an AST for the proto language (and is what protoc
produces as it parses). And for parsing options and the literal values in
particular, take a look at UninterpretedOption
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L701>.
Options are first parsed into this structure, and then "interpreted" into
the attributes of *Options messages in a second pass. You'll see that the
approach there includes the negation in the literal integer value but
also distinguishes
between the two
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L716>
in the AST.


>
> > > Case in point, the value 107026150751750362 gets encoded as
> > > 0X17C3BB7913C48DA (upper-case). Whereas it's negative counterpart,
> > > -107026150751750362, really does get encoded as 0xFE83C4486EC3B726.
> > > Signage included, if memory serves.
> > >
> > > In these cases, I think the sign bit falls in the "optional" category?
> >
> > So... As far as I can determine, there are a couple of ways to
> > interpret this, semantically speaking. But this potentially informs
> > whatever parsing stack you are using as well.
> >
> > I'm using Boost Spirit Qi, for instance, which supports radix-based
> > integer parsing well enough, but has its own set of issues when
> > dealing with signage. That being said...
> >
> > 1. Treat the value itself as positive one way or another, with an
> > optional sign attribute (i.e. '+' or '-'). This would potentially
> > work, especially when there is base 16 (hex) or base 8 (octal)
> > involved.
> >
> > 2. Otherwise, open to suggestions, but for Qi constraints; that I know
> > of, fails to parse negative signed hexadecimal/octal encoded values.
> >
> > Again, kind of a symptom of an imprecise grammar specification. I can
> > get a sense for how to handle it, but does it truly capture "intent".
> >
> > Thanks in advance for any light that can be shed.
> >
> > > Cheers, thanks,
> > >
> > > Michael
> > > On Sun, Nov 11, 2018 at 10:56 AM Josh Humphries 
> wrote:
> > > >
> > > > For the case of zero by itself, per the spec, it will be parsed as
> an octal literal with value zero -- so functionally equivalent to a decimal
> literal with value zero. And for values with multiple digits, a leading
> zero means it is an octal literal. Decimal values will not have a leading
> zero.
> > > >
> > > > 
> > > > Josh Humphries
> > > > jh...@bluegosling.com
> > > >
> > > >
> > > > On Sat, Nov 10, 2018 at 10:16 PM Michael Powell <
> mwpowell...@gmail.com> wrote:
> > > >>
> > > >> Hello,
> &g

Re: [protobuf] Spec v2 int-lit snafu?

[Josh Humphries]

For the case of zero by itself, per the spec, it will be parsed as an octal
literal with value zero -- so functionally equivalent to a decimal literal
with value zero. And for values with multiple digits, a leading zero means
it is an octal literal. Decimal values will not have a leading zero.


*Josh Humphries*
jh...@bluegosling.com


On Sat, Nov 10, 2018 at 10:16 PM Michael Powell 
wrote:

> Hello,
>
> I think 0 can be a decimal-lit, don't you think? However, the spec
> reads as follows:
>
> intLit = decimalLit | octalLit | hexLit
> decimalLit = ( "1" … "9" ) { decimalDigit }
> octalLit   = "0" { octalDigit }
> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
>
> Is there a reason, semantically speaking, why decimal must be greater
> than 0? And that's not including a plus/minus sign when you factor in
> constants.
>
> Of course, parsing, order matters, similar as with the escape
> character phrases in the string-literal:
>
> hex-lit | oct-lit | dec-lit
>
> And so on, since you have to rule out 0x\d+ for hex, followed by 0\d* ...
>
> Actually, now that I look at it "0" (really, "decimal" 0) is lurking
> in the oct-lit phrase.
>
> Kind of a grammatical nit-pick, I know, but I just wanted to be clear
> here. Seems like a possible source of confusion if you aren't paying
> careful attention.
>
> Thoughts?
>
> Best regards,
>
> Michael Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto2 possible field options

[Josh Humphries]

On Thu, Nov 1, 2018 at 11:36 AM Michael Powell 
wrote:

> On Thu, Nov 1, 2018 at 11:08 AM Josh Humphries 
> wrote:
> >
> > Michael,
> > They map to field names in the options messages defined in
> descriptor.proto.
>
> This is pretty interesting. So they've used the Proto spec to describe
> itself, ey?
>
> > Also, if you define custom options (by extending those options
> messages), you refer to them using qualified name enclosed in parentheses.
> That proto file above also serves as a sort of AST for proto source files,
> where FileDescriptorProto is the root of the AST. You can then look at
> UninterpretedOption for seeing the AST structure for parsing options.
>
> So, for instance, my field options could include something like this:
>
> ... [default_value = 3, oneof_index = 1, json_name = "myField", ...]
>
> And, so on, ad nauseam for the possible field options?
>
>
> https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L136


Not quite. Those are fields on the *descriptor*, which map to other
elements in the source. (The oneof index, for example, is computed based on
the order of oneof declarations in the enclosing DescriptorProto and is
unset for fields that are not enclosed in a oneof in the source file. You
can't define that via an option.)

The options are all defined in special *Options messages, like this one:
https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L507
The various descriptor protos have a field for the corresponding options
type (for example, this
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L213>
).

The two exceptions are both for field options only: default and json_name.
Contrary to what I just told you, these are not fields of the FieldOptions
message, but they are defined via options in the actual source. I call them
"pseudo-options" -- they behave just like options, but when incorporated
into the descriptor proto, they get *promoted* out of the options message
and into the enclosing FieldDescriptorProto.


>
>
> > Josh Humphries
> > jh...@bluegosling.com
> >
> >
> > On Wed, Oct 31, 2018 at 7:30 PM Michael Powell 
> wrote:
> >>
> >> Hello,
> >>
> >> In the Proto2 specification, "field options" are sort of meta-tucked
> >> away. I don't think the specification really names them. Is there a
> >> list of them somewhere in the docs?
> >>
> >> i.e. field = label type fieldName "=" fieldNumber [ "[" fieldOptions
> "]" ] ";"
> >> fieldOptions = fieldOption { ","  fieldOption }
> >> fieldOption = optionName "=" constant
> >>
> >> Specifically, "optionName" ? If memory serves from examples, things
> >> like "default" ?
> >>
> >> Thanks!
> >>
> >> Best regards,
> >>
> >> Michael Powell
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+unsubscr...@googlegroups.com.
> >> To post to this group, send email to protobuf@googlegroups.com.
> >> Visit this group at https://groups.google.com/group/protobuf.
> >> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: [Proto2] Language spec help

[Josh Humphries]

Re: syntax: It really means string literal, whose value must be "proto2" or
"proto3".

Looks like it's been cleaned up, but not long ago it also had an incorrect
definition for the service syntax, suggesting an alternate way to define
streaming methods that protoc did not actually support.

Anyhow, I've gone through the exercise of writing a parser in Go
<https://godoc.org/github.com/jhump/protoreflect/desc/protoparse> and
learned a lot. The parser I wrote leverages yacc, which may provide a
useful starting point for creating a parser in C#:
https://github.com/jhump/protoreflect/blob/master/desc/protoparse/proto.y

I know the code in the productions is all Go, but the layout of the grammar
is more or less right. (I have tests that load a variety of proto files,
with a variety of language features therein, and makes sure that my package
produces equivalent descriptors to protoc.)


*Josh Humphries*
jh...@bluegosling.com


On Wed, Oct 31, 2018 at 12:23 PM Michael Powell 
wrote:

> On Wed, Oct 31, 2018 at 12:22 PM Michael Powell 
> wrote:
> >
> > Concerning Constant, literally from the v2 spec:
>
> Rather, Syntax section, excuse me...
>
> > syntax = "syntax" "=" quote "proto2" quote ";"
> >
> > Do I read that correctly you can expect either 'proto2' or "proto2",
> > but never 'proto2" nor "proto2' ?
> >
> > If accurate, that just seems to me to be lazy spec authorship...
> >
> > Thanks!
> > On Wed, Oct 31, 2018 at 12:07 PM Michael Powell 
> wrote:
> > >
> > > Hello,
> > >
> > > I am writing a parser for the Proto language specification starting
> > > with v2. I need a little help interpreting one of the lines if you
> > > please:
> > >
> > > In the "String literals" section, what does this mean:
> > >
> > > charValue = hexEscape | octEscape | charEscape | /[^\0\n\\]/
> > >
> > > Specifically, the trailing list of character soup? I want to say that
> > > there are escaped characters in the sequence? Or am I to take that
> > > string literally? Or notwithstanding the enclosing forward slashes?
> > >
> > > Thanks much in advance!
> > >
> > > Best regards,
> > >
> > > Michael Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto2 possible field options

[Josh Humphries]

Michael,
They map to field names in the options messages defined in descriptor.proto
<https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L286>
.

Also, if you define custom options (by extending those options messages),
you refer to them using qualified name enclosed in parentheses. That proto
file above also serves as a sort of AST for proto source files, where
FileDescriptorProto is the root of the AST. You can then look at
UninterpretedOption for seeing the AST structure for parsing options.


*Josh Humphries*
jh...@bluegosling.com


On Wed, Oct 31, 2018 at 7:30 PM Michael Powell 
wrote:

> Hello,
>
> In the Proto2 specification, "field options" are sort of meta-tucked
> away. I don't think the specification really names them. Is there a
> list of them somewhere in the docs?
>
> i.e. field = label type fieldName "=" fieldNumber [ "[" fieldOptions "]" ]
> ";"
> fieldOptions = fieldOption { ","  fieldOption }
> fieldOption = optionName "=" constant
>
> Specifically, "optionName" ? If memory serves from examples, things
> like "default" ?
>
> Thanks!
>
> Best regards,
>
> Michael Powell
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Embedding arbitrary JSON in a message

[Josh Humphries]

If you are using JavaScript, there may not be a good solution. Ironically,
the JavaScript implementation of protos does not support the canonical JSON
representation for protobuf types (which would be needed to convert
arbitrary JSON into a google.protobuf.Value).

Most other languages, though, should work fine. Here's a sketch in Go:

*example.proto*
syntax = "proto3";
import "google/protobuf/struct.proto";

message SomeMessage {
google.protobuf.Value arbitrary_json = 1;
}

*example.go*
package example

import "github.com/golang/protobuf/ptypes/struct"
import "github.com/golang/protobuf/jsonpb"

func example(json string) (*SomeMessage, error) {
msg := {ArbitraryJson: {}}
jsm := jsonpb.Marshaler{}
if err := jsm.Unmarshal(json, msg.ArbitraryJson); err != nil {
return nil, err
}
return msg, nil
}

You could also use the "encoding/json" package to then convert any kind of
Go value into a google.protobuf.Value: you just have to marshal to JSON
first, and from there into the protobuf. While using JSON as intermediate
format is not efficient, you do get the benefit that a
google.protobuf.Value created this way is guaranteed to be well-formed,
whereas just using a string or bytes field could be garbage that is not
proper JSON.


*Josh Humphries*
jh...@bluegosling.com


On Wed, Sep 26, 2018 at 9:44 AM Jimit Modi  wrote:

> Hello @Siddharth, @John,
>
> Did you guys found any solution around it. We are also stuck at very same
> thing.
>
> On Friday, August 17, 2018 at 3:36:29 PM UTC+5:30, Siddharth Kherada wrote:
>>
>> Can I please get an example of how to do this is any language with proto
>> file?
>>
>> I desperately need help with this. I am stuck for a week now on this
>> issue.
>>
>> Thanks,
>> Sid
>>
>> On Thursday, 21 June 2018 15:32:46 UTC-4, Josh Humphries wrote:
>>>
>>> Oops, I meant to point you to google.protobuf.Value:
>>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L63
>>>
>>> It can represent *any* kind of JSON value. The Struct type is what is
>>> used to represent JSON *objects* (there is also ListValue, for arrays,
>>> as well as support for JS primitive types and null).
>>>
>>>
>>> 
>>> *Josh Humphries*
>>> jh...@bluegosling.com
>>>
>>> On Thu, Jun 21, 2018 at 3:25 PM, Josh Humphries 
>>> wrote:
>>>
>>>> Hi, John,
>>>> Take a look at the well-known type google.protobuf.Struct. It is
>>>> basically a JSON value, modeled as a proto. It's JSON representation is
>>>> exactly what you want, too:
>>>>
>>>>
>>>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L52
>>>>
>>>>
>>>>
>>>> 
>>>> *Josh Humphries*
>>>> jh...@bluegosling.com
>>>>
>>>> On Thu, Jun 21, 2018 at 3:20 PM, John Lilley  wrote:
>>>>
>>>>> Disclaimer: I am totally new to protobuf, engaged in an exploratory
>>>>> POC.  Please forgive dumb questions :-)
>>>>>
>>>>> We are looking at migrating an existing, JSON-based protocol in which
>>>>> hand-coded C++ is written to perform serdes between objects and JSON. We
>>>>> want to replace the hand-coding with an automated approach that can be
>>>>> shared between C++ and Java.  However, a stumbling block I see is that 
>>>>> some
>>>>> messages have an arbitrary field full of JSON like:
>>>>>
>>>>> {
>>>>>"name":"john",
>>>>>"address":"123 main st",
>>>>>"attributes":{ any JSON can go here }
>>>>> }
>>>>>
>>>>> While I realize that we could stringify the JSON, this breaks our
>>>>> published API.  Is there any way I can use protobuf to perform serdes
>>>>> between message like this and some struct like:
>>>>>
>>>>> {
>>>>>string name;
>>>>>string address;
>>>>>json attributes;
>>>>> }
>>>>>
>>>>> I'm even OK if the internal data is stringified JSON:
>>>>>
>>>>> {
>>>>>string name;
>>>>>string address;
>>>>>string attributes;
>>>>> }
>>>>>
>>>>> So long as the exchanged JSON isn't stringified.  In other words, this
>&

Re: [protobuf] Documentation Riddle

[Josh Humphries]

On Wed, Sep 26, 2018 at 3:42 AM omid pourhadi 
wrote:

> I have some questions :
>
> 1. Is it a good idea to use Any for transferring a serialized/deserialized
> java object from client to server for example consider this ?
>

A google.protobuf.Any encapsulates a *protobuf message *whose type can
vary, not an arbitrary type from an arbitrary language. So, no, you can't
use it to represent a serialized Java object because you won't have a valid
type URL for that object. The type URL is meant to refer to a protobuf type.

Instead, you would need to use bytes. Or you could create your own type
that is similar to Any:

message Object {
string type = 1;
bytes value = 2;
}

Other options are to use google.protobuf.Struct, but that is limited to
what can be represented by a JSON object. If your Java objects have a
canonical JSON representation then this might be a useful solution.

The final alternative is to create your own set of generic messages, very
similar to the types in "google/protobuf/struct.proto" but for modeling
Java objects in a higher-fidelity manner than just as serialized bytes.


>
> import "google/protobuf/empty.proto";
> import "google/protobuf/any.proto";
> import "google/protobuf/type.proto";
>
> service AcceptorService {
>   rpc search(google.protobuf.Any) returns (google.protobuf.ListValue) {};
>
> }
>
> 2. How can I use ListValue ?
>

The google.protobuf.ListValue type can only represent the same category of
values that you can represent with a JSON Array. So it can't represent a
list of strongly-typed values. (Instead, the best you could do is a list of
google.protobuf.Struct to represent a List). So if you instead are
trying to represent a Java List, you'd be better off creating your
own type:

message ListOfObject {
// this example uses bytes, serialized Java objects, as the values
// but could reference other types as mentioned above for how you
// might represent an arbitrary Java object in protobuf
repeated bytes objects = 1;
}

You could then serialize Java objects into elements of the objects field.


>
> 3. How can I define a Map in a message (I found
> MapOfObjects in maps.proto but it is using string)?
>

I don't know what maps.proto is. Perhaps you are working with a set of
message types where someone else has already tried to model much of your
Java domain objects already? Anyhow, you'd need to do something like
mentioned above: map where values are serialized Java
objects (or some other value type, based on how you decide to model an
arbitrary Java object).

FWIW, a generally better idea is to limit the domain of what kind of object
can be transferred and model all of them as messages in proto. That would
allow you to use google.protobuf.Any instead of bytes. You could then
define your list and maps as:

message dummy {
repeated google.protobuf.Any list = 1;
map map = 2;
}

Note however that maps in protobuf are limited to integral, bool, and
string keys: you cannot use messages or bytes values as the key for a
protobuf map (unlike in Java, where anything can be a map key). If you have
a data structure like that in Java, you'd instead have to model it like so:

// this example is for Map
message MapObjectStringEntry {
google.protobuf.Any key = 1;
string value = 2;
}

message dummy {
repeated MapObjectStringEntry map = 1;
}

Hope this helps. Good luck.


> message dummy{
>  map filters = 1;
> }
>
> Thanks in advanced
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Reflection of well known types in C++

[Josh Humphries]

They are well-known by their fully-qualified name, e.g.
"google.protobuf.DoubleValue". And you can get the fully-qualified names
via C++ reflection.

So you're marshaling logic woudl need to special-case the relevant message
names and subsequently can make assumptions about their structure (since
they are "well known"). As far as "auto-unboxing", keep in mind that one
significant use case for these well-known types is that they can be nil,
whereas there is obviously no analog for the unboxed primitive type (zero
is the closest, but then you cannot distinguish between whether the
original was nil or explicit zero).


*Josh Humphries*
jh...@bluegosling.com



On Fri, Jul 27, 2018 at 2:08 PM Chris Buffett 
wrote:

> I'm wondering if protobuf supports a way to determine if a type is a
> well-known type via reflection in C++? I'm working on a marshalling layer
> for a custom encoding format and I need to unbox well-known types into
> their primitive type (e.g., Double to double). I've so far been unable to
> find any information on if it's possible to definitively tell if a type is
> well known or not, other than .
>
> For example, if I have the following protobuf,
>
> message A
> {
> message B
> {
> double val = 1;
> google.protobuf.Doublevalue prev_val = 2;
> }
>
> repeated B values;
> }
>
> I need to convert into something with the following format
> class B
> {
> double val;
> double prev_val;
> }
>
> class A
> {
> list values;
> }
>
> The issue I see is that based on the protobuf format, the output would be
> ambiguous. I could also generate the following because I'm unable to tell
> whether I should unbox the type or generate a wrapping class.
>
> class B
> {
> double val;
> class C
> {
> double prev_val;
> }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to generate FileDescriptor Object which has custom options?

[Josh Humphries]

On Mon, Jul 23, 2018 at 12:56 PM Ankit Patel 
wrote:

> Hi Josh,
>
> Thanks for responding. I do not have access to the codegen classes from
> the proto compiler. My requirement is to use the proto files and create
> descriptors from it and rely on that. I have the proto file where the
> extension is defined and I am using that to create a registry and then feed
> it to the File descriptor of my proto as shown in the above code. But It is
> still not reading that registry and updating itself. What am I doing wrong?
>

Hi, Ankit,
I apologize that my first response was incorrect. I clearly didn't ingest
your entire code example. After reviewing more closely, I'll admit, it
looks like you are doing everything correctly. Hopefully someone else on
the list can spot the issue.

In the meantime, I'd probably use a debugger to step through it,
particularly the call to internalUpdateFileDescriptor, to see what's going
on internally. An alternative you might try is just parsing "Hello.pb" with
your initialized extension registry, instead of parsing it up front without
and then having to call the internal update method later. (Not that it
should behave differently -- but it could be interesting to verify whether
or not it behaves differently.)


>
> Thanks,
> Ankit
>
> On Friday, July 20, 2018 at 5:56:35 PM UTC-7, Josh Humphries wrote:
>>
>> The issue is that when you access the descriptor that way, it does not
>> know about the custom options.
>>
>> You need to create an ExtensionRegistry. You'll also need the Java code
>> produced from the annotations.proto file: that generated code will have a
>> way to access the extensions (to add them to your registry), and IIRC even
>> includes a way to register all extensions in the file.
>>
>> Then you'll have to re-parse the descriptor proto with the
>> ExtensionRegistry -- that way, the field will be recognized and parsed
>> correctly instead of considered an unknown field. IIRC (I've been doing Go
>> for a while, so my recollection of the Java protobuf runtime might be
>> fading a little), to re-parse them you have to serialize it to bytes (in
>> this case, the method descriptor proto) and then de-serialize again. The
>> resulting de-serialized message will now have the options in a queryable
>> form.
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>>
>>
>> On Fri, Jul 20, 2018 at 8:30 PM Ankit Patel  wrote:
>>
>>> Hi All,
>>>
>>> I have a simple .proto file
>>>
>>> syntax = "proto3";
>>>
>>> package helloworld.v1;
>>>
>>> option java_package = "helloworld.v1";
>>> option java_multiple_files = true;
>>>
>>> import "google/api/annotations.proto";
>>>
>>> service HelloWorldService {
>>> rpc sayHello (SayHelloRequest) returns (SayHelloResponse) {
>>> option (google.api.http) = {
>>> post: "/v1/example/echo"
>>> body: "*"
>>> };
>>> }
>>> }
>>>
>>> message SayHelloRequest {
>>> string favourite_thing = 1;
>>> }
>>>
>>> message SayHelloResponse {
>>> string reply = 1;
>>> }
>>>
>>> I am generating a proto descriptor set from it using the proto compiler.
>>>
>>> After that I am trying to read the proto descriptor programatically in
>>> the java code like this
>>> FileDescriptorSet Hello = FileDescriptorSet.parseFrom(new
>>> FileInputStream(new File("D:\\temp\\descriptor\\Hello.pb")));
>>> FileDescriptorSet Annotations = FileDescriptorSet.parseFrom(new
>>> FileInputStream(new File("D:\\temp\\descriptor\\annotations.pb")));
>>> FileDescriptorSet Http = FileDescriptorSet.parseFrom(new
>>> FileInputStream(new File("D:\\temp\\descriptor\\http.pb")));
>>> FileDescriptorSet Descriptor = FileDescriptorSet.parseFrom(new
>>> FileInputStream(new File("D:\\temp\\descriptor\\descriptor.pb")));
>>>
>>> FileDescriptor httpD = FileDescriptor.buildFrom(Http.getFile(0), new
>>> FileDescriptor[] {}, true);
>>> FileDescriptor descriptorD =
>>> FileDescriptor.buildFrom(Descriptor.getFile(0), new FileDescriptor[] {},
>>> true);
>>> FileDescriptor annotationsD =
>>> FileDescriptor.buildFrom(Annotations.getFile(0), new FileDescriptor[]
>>> {httpD, descriptorD});
>>>
>>> ExtensionRegistry registry = ExtensionRegistry.newInstance();
>>> FieldDescriptor httpFD = annotationsD.findExtensionByName("http&qu

Re: [protobuf] How to generate FileDescriptor Object which has custom options?

[Josh Humphries]

The issue is that when you access the descriptor that way, it does not know
about the custom options.

You need to create an ExtensionRegistry. You'll also need the Java code
produced from the annotations.proto file: that generated code will have a
way to access the extensions (to add them to your registry), and IIRC even
includes a way to register all extensions in the file.

Then you'll have to re-parse the descriptor proto with the
ExtensionRegistry -- that way, the field will be recognized and parsed
correctly instead of considered an unknown field. IIRC (I've been doing Go
for a while, so my recollection of the Java protobuf runtime might be
fading a little), to re-parse them you have to serialize it to bytes (in
this case, the method descriptor proto) and then de-serialize again. The
resulting de-serialized message will now have the options in a queryable
form.


*Josh Humphries*
jh...@bluegosling.com



On Fri, Jul 20, 2018 at 8:30 PM Ankit Patel 
wrote:

> Hi All,
>
> I have a simple .proto file
>
> syntax = "proto3";
>
> package helloworld.v1;
>
> option java_package = "helloworld.v1";
> option java_multiple_files = true;
>
> import "google/api/annotations.proto";
>
> service HelloWorldService {
> rpc sayHello (SayHelloRequest) returns (SayHelloResponse) {
> option (google.api.http) = {
> post: "/v1/example/echo"
> body: "*"
> };
> }
> }
>
> message SayHelloRequest {
> string favourite_thing = 1;
> }
>
> message SayHelloResponse {
> string reply = 1;
> }
>
> I am generating a proto descriptor set from it using the proto compiler.
>
> After that I am trying to read the proto descriptor programatically in the
> java code like this
> FileDescriptorSet Hello = FileDescriptorSet.parseFrom(new
> FileInputStream(new File("D:\\temp\\descriptor\\Hello.pb")));
> FileDescriptorSet Annotations = FileDescriptorSet.parseFrom(new
> FileInputStream(new File("D:\\temp\\descriptor\\annotations.pb")));
> FileDescriptorSet Http = FileDescriptorSet.parseFrom(new
> FileInputStream(new File("D:\\temp\\descriptor\\http.pb")));
> FileDescriptorSet Descriptor = FileDescriptorSet.parseFrom(new
> FileInputStream(new File("D:\\temp\\descriptor\\descriptor.pb")));
>
> FileDescriptor httpD = FileDescriptor.buildFrom(Http.getFile(0), new
> FileDescriptor[] {}, true);
> FileDescriptor descriptorD =
> FileDescriptor.buildFrom(Descriptor.getFile(0), new FileDescriptor[] {},
> true);
> FileDescriptor annotationsD =
> FileDescriptor.buildFrom(Annotations.getFile(0), new FileDescriptor[]
> {httpD, descriptorD});
>
> ExtensionRegistry registry = ExtensionRegistry.newInstance();
> FieldDescriptor httpFD = annotationsD.findExtensionByName("http");
> Message httpRuleM =
> DynamicMessage.newBuilder(httpD.findMessageTypeByName("HttpRule")).build();
> registry.add(httpFD, httpRuleM);
>
> FileDescriptor helloD = FileDescriptor.buildFrom(Hello.getFile(0), new
> FileDescriptor[] {annotationsD});
> FileDescriptor.internalUpdateFileDescriptor(helloD, registry);
>
> helloD.findServiceByName("HelloWorldService").findMethodByName("sayHello").getOptions();
>
> The api call in the last line puts the "http" in the unknown field, and
> the reason is because its a custom option and not registered. But as you
> can see I am doing that.
>
> Can someone please guide me on what am I doing wrong?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Embedding arbitrary JSON in a message

[Josh Humphries]

Oops, I meant to point you to google.protobuf.Value: https:/
/github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L63

It can represent *any* kind of JSON value. The Struct type is what is used
to represent JSON *objects* (there is also ListValue, for arrays, as well
as support for JS primitive types and null).



*Josh Humphries*
jh...@bluegosling.com

On Thu, Jun 21, 2018 at 3:25 PM, Josh Humphries 
wrote:

> Hi, John,
> Take a look at the well-known type google.protobuf.Struct. It is
> basically a JSON value, modeled as a proto. It's JSON representation is
> exactly what you want, too:
>
> https://github.com/google/protobuf/blob/master/src/google/
> protobuf/struct.proto#L52
>
>
>
> 
> *Josh Humphries*
> jh...@bluegosling.com
>
> On Thu, Jun 21, 2018 at 3:20 PM, John Lilley  wrote:
>
>> Disclaimer: I am totally new to protobuf, engaged in an exploratory POC.
>> Please forgive dumb questions :-)
>>
>> We are looking at migrating an existing, JSON-based protocol in which
>> hand-coded C++ is written to perform serdes between objects and JSON. We
>> want to replace the hand-coding with an automated approach that can be
>> shared between C++ and Java.  However, a stumbling block I see is that some
>> messages have an arbitrary field full of JSON like:
>>
>> {
>>"name":"john",
>>"address":"123 main st",
>>"attributes":{ any JSON can go here }
>> }
>>
>> While I realize that we could stringify the JSON, this breaks our
>> published API.  Is there any way I can use protobuf to perform serdes
>> between message like this and some struct like:
>>
>> {
>>string name;
>>string address;
>>json attributes;
>> }
>>
>> I'm even OK if the internal data is stringified JSON:
>>
>> {
>>string name;
>>string address;
>>string attributes;
>> }
>>
>> So long as the exchanged JSON isn't stringified.  In other words, this is
>> bad:
>> {
>>"name":"john",
>>"address":"123 main st",
>>"attributes":"{ \"attr1\":\"value1\", \"attr2\":[\"elem1\", \"elem2\"]
>>  }"
>> }
>>
>> It needs to be exchanged like
>> {
>>"name":"john",
>>"address":"123 main st",
>>"attributes":{ "attr1":"value1", "attr2":["elem1", "elem2"] }"
>> }
>>
>> Is this possible?
>>
>> Thanks
>> john
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] ANN: 1.0 GA of ProtoPoet, Java API for generating Protobuf files

[Josh Humphries]

Hey, Derek,
This looks like a pretty cool project. One interesting thing about protos
is that the language model has value outside of just generating source
code: the same API should be able to serve the purpose of generating
descriptors, for runtime/dynamic usage of protobufs. Is that a feature of
this library by chance?

While this is targeted as Java, if anyone building stuff in Go has need for
something similar, I've built very similar functionality into one of my own
Go projects: https://github.com/jhump/protoreflect

In particular, the desc/builder
<https://godoc.org/github.com/jhump/protoreflect/desc/builder> package
provides a similar API, for building a model of proto source files and
elements. But its output is not generated source code but instead
descriptors. One can then use the desc/protoprint
<https://godoc.org/github.com/jhump/protoreflect/desc/protoprint> package
to generate source files from those descriptors.


FWIW, I am also a fan of Java Poet and recently began work on something
similar, but for generating Go code. (I of course called it Go Poet.) It's
a long way from version 1, but maybe when it gets to that milestone, I'll
post an announcement about it to the go-nuts mailing list.




*Josh Humphries*
jh...@bluegosling.com

On Sat, Jun 2, 2018 at 3:08 PM, Derek Perez  wrote:

> Hey all,
>
> I've been working on this project in my spare time and its finally at what
> I would call 1.0 level. The concept was inspired by Square's JavaPoet
> library for Java which is used to generated Java source code. This library
> generates Protobuf source code with an intuitive Java API. There's a
> javadoc, README examples, and the code is pretty easy to read/follow. Hope
> someone else finds it useful!
>
> https://github.com/perezd/protopoet
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Tool for programmatically adding field options in .proto files

[Josh Humphries]

One possible solution is to use a bits of the protoreflect library for Go
that I've written:

https://godoc.org/github.com/jhump/protoreflect/desc/protoparse
When parsing, you'll want to use the flag to include source code info. This
will preserve a little more comment info than descriptor produced by
protoc. FWIW, the descriptor with source info can be had from protoc with
an invocation like so:
protoc -o outputfile --include_source_info --include_imports some.proto

Once the files are parsed, you can easily recursively visit all of the
fields and modify the field descriptors to add a default value.

https://godoc.org/github.com/jhump/protoreflect/desc/protoprint
Once the option is added, you can write the modified descriptors back to
source files with this package. This will preserve a lot of comment info
(though admittedly not all).

The main downside is that this in no way preserves formatting from the
original file. The formatting produced isn't bad, but it doesn't quite look
like hand-written code due to the extra whitespace you'll see (blank line
between every element). To preserve formatting, you probably will need to
write a custom parser.

YMMV


*Josh Humphries*
jh...@bluegosling.com

On Fri, May 4, 2018 at 3:04 PM, Mark Hoffman <meho...@gmail.com> wrote:

> Hi,
>
> I have the tasks of retrofitting a large repo of .proto files to include a
> field option with a default value to every field, except for enums. We
> can't assume if the field option is not present that it has a default, for
> security reasons. Before I go off an write a parser to handle this one time
> task, I was justing wondering something might already exist. One other
> issue, I need to preserve the comments in the .proto files.
>
> Thanks.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] read ahead from protobuf stream

[Josh Humphries]

You could use a length-delimited stream of google.protobuf.Any messages.
The Any message has a URL that encodes the fully-qualified type of the
message and then also the bytes of the corresponding serialized proto. When
I say length-delimited, I mean prefix each message with a varint-encoded
length (or fixed 32-bit length would work, too). That is a common way to
define a stream of protos, and there is some library support -- at least in
Java -- for the varint-encoded delimited streams.


*Josh Humphries*
jh...@bluegosling.com

On Sat, Apr 28, 2018 at 12:02 PM, Stefan Seefeld <stefan.seef...@gmail.com>
wrote:

> Hello,
>
> I'm trying to use protocol buffers as a means to stream data, where an
> application would iteratively read messages from some input source.
> Messages have different types, so to be able to call
> `m.ParseFromIstream(input)` with an object `m` of appropriate type, I need
> to read ahead to identify the type of the next message so I can instantiate
> it. Do the IO APIs provide such a feature (I'm most interested into C++ and
> Python) ? I read about different ways to handle this (using `extend` to
> mimic polymorphism, mostly), I find that quite cumbersome and inefficient.
>
> So what's the best way to approach this ?
>
> Thanks,
>   Stefan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: latest master: "make check" failing on OS X, can't find "google/protobuf/compiler/profile.pb.h"

[Josh Humphries]

Never mind. It must have been some artifacts from an older version sitting
in my source tree. After running "make clean" and retrying, all is well.

----
*Josh Humphries*
jh...@bluegosling.com

On Sun, Feb 25, 2018 at 7:36 PM, Josh Humphries <jh...@bluegosling.com>
wrote:

> I am trying to build latest from source on a Macbook Pro.
>
> All steps to configure and then "make" protoc work. But then I get the
> following error when I run "make check":
>
> *no_warning_test.cc:81:10: **fatal error: 
> **'google/protobuf/compiler/profile.pb.h'
> file not found*
>
> I can find no such file in the repo nor can I find any .proto source file
> from which it would be generated.
>
> Anyway have any tips on what's wrong with my environment?
>
> 
> *Josh Humphries*
> jh...@bluegosling.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] latest master: "make check" failing on OS X, can't find "google/protobuf/compiler/profile.pb.h"

[Josh Humphries]

I am trying to build latest from source on a Macbook Pro.

All steps to configure and then "make" protoc work. But then I get the
following error when I run "make check":

*no_warning_test.cc:81:10: **fatal error:
**'google/protobuf/compiler/profile.pb.h'
file not found*

I can find no such file in the repo nor can I find any .proto source file
from which it would be generated.

Anyway have any tips on what's wrong with my environment?


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to convert protocol-buffer to a HashMap in java?

[Josh Humphries]

A much more efficient approach would be to use the getAllFields() method of
Message and then translate the message into a map of maps. The keys of the
getAllFields() map are field descriptors, from which you can extract the
field name. If a value is a nested message, you can recursively apply the
same transformation.

However, if you are trying to emulate the map you get from JSON
serialization, it will be trickier. Several types (the "well-known types")
have custom JSON forms. And proto has a standard way to map field names (as
defined in proto) to JSON keys (converts lower--snake-case -> camel-case
with initial lower-case). So if that's what you're after, it's probably is
better to just rely on JSON serialization instead of trying to reproduce
that logic.



*Josh Humphries*
jh...@bluegosling.com

On Sun, Feb 18, 2018 at 10:30 PM, Debraj Manna <subharaj.ma...@gmail.com>
wrote:

> Anyone any thoughts?
>
> On 16-Feb-2018 2:27 PM, "tuk" <subharaj.ma...@gmail.com> wrote:
>
>> Cross-posting from stackoverflow
>> <https://stackoverflow.com/questions/48821916/how-to-convert-protocol-buffer-message-to-a-hashmap-in-java>
>>
>> I have a protobuf message of the form
>>
>> enum PolicyValidationType {
>> Number = 0;}
>>
>>
>> message NumberPolicyValidation {
>> optional int64 maxValue = 1;
>> optional int64 minValue = 2;}
>>
>> message PolicyObject {
>> required string key = 1;
>> optional string value = 2;
>> optional string name = 3;
>> optional PolicyValidationType validationType = 4;
>> optional NumberPolicyValidation numberPolicyValidation = 5;}
>>
>>
>> For example
>>
>> policyObject {
>>   key: "sessionIdleTimeoutInSecs"
>>   value: "1800"
>>   name: "Session Idle Timeout"
>>   validationType: Number
>>   numberPolicyValidation {
>> maxValue: 3600
>> minValue: 5
>>   }}
>>
>>
>> Can someone let me know how can I convert this to a Map like below:-
>>
>> {validationType=Number, name=Session Idle Timeout, 
>> numberPolicyValidation={maxValue=3600.0, minValue=5.0}, value=1800, 
>> key=sessionIdleTimeoutInSecs}
>>
>>
>> One way I can think of is convert this to a json and then convert the
>> json to map?
>>
>> PolicyObject policyObject;...JsonFormat jsonFormat = new JsonFormat();final 
>> String s = jsonFormat.printToString(policyObject);Type objectMapType = new 
>> TypeToken<HashMap<String, Object>>() {}.getType();Gson gson = new 
>> GsonBuilder().registerTypeAdapter(new 
>> TypeToken<HashMap<String,Object>>(){}.getType(), new 
>> PrimitiveDeserializer()).create();Map<String, Object> mappedObject = 
>> gson.fromJson(s, objectMapType);
>>
>>
>> I think there must be some better way. Can someone suggest any better
>> approach?
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Protocol Buffers" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/protobuf/ElF4l1j1Pfk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Where is Protocol Buffers wire level specification?

[Josh Humphries]

This describes the basic encoding of protobuf messages into bytes:
https://developers.google.com/protocol-buffers/docs/encoding

The format does not include a length header. If you have data that
concatenates protobuf messages, then each such message will typically be
prefixed by a varint-encoded length, to delimit multiple messages in a
single stream. (Though other usages of proto, such as gRPC, have different
framing mechanisms for concatenating multiple messages in a single stream.)



*Josh Humphries*
jh...@bluegosling.com

On Tue, Jan 9, 2018 at 12:01 PM, Sergei Gnezdov <sergei.gnez...@gmail.com>
wrote:

> I see a user level spec but I don't see a spec that defines a binary
> format.
>
>
> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Compile protobuf to use Java primitive classes instead of well-known types

[Josh Humphries]

What you're asking for is not currently do-able. There is an existing
request to add this functionality though:
https://github.com/google/protobuf/issues/2055

In the meantime, you can accomplish this using a custom plugin that is run
in addition to the normal java code-gen. Plugins can generate extra code
into existing Java source files using insertion points
<https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/plugin.proto#L161>.
So you could generate overloaded forms of the setter methods into the
relevant builder classes (via "builder_scope" insertion points).


*Josh Humphries*
jh...@bluegosling.com

On Thu, Jan 4, 2018 at 7:13 AM, Renatas M <rm.laiki...@gmail.com> wrote:

> Lets say I have test.proto file:
>
> syntax = "proto3";
>
> option java_package = "testing";
> option java_outer_classname = "Test_v1";
>
> import "google/protobuf/wrappers.proto";
>
> message TestMessage {
> .google.protobuf.Int32Value integerField = 1;
> }
>
>
> If I compile (using protoc v3.5.0) it to c# code I will get nullable type
> properties:
>
> public int? IntegerField {
>get { return integerField_; }
>set { integerField_ = value; }
>  }
>
>
> But if I compile it to Java code, fields will be of well-known type:
>
> public Builder setIntegerField(com.google.protobuf.Int32Value value) {
>   if (integerFieldBuilder_ == null) {
> if (value == null) {
>   throw new NullPointerException();
> }
> integerField_ = value;
> onChanged();
>   } else {
> integerFieldBuilder_.setMessage(value);
>  }
>
>
> I am moving project from proto2 to proto3 so I would like to avoid using
> well-known types, because it will require a lot of work to change related
> code. With c# projects I will not require to modify anything but in Java I
> will need to do as in the following example:
>
> TestMessage.Builder builder = TestMessage.newBuilder();
> builder.setIntegerField(5); //<-- instead of this (with proto2)
> builder.setIntegerField(Int32Value.newBuilder().setValue(5)); //<-- I
> will need to change to something like this (with proto3)
>
>
> Is there a way to compile proto file to Java so setters would accept
> primitive classes/wrappers (String, Integer, etc.) as parameters?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is protobuf3 a superset of protobuf2?

[Josh Humphries]

Adam,
I understand there is no *current* plan to deprecate proto2. But is it
really expected to be supported forever?

If that's the case, I suppose the only reason to choose proto3 over proto2
(if you happen to want the features and semantics of proto2) might be that
not all languages/runtimes have support for proto2. Though I thought this
was a short/medium-term issue. If all languages/runtimes eventually support
both, it seems like a strange decision to continue supporting both
indefinitely.



*Josh Humphries*
jh...@bluegosling.com

On Tue, Dec 19, 2017 at 2:23 PM, Adam Cozzette <acozze...@google.com> wrote:

> Actually we have no plans to deprecate proto2 and we are still actively
> developing it, so you can really choose either one without having to worry
> about support going away.
>
> On Sat, Dec 16, 2017 at 11:42 AM, Josh Humphries <jh...@bluegosling.com>
> wrote:
>
>> I think proto3 was intended to be simpler -- an evolution of protobuf in
>> a direction that is more refined and elides superfluous features.
>> Eventually (though not likely any time soon), support for proto2 will go
>> away.
>>
>> The main omission in proto3 that I personally felt strongly about was the
>> lack of preserving unknown fields. But that functionality has been restored
>> for many languages/runtimes in 3.5 (and coming soon, hopefully, for Go and
>> the others).
>>
>> All of the other omissions in proto3 have alternate
>> solutions/work-arounds that aren't baked into the language (like wrapper
>> types and Any), which shows that these features didn't truly need to be
>> language features to begin with. So it allows the language itself to remain
>> simple, which means libraries/tooling are simpler to build on top.
>>
>> So a big advantage of proto3 is simplicity (and for some
>> languages/runtimes, that will translate into improved performance). And
>> another advantage is future compatibility (since, in all likelihood, proto2
>> will eventually go away.)
>>
>>
>> If creating new files, it makes sense to use proto3. If you already have
>> a corpus of proto2 files, a reasonable migration path forward is to start
>> evolving them to proto3 (or, if the languages/runtimes you use don't yet
>> preserve unrecognized fields, wait for that functionality to be fully
>> implemented everywhere and then start evolving).
>>
>>- Make required fields optional. Required fields have always been
>>dangerous. If a field is required in your protocol, it can be enforced in
>>application code instead of in the de-serialization logic of the protobuf
>>runtime.
>>- Use Any instead of extensions. An extension range can generally be
>>replaced with a repeated Any field. Extensions with scalar types will have
>>to be "boxed" into single-field messages.
>>- Use wrapper types for scalar fields where absence must be
>>distinguishable from zero.
>>- Usages of default values can be changed to use wrapper types and
>>application logic that decides non-zero values to use when the field is 
>> not
>>present.
>>- Groups should be replaced with nested messages.
>>
>> Once a file is no longer using any proto2 features, it can be changed to
>> proto3 syntax. Unfortunately, there is not yet a way to incrementally move
>> to proto3. Once you change the syntax of the file, the generated code will
>> change, and all clients of that generated code must be changed atomically
>> in order to compile (at least for languages/runtimes where generated code
>> for the two syntaxes is materially different/incompatible). I would hope
>> that this is something that will be addressed in future versions of
>> protobuf, to facilitate migrations away from proto2.
>>
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>> On Sat, Dec 16, 2017 at 2:17 PM, ajcurtis84 <ajcurti...@gmail.com> wrote:
>>
>>> Thank you everyone for all the great input!
>>>
>>> Based on this discussion, what is the advantage of using proto3? It
>>> appears that proto2 is more feature rich. JSON isn't a compelling reason
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/p

Re: [protobuf] Is protobuf3 a superset of protobuf2?

[Josh Humphries]

I think proto3 was intended to be simpler -- an evolution of protobuf in a
direction that is more refined and elides superfluous features. Eventually
(though not likely any time soon), support for proto2 will go away.

The main omission in proto3 that I personally felt strongly about was the
lack of preserving unknown fields. But that functionality has been restored
for many languages/runtimes in 3.5 (and coming soon, hopefully, for Go and
the others).

All of the other omissions in proto3 have alternate solutions/work-arounds
that aren't baked into the language (like wrapper types and Any), which
shows that these features didn't truly need to be language features to
begin with. So it allows the language itself to remain simple, which means
libraries/tooling are simpler to build on top.

So a big advantage of proto3 is simplicity (and for some
languages/runtimes, that will translate into improved performance). And
another advantage is future compatibility (since, in all likelihood, proto2
will eventually go away.)


If creating new files, it makes sense to use proto3. If you already have a
corpus of proto2 files, a reasonable migration path forward is to start
evolving them to proto3 (or, if the languages/runtimes you use don't yet
preserve unrecognized fields, wait for that functionality to be fully
implemented everywhere and then start evolving).

   - Make required fields optional. Required fields have always been
   dangerous. If a field is required in your protocol, it can be enforced in
   application code instead of in the de-serialization logic of the protobuf
   runtime.
   - Use Any instead of extensions. An extension range can generally be
   replaced with a repeated Any field. Extensions with scalar types will have
   to be "boxed" into single-field messages.
   - Use wrapper types for scalar fields where absence must be
   distinguishable from zero.
   - Usages of default values can be changed to use wrapper types and
   application logic that decides non-zero values to use when the field is not
   present.
   - Groups should be replaced with nested messages.

Once a file is no longer using any proto2 features, it can be changed to
proto3 syntax. Unfortunately, there is not yet a way to incrementally move
to proto3. Once you change the syntax of the file, the generated code will
change, and all clients of that generated code must be changed atomically
in order to compile (at least for languages/runtimes where generated code
for the two syntaxes is materially different/incompatible). I would hope
that this is something that will be addressed in future versions of
protobuf, to facilitate migrations away from proto2.


----
*Josh Humphries*
jh...@bluegosling.com

On Sat, Dec 16, 2017 at 2:17 PM, ajcurtis84 <ajcurti...@gmail.com> wrote:

> Thank you everyone for all the great input!
>
> Based on this discussion, what is the advantage of using proto3? It
> appears that proto2 is more feature rich. JSON isn't a compelling reason
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is protobuf3 a superset of protobuf2?

[Josh Humphries]

To clarify, proto3 supports extensions in that you can define custom
options or create extensions for proto2 messages (since it is valid to
import proto2 files). But it does not support defining an extendable
message (e.g. no extension ranges can be defined with proto3).

Proto3 is almost more like a *sub*set of proto2:

   1. You cannot define extension ranges in proto3
   2. You cannot define required fields in proto3
   3. You cannot define default values in proto3. Default values are always
   the zero value for a type.
   4. You cannot define groups in proto3
   5. Proto3 has no has* methods because, by design, an absent value is
   indistinguishable from its default (e.g. zero) value.
   6. You cannot define an enum whose first value does not have the numeric
   value of zero.
   7. Up until 3.5, proto3 did not preserve unrecognized fields. Even as of
   3.5, this support is still not complete in all languages (such as Go), even
   though it is supported for proto2 (for the languages/runtimes that have
   proto2 support anyway).

All of the "new" features ascribed to proto3 are really features of the
protoc compiler, starting with v3.0, not of proto3. The new features
include the following:

   1. Map types
   2. Streaming RPCs
   3. Standard JSON representation
   4. Well-known types (including Any)
   5. New language-level: proto3 syntax

So it's important not to conflate "protobuf v3" with bullet 5 above,
"proto3". All of those other features can still be used with proto2 source
files (aside, possibly, from using JSON format with a message with
extensions).




*Josh Humphries*
jh...@bluegosling.com

On Fri, Dec 15, 2017 at 6:56 PM, 'Adam Cozzette' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Neither one is really a superset of the other. Proto3 does not support
> extensions but uses the Any type as a replacement.
>
> On Fri, Dec 15, 2017 at 3:40 PM, ajcurtis84 <ajcurti...@gmail.com> wrote:
>
>> Hello,
>>
>> I have not found anything in the documentation that explicitly says this.
>> The only indication is that the proto3 documentation refers to proto2.
>> Things like extensions are available in the proto3 syntax?
>>
>> thanks
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to combine FieldOptions with Default Values in Messages

[Josh Humphries]

Even though it's not a field on FieldOptions, you can supply a
"pseudo-option" for default value named simply default:
https://developers.google.com/protocol-buffers/docs/proto#optional
When protoc compiles this source, the default value ends up in a field on
FieldDescriptorProto. There is one other "pseudo-option" in protocol
buffers to be aware of: "json_name", also for fields, also ends up in
FieldDescriptorProto instead of FieldOptions when compiled.

Language runtimes that support syntax "proto2" and default values should
generate useful API, so that if you use a *method* to query the field
(instead of, for example, directly examining a field in an object/struct),
the method should return the specified default if the field has not been
explicitly set.

The default value that is stored in the descriptor is stored as a string.
The one thing to keep in mind is that bytes fields have a strange encoding:
the string value uses C-escapes for non-printable characters. So you'll
have to decode the default for bytes fields. For other types, the string
encoding is exactly what you expect (enums are represented as enum value
names, not numbers).

BTW, the default option in protos does not allow you to define a default
value for aggregate fields (e.g. no defaults allowed for repeated/map
fields or fields whose type is a nested message).




*Josh Humphries*
jh...@bluegosling.com

On Sun, Nov 12, 2017 at 2:11 AM, <shiyw...@redhat.com> wrote:

> I have an message using FieldOptions but I also want to use default values
> as well, how do I combine two of those things ?
>
> extend google.protobuf.MessageOptions {
> optional CommandInfo cmd = 5678;
> }
>
>
> message TestCmd {
>
> optional bool BoolFlag = 1 [(info) = { name: "bool-flag" value: "false" 
> shorthand: "c" usage: "test"} ];
> optional string StringFlag = 2 [ (info) = { name: "string-flag" value: "" 
> shorthand: "o" usage: "test"}];
>
> }
>
>
>
> I want that TestCmd.BoolFlag default value to be bool value  "false", 
> currently I'm using string value then convert it to bool, How do I
>
> specify the default value in a type-safe way ?
>
>
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: How to use custom options in official docs

[Josh Humphries]

It works fine when I run it.

> *go test -v .*
=== RUN   TestABCMessage
opt1:123 opt2:"baz"
--- PASS: TestABCMessage (0.00s)
PASS
ok  test 0.007s



However, your test file was incomplete. Maybe you have imported the wrong
stuff? Double-check that you are linking against the latest version of the
github.com/golang/protobuf packages.
I had to add imports, and I also changed the handling of the output since I
don't know what "spew" is:



package cmd

import (
  "fmt"
  "testing"

  "github.com/golang/protobuf/descriptor"
  "github.com/golang/protobuf/proto"
)

func TestABCMessage(t *testing.T) {
   var msg Bar

   _, md := descriptor.ForMessage()

   opts := md.Field[0].GetOptions()
   foo, err := proto.GetExtension(opts, E_FooOptions)
   if err != nil {
  t.Errorf("failed: %v", err)
  } else {
  fmt.Println(foo)
   }
}




*Josh Humphries*
jh...@bluegosling.com

On Wed, Oct 25, 2017 at 10:54 PM, <shiyw...@redhat.com> wrote:

> /tmp/TestABCMessage_in_version_abc_test_gogo -test.v -test.run
> ^TestABCMessage$
> (*errors.errorString)(0xc420214af0)(proto: not an extendable proto)
> (interface {}) 
>
> On Wednesday, October 25, 2017 at 8:38:51 PM UTC+8, shiy...@redhat.com
> wrote:
>>
>> the example in https://developers.google.com/protocol-buffers/docs/proto
>> #customoptions below
>>
>> message FooOptions {
>>   optional int32 opt1 = 1;
>>   optional string opt2 = 2;
>> }
>>
>> extend google.protobuf.FieldOptions {
>>   optional FooOptions foo_options = 1234;
>> }
>>
>> // usage:
>> message Bar {
>>   optional int32 a = 1 [(foo_options).opt1 = 123, (foo_options).opt2 = 
>> "baz"];
>>   // alternative aggregate syntax (uses TextFormat):
>>   optional int32 b = 2 [(foo_options) = { opt1: 456 opt2: "xaa" }];
>> }
>>
>> assume I have a message Bar here:
>>
>> var msg Bar
>> _, md := ForMessage()
>> md.Field[0].GetOptions().String()  only a string, How do i parse that ??
>>
>> How to I get the exact value of Bar.b.opt1 which is 123 (default value)
>>
>> What I want is like this
>>
>> md.Field[0].GetOptions().GetOpt1()   // 123
>> md.Field[0].GetOptions().GetOpt2()// baz
>> md.Field[1].GetOptions().GetOpt1()   // 456
>> md.Field[1].GetOptions().GetOpt2()   // xaa
>>
>> Can we achieve that ?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to use custom options in official docs

[Josh Humphries]

The proto package has methods for accessing extensions. And the generated
.pb.go file contains exported symbols that describe those extensions. Put
them together, and you can get the custom options out -- something like so:

opts := md.Field[0].GetOptions()
foo, err := proto.GetExtension(opts, E_FooOptions)
if err != nil {
  fmt.Println(foo.GetOpt1()) // 123
  fmt.Println(foo.GetOpt2()) // baz
}

opts = md.Field[1].GetOptions()
foo, err = proto.GetExtension(opts, E_FooOptions)
if err != nil {
  fmt.Println(foo.GetOpt1()) // 456
  fmt.Println(foo.GetOpt2()) // xaa
}


*Josh Humphries*
jh...@bluegosling.com

On Wed, Oct 25, 2017 at 8:38 AM, <shiyw...@redhat.com> wrote:

> the example in https://developers.google.com/protocol-buffers/docs/
> proto#customoptions below
>
> message FooOptions {
>   optional int32 opt1 = 1;
>   optional string opt2 = 2;
> }
>
> extend google.protobuf.FieldOptions {
>   optional FooOptions foo_options = 1234;
> }
>
> // usage:
> message Bar {
>   optional int32 a = 1 [(foo_options).opt1 = 123, (foo_options).opt2 = "baz"];
>   // alternative aggregate syntax (uses TextFormat):
>   optional int32 b = 2 [(foo_options) = { opt1: 456 opt2: "xaa" }];
> }
>
> assume I have a message Bar here:
>
> var msg Bar
> _, md := ForMessage()
> md.Field[0].GetOptions().String()  only a string, How do i parse that ??
>
> How to I get the exact value of Bar.b.opt1 which is 123 (default value)
>
> What I want is like this
>
> md.Field[0].GetOptions().GetOpt1()   // 123
> md.Field[0].GetOptions().GetOpt2()// baz
> md.Field[1].GetOptions().GetOpt1()   // 456
> md.Field[1].GetOptions().GetOpt2()   // xaa
>
> Can we achieve that ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Golang: issues/discussions/pull requests

[Josh Humphries]

In the past, when I've asked questions in this mailing list or hoped to
discuss details of the protobuf package for Go, I've been pointed to the
Github project.

And, for some period, I had luck having discussions on Github issues and
even getting pull requests looked at. However, lately, I've been having a
very hard time getting questions answered or fixes reviewed.

Is there something big that is diverting attention from the engineers
working on Go protobuf stuff? Any suggestions on who I can tag in a Github
issue or pull request? I understand that reviews could take time to get to
(though the backlog is not very large and the rate of incoming PRs looks
pretty low). But I can't even get a reply to any question or remark, much
less a review.

Is there another communication channel I should be trying to use? Maybe
Slack?


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Are wrapper.proto messages automatically packed / unpacked with

[Josh Humphries]

It sounds like you are observing the expected behavior regarding how these
wrapper types are marshalled/unmarshalled to/from JSON:
https://developers.google.com/protocol-buffers/docs/proto3#json

They get "unboxed" in JSON, as do various other well-known types such as
structs, timestamps, and durations.


----
*Josh Humphries*
jh...@bluegosling.com

On Tue, Oct 24, 2017 at 5:54 AM, Jack Wootton <jackwoot...@gmail.com> wrote:

> I'm using wrappers.proto in my own API that sits behind Google's
> Extensible Service Proxy
> <https://github.com/cloudendpoints/endpoints-tools>. The proxy
> trans-codes JSON <-> ProtoBuf. Code for trans-coding is
>
> https://github.com/cloudendpoints/esp/blob/ee24633d0e6837f7d7fa05302f4ea6
> bd419a7d8e/src/api_manager/utils/marshalling.cc#L53. For example:
>
>
> Status ProtoToJson(const Message& message, std::string* result, int
> options) {
>  ::google::protobuf::util::JsonPrintOptions json_options;
>  if (options & JsonOptions::PRETTY_PRINT) {
>json_options.add_whitespace = true;
>  }
>  if (options & JsonOptions::OUTPUT_DEFAULTS) {
>json_options.always_print_primitive_fields = true;
>  }
>  // TODO: Skip going to bytes and use ProtoObjectSource directly.
>  ::google::protobuf::util::Status status = ::google::protobuf::util::Bina
> ryToJsonString( GetTypeResolver(), GetTypeUrl(message),
> message.SerializeAsString(),
>result, json_options);
>return Status::FromProto(status);
>  }
>
> Note the call to message.SerializeAsString()
>
> I've noticed my web client receives any wrapper messages, without their
> wrapper. For example, my server constructs a message
>
> message NumberOptions {
> google.protobuf.Int32Value minimum = 1;
> google.protobuf.Int32Value maximum = 2;
> }
>
> with the following values:
>
> minimum {
>   value: 1
> }
> maximum {
>   value: 100
> }
>
> However, after my message passes through the ESP (and is serislised to
> JSON), my client receives:
>
> minimum: 1
> maximum: 100
>
> Does SerializeAsString automatically unpack any wrapper messages? If so,
> can I also send JSON:
>
> minimum: 1
> maximum: 100
>
> ..and expect wrappers to be created (if the message is defined to use
> wrappers), i.e.
>
> message NumberOptions {
> google.protobuf.Int32Value minimum = 1;
> google.protobuf.Int32Value maximum = 2;
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to generate google.protobuf.Type object from a file

[Josh Humphries]

On Fri, Oct 20, 2017 at 9:05 PM, 'Adam Cozzette' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> I've CC'd Jisi who can confirm this, but I believe we actually went in a
> different direction and did not implement the type server idea. In practice
> we use the type_url field as an identifier but there is no actual server
> returning Type objects.
>

I've heard this a couple of times before, that the type server stuff was
abandoned. Could the docs be updated to reflect this? They still indicate
that type definitions will be downloadable from servers via the type URL.


>
> On Thu, Oct 19, 2017 at 11:52 AM,  wrote:
>
>> Hi All,
>>
>> The documentation for google.protobuf.Any says that an HTTP GET on the
>> URL should return a Type object in binary. However, I can't find any
>> documentation on how create a Type object from a protobuf definition file.
>> Given a compiled Message, how can I get a Type?
>>
>> Thanks,
>> Pradeep
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to generate google.protobuf.Type object from a file

[Josh Humphries]

FWIW, I've implemented this logic in Go in a library called protoreflect,
with the intention that it could be used for the type server functionality,
as documented.

Take a look at the methods MessageAsPType and EnumAsPType in MessageRegistry:
https://godoc.org/github.com/jhump/protoreflect/dynamic/msgregistry


*Josh Humphries*
jh...@bluegosling.com

On Thu, Oct 19, 2017 at 2:52 PM, <pradeep...@gmail.com> wrote:

> Hi All,
>
> The documentation for google.protobuf.Any says that an HTTP GET on the URL
> should return a Type object in binary. However, I can't find any
> documentation on how create a Type object from a protobuf definition file.
> Given a compiled Message, how can I get a Type?
>
> Thanks,
> Pradeep
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Nested extensions generate invalid code

[Josh Humphries]

That looks like a clear-cut bug in protoc-gen-go to me:
https://github.com/golang/protobuf/issues/new

I suspect there's some issue where it recognizes it is inside a proto3
message, so it generates scalar fields without a pointer. But only proto2
messages are allowed to have extensions (in this case, descriptor.proto is
still syntax = "proto2"), so it needs to special-case extension fields.



----
*Josh Humphries*
jh...@bluegosling.com

On Mon, Sep 25, 2017 at 5:05 AM, Tomoyuki Saito <aocch...@gmail.com> wrote:

> When trying to use extensions with a parent message, a build error happens
> with a message like "cannot convert nil to type bool."
>
> In generator.go
> <https://github.com/golang/protobuf/blob/130e6b02ab059e7b717a096f397c5b60111cae74/protoc-gen-go/generator/generator.go#L1678>,
> I'm expecting the extension fields should reach needsStar()
> <https://github.com/golang/protobuf/blob/130e6b02ab059e7b717a096f397c5b60111cae74/protoc-gen-go/generator/generator.go#L1678>,
> but actually they won't because they have a parent and the parent's
> syntax is proto3
> <https://github.com/golang/protobuf/blob/130e6b02ab059e7b717a096f397c5b60111cae74/protoc-gen-go/generator/generator.go#L1674>
> .
> Is this working as expected? Or, is this a bug?
>
> Versions
>
> protoc version: libprotoc 3.3.2
> golang/protobuf: using the current master https://github.com/
> golang/protobuf/tree/130e6b02ab059e7b717a096f397c5b60111cae74
>
> Commands for reproducing an issue
> ~/go/src/github.com/occho/protobuf-test/test$ pwd
> /go/src/github.com/occho/protobuf-test/test
> ~/go/src/github.com/occho/protobuf-test/test$ ls
> test.proto
> ~/go/src/github.com/occho/protobuf-test/test$ protoc
> --go_out=${GOPATH}/src test.proto
> ~/go/src/github.com/occho/protobuf-test/test$ go build
> # github.com/occho/protobuf-test/test
> ./test.pb.go:41:23: cannot convert nil to type bool
> ./test.pb.go:50:25: cannot convert nil to type string
> ./test.pb.go:59:24: cannot convert nil to type int32
>
>
> File: test.proto
> syntax = "proto3";
>
> package test;
>
> import "google/protobuf/descriptor.proto";
>
> option go_package = "github.com/occho/protobuf-test/test";
>
> message Namespace {
>   extend google.protobuf.FieldOptions {
> bool namespace_bool = 50001;
> string namespace_string = 50002;
> int32 namespace_int32 = 50003;
>   }
> }
>
> extend google.protobuf.FieldOptions {
>   bool my_bool = 51001;
>   string my_string = 51002;
>   int32 my_int32 = 51003;
> }
>
> Generated file: test.pb.go
> // Code generated by protoc-gen-go. DO NOT EDIT.
> // source: test.proto
>
> /*
> Package test is a generated protocol buffer package.
>
> It is generated from these files:
> test.proto
>
> It has these top-level messages:
> Namespace
> */
> package test
>
> import proto "github.com/golang/protobuf/proto"
> import fmt "fmt"
> import math "math"
> import google_protobuf "github.com/golang/protobuf/
> protoc-gen-go/descriptor"
>
> // Reference imports to suppress errors if they are not otherwise used.
> var _ = proto.Marshal
> var _ = fmt.Errorf
> var _ = math.Inf
>
> // This is a compile-time assertion to ensure that this generated file
> // is compatible with the proto package it is being compiled against.
> // A compilation error at this line likely means your copy of the
> // proto package needs to be updated.
> const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
>
> type Namespace struct {
> }
>
> func (m *Namespace) Reset(){ *m = Namespace{} }
> func (m *Namespace) String() string{ return
> proto.CompactTextString(m) }
> func (*Namespace) ProtoMessage()   {}
> func (*Namespace) Descriptor() ([]byte, []int) { return fileDescriptor0,
> []int{0} }
>
> var E_Namespace_NamespaceBool = {
> ExtendedType:  (*google_protobuf.FieldOptions)(nil),
> ExtensionType: (bool)(nil),
> Field: 50001,
> Name:  "test.Namespace.namespace_bool",
> Tag:   "varint,50001,opt,name=namespace_bool,json=namespaceBool",
> Filename:  "test.proto",
> }
>
> var E_Namespace_NamespaceString = {
> ExtendedType:  (*google_protobuf.FieldOptions)(nil),
> ExtensionType: (string)(nil),
> Field: 50002,
> Name:  "test.Namespace.namespace_string",
> Tag:   "bytes,50002,opt,name=namespace_string,json=
> namespaceString",
> Filename:  "test.proto",
> }
>
> var E_Namespace_NamespaceInt32 = {
> ExtendedType:  (*google_protobuf.FieldOpti

Re: [protobuf] How does this .proto file result in this generated code?

[Josh Humphries]

This code is emitted by the GRPC plugin for Java. For Go, all code gen is a
plugin (protoc-gen-go
<https://github.com/golang/protobuf/tree/master/protoc-gen-go>) and it
includes the GRPC functionality. For Java, the core protoc knows how to
emit the normal generated code, but you need the right protoc plugin
<https://github.com/grpc/grpc-java/tree/master/compiler> to generate
GRPC-related Java stuff -- like service interfaces and methods for
registering a server implementation.


*Josh Humphries*
jh...@bluegosling.com

On Fri, Jul 14, 2017 at 3:26 PM, Laird Nelson <ljnel...@gmail.com> wrote:

> (I *believe* this question is about protocol buffers, and not gRPC, but I
> may be wrong.)
>
> There is a .proto file that exists as part of the Helm project:
> https://github.com/kubernetes/helm/blob/v2.5.0/_
> proto/hapi/rudder/rudder.proto
>
> As part of Helm's build process, a Go binding is generated here:
> https://github.com/kubernetes/helm/blob/v2.5.0/
> pkg/proto/hapi/rudder/rudder.pb.go
>
> I had a question about this function: https://github.com/
> kubernetes/helm/blob/v2.5.0/pkg/proto/hapi/rudder/rudder.pb.go#L536
>
> How does that function come into being given the .proto file?
>
> For background, I've done the Java equivalent (generated bindings in Java
> <https://microbean.github.io/microbean-helm/apidocs/hapi/services/rudder/package-summary.html>
> instead of Go, off the same .proto files).  Why doesn't the analogous
> Java method show up here, given that I'm generating off the same sources?
>
> Thanks,
> Best,
> Laird
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] pros and cons to upgrade protobuf v3

[Josh Humphries]

You can use the latest version but still use a corpus files syntax="proto2"
files.

Marking required fields as optional is not the only difference (and, in
proto2, that is not a particularly safe operation -- you must be certain
you've updated *all* readers of those messages with the new definition
*before* any writers actually choose to omit those fields).

If you are using default values or extensions, your migration to proto3
syntax will be trickier as neither is supported. Also, proto3 doesn't
provide accessor methods to detect whether a field is present or not (it is
implicitly absent if it has the "zero value" for its type).

Luckily, you can start using the latest protoc and then slowly migrate. The
proto compiler does allow a mix of proto2 and proto3 syntax files -- proto3
files can import proto2 files (though I can't recall if the other direction
is allowed -- I think it is though, to support custom options that might
reference messages defined in proto3). One path for migration is to start
using proto3 for all new files. This gives you optionality as far as how
and when to convert the rest.

If you don't use default values or extensions, converting a proto2 syntax
file to proto3 is quite simple. But, depending on the language you use, it
will impact application code that interacts with the generated code.
(Proto3 files do not generate pointers in the structs for scalar fields,
for example.) Also note that custom options *are* still supported, but
since they rely on extensions they must be defined in a file with proto2
syntax.


*Josh Humphries*
jh...@bluegosling.com

On Fri, Jul 14, 2017 at 7:33 AM, hce h <jupiter@gmail.com> wrote:

> Hi,
>
> I had a debate with my colleagues regarding to upgrade protobuf v3 because
> I'd like to use MessageToJsonString and JsonStringToMessage for GUI
> development of web application and mobile app, but my colleagues are not
> convinced, they argued GO can do it very easily and they are worrying about
> to break the systems down as our protobuf v2 codes are used crossing many
> systems. I always think it is logical to use the latest version of the
> software which should fix bugs and add more features, in addition to that,
> I don't like v2 required field and many of our codes have problems when the
> process returns an error message before reaching the destination, the error
> message was not able to fill in all required fields caused libprotobufbuf
> broken the system daemon, I have recently changed many required fields to
> optional, I thought protobuf v3 should be more stable than the v2, but I am
> also new to protobuf v3, appreciate your comments.
>
> Thank you.
>
> Kind regards,
>
> -j
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] [golang] API provides no way to access unknown extensions

[Josh Humphries]

For messages generates from syntax="proto2" files, there is a way to access
unrecognized fields (as the stream of encoded bytes) via the
XXX_unrecognized field. But there is no similar way to access unrecognized
fields that happened to be in an extension range.

If I were to add an issue to the github project to remedy this (and maybe
even send a pull request), would it get serious consideration? Or is it
intentional to keep such details completely inaccessible?


*More background:*

I have a library, protoreflect
<https://godoc.org/github.com/jhump/protoreflect/dynamic>, that supports
runtime-dynamic messages, using descriptors to define the schemas. One
thing I want to be able to do is to re-parse the descriptors using a custom
registry of runtime-defined extensions (for custom options). But I don't
seem to be able to dig these out of an already-parsed descriptor proto.

*The use case*: I am building a dynamic GRPC stub that works by querying
the server for its descriptors (using service reflection). I am providing a
CLI that allows users to pass in JSON that gets parsed into the appropriate
request message, serialized to the binary format, and sent in a GRPC
request. Similar happens for de-serializing the responses and then dumping
the message contents in a human-readable text form.

In order to support custom options, I'd like to be able to efficiently
extract the unrecognized extensions from descriptors and re-parse them
using custom logic, that uses extension field descriptors that were also
downloaded from the same server.

For now, it looks like I need to marshal the options message to bytes, and
then I could unmarshal it into a dynamic message (I already have support
for unmarshalling with a custom registry of extensions). But I was hoping
for something more efficient -- like being able to only do this for the
unrecognized extensions. (I can already do this for other unrecognized
fields thanks to the export XXX_unrecognized field that gets added to
generated proto2 message structs).

Thoughts?


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Replacing 'extensions' in proto3

[Josh Humphries]

On Mon, Jun 5, 2017 at 4:48 PM, R.C.  wrote:

> Hi Bo,
>
> Thanks for your response. Would using 'Any' in this fashion change the
> format on the wire? How can I edit the proto file to maintain the format on
> the wire (maintaining backward compatibility with older message decoders)?
>

Yes, using an `Any` field changes the wire representation. If you are
trying to maintain binary compatibility in your move to proto3, you will
have to replace extensions with actual fields with the same type and tag on
the Vehicles message:

message Vehicles {
  optional Car car = 100;
  optional Bike bike = 200;
}

You would also then need to re-structure your files, like moving the
definition of Vehicle into its own file, so you don't accidentally
introduce any import cycles.



>
> Thanks,
> rc
>
> On Monday, June 5, 2017 at 1:21:03 PM UTC-7, Bo Yang wrote:
>>
>> message Vehicles {
>>   Any extending = 1;
>> }
>>
>> // cpp code ==
>> Car car = new Car();
>> UnpackAny(vehicles.extending, );
>>
>> On Mon, Jun 5, 2017 at 11:48 AM R.C.  wrote:
>>
>>> Hi,
>>>
>>> I am looking to move an application written with proto2 to use proto3. I
>>> understand 'extensions' is disabled in proto3 and am looking for some
>>> guidance to change the below to proto3:
>>>
>>>
>>> 
>>> common.proto:
>>>
>>> message Vehicles {
>>> extensions 1 to max
>>> }
>>>
>>> 
>>> car.proto:
>>>
>>> import "common.proto";
>>>
>>> message Car {
>>> string type = 1;
>>> string color = 2;
>>> }
>>>
>>> extend Vehicles {
>>> optional Car CarExt  = 100;
>>> }
>>>
>>> 
>>>
>>> bike.proto:
>>>
>>> import "common.proto";
>>>
>>> message Bike {
>>> string type = 1;
>>> string color = 2;
>>> }
>>>
>>> extend Vehicles {
>>> optional Bike BikeExt = 200;
>>> }
>>> 
>>>
>>> Since 'optional' is default i can remove the 'optional' keyword but I am
>>> not sure how 'Any' can be used in such scenario in the place of
>>> 'extensions'.
>>>
>>> Thanks,
>>> rc
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] new protoc keywords?

[Josh Humphries]

I think the right approach to something like this would be to use the
existing service definition support and write a protoc plugin that does the
code gen. To layer on extra things -- for concepts/features that OSGI
supports but that can't be modeled just with proto service definitions --
 you could probably use custom options for services and methods. More
complicated things may require clever modeling with the request/response
types (and maybe custom message and field options, too). These primitives
can likely take you pretty far, without introducing any special syntax or
keywords in protos.



*Josh Humphries*
jh...@bluegosling.com

On Tue, Apr 25, 2017 at 6:53 PM, Scott Lewis <scottsle...@gmail.com> wrote:

> I've been happily using protocol buffers to implement Java <-> Python
> interaction using OSGi Remote Services [1,2].
>
> Remote Services is a spec that extends OSGi services to out-of-process
> access.   A nice thing about the Remote Services spec is that it is
> transport-independent...allowing arbitrary wire protocols and
> serialization formats via a distribution provider.  For example, I've
> created distribution providers based upon Py4j [1], and grcp/protocol
> buffers [2].  The distribution provider(s) can be selected at runtime for
> individual remote services.
>
> One thought I've had would be to allow remote services to be declared in
> .proto files, and generate the osgi remote service a) metadata; b) impl
> code; c) client code.
>
> I think one way to do this would be to implement protoc language additions
> for handling the service keyword, as grcp does.  Another way would be to
> add something like osgiremoteservice keyword to protoc for handling
> declaring and generating osgi remote services across multiple languages.
>
> Question:  Is there a way to add a new keyword to protoc?   I would prefer
> not to just provide an impl of the existing proto3 'service' keyword,
> because an osgi remote service has other/additional behavior (e.g.
> dynamics, async access, etc) based upon the OSGi service registry.
>
> Thanks,
>
> Scott
>
> [1] https://github.com/ECF/Py4j-RemoteServicesProvider
>
> [2] https://github.com/ECF/grpc-RemoteServicesProvider
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Type URLs and Any types

[Josh Humphries]

On Thu, Apr 20, 2017 at 2:03 PM, Feng Xiao <xiaof...@google.com> wrote:

>
>
> On Wed, Apr 19, 2017 at 5:25 PM, Josh Humphries <jh...@bluegosling.com>
> wrote:
>
>> The protobuf docs for the Any type
>> <https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Any>
>> talk about the URL being a real URL, where an HTTP GET request will reply
>> with encoded google.protobuf.Type message. The doc further states that
>> URLs with no scheme assume HTTPS (and runtime support that I've seen looks
>> like it intentionally leaves out the scheme when generating type URLs).
>>
>> However, the given URL examples for all well-known types don't work. Any
>> GET request for https://type.googleapis.com/ fails with a 404.
>> The actual example in the Any doc is for google.protobuf.Duration, and
>> https://type.googleapis.com/google.protobuf.Duration simply does not
>> work.
>>
>> I don't necessarily need these to work, but I have written stuff for
>> resolving types per the doc and have no out-of-the-box URLs I can play with
>> to test it.. It seems like an issue that none of the documented examples
>> actually work.
>>
> Can you create an github issue and give more context about how you plan to
> use it? Right now the type service doesn't exist because we find no use
> case of it for our API service stack.
>

I'm actually writing something for dynamic services. It currently uses GRPC
service reflection, but I was looking at supporting the google.protobuf.Api
type, too (first converting Api, Type, and Enum objects into equivalent
descriptors). I was also looking at implementing dynamic type resolution
for Any types the same way: download the Type object, generate a message
descriptor from it, and then use that to de-serialize the payload.

I don't need to implement this for my project. I just figured I would flesh
these things out, for the sake of completeness, since it's part of the spec
in the published documentation on
https://developers.google.com/protocol-buffers/.

So I'd appreciate the docs being updated to reflect reality. If reality is
that there will be no dynamic type resolution strategy and that clients of
Any messages must have all message URLs statically registered in order to
parse the payloads, that's fine. It just needs to be documented that way.


>
>
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Type URLs and Any types

[Josh Humphries]

The protobuf docs for the Any type
<https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Any>
talk about the URL being a real URL, where an HTTP GET request will reply
with encoded google.protobuf.Type message. The doc further states that URLs
with no scheme assume HTTPS (and runtime support that I've seen looks like
it intentionally leaves out the scheme when generating type URLs).

However, the given URL examples for all well-known types don't work. Any
GET request for https://type.googleapis.com/ fails with a 404. The
actual example in the Any doc is for google.protobuf.Duration, and
https://type.googleapis.com/google.protobuf.Duration simply does not work.

I don't necessarily need these to work, but I have written stuff for
resolving types per the doc and have no out-of-the-box URLs I can play with
to test it.. It seems like an issue that none of the documented examples
actually work.


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Enum values and proto3 Any types

[Josh Humphries]

That totally works. I don't actually need anything at the moment. But I was
looking at the spec and implementing some stuff around it -- like for
downloading and resolving type schemas from type URLs -- and noticed it as
an omission. The main omission is that nothing describes the contents of
google.protobuf.Field#type_url when the field's kind is enum. If it is an
encoded google.protobuf.Enum (which is the only thing that really makes
sense), it would be nice to state so in the spec. It might also be nice to
clearly call out in the doc for google.protobuf.Any that scalar types must
be boxed via wrappers (and enum types must be boxed via custom message
types).

FWIW, a generic wrapper type for an enum is certainly possible and would
look something like this:

message EnumValue {
  // The URL, optionally omitted scheme, for the enumeration type.
  type_url string
  valueint32
}

Unfortunately, EnumValue is a name conflict with the existing type that
describes an element in the enumeration. Maybe WrappedEnumValue?



*Josh Humphries*
jh...@bluegosling.com

On Mon, Apr 17, 2017 at 12:54 PM, Adam Cozzette <acozze...@google.com>
wrote:

> I think the right solution for putting an enum or primitive type in an Any
> field would be to use a wrapper type like you mentioned. Since there's no
> real way to create a generic wrapper for all enums, you would probably need
> to create a specific wrapper for each enum you want to store in an Any
> field. Does that work for your situation?
>
> On Thu, Apr 13, 2017 at 7:31 PM, Josh Humphries <jh...@bluegosling.com>
> wrote:
>
>> I have a question about Any types in proto3. I'm guessing they aren't
>> quite fully-fleshed out since I haven't found a lot in the way of doc or
>> real support in the Java and Go implementations.
>>
>> My question: can you use enums or scalar types as values of an Any field?
>> If so, how do they look?
>>
>> For scalar types, I can understand if the answer is to use the wrapper
>> well-known types instead. *But* there doesn't appear to be a wrapper for
>> enum values. There is a well-known type named google.protobuf.EnumValue,
>> however it is part of the "schema description" well-known types and doesn't
>> carry enough data for wrapping an enum value (mainly, it's missing a
>> reference to its enum type, e.g. a type URL).
>>
>> It would be nice if the Any message's type_url field could be a URL that
>> returned an enum description, not just a message description. But the docs
>> state that the contents at that URL are an encoded google.protobuf.Type,
>> which describes a message and does not support enum types. Should it
>> instead have a single one-of field that can be an enum or message
>> description?
>>
>> Similarly, the type_url field for google.protobuf.Field is in a similar
>> conundrum. The doc states it is *"t**he field type URL, without the
>> scheme, for message or enumeration types"*, but nothing really talks
>> about enumeration types. The naive assumption is that the URL contents are
>> an enum description (e.g. an encoded google.protobuf.Enum). But if
>> that's the case, I could try to put an enum type into an Any field and
>> indicate that same URL. I then get into trouble because I'd be looking to
>> parse a google.protobuf.Type, in order to understand the Any message
>> contents, but actually get back the bytes for a google.protobuf.Enum.
>>
>> That doesn't seem sound.
>>
>> 
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Enum values and proto3 Any types

[Josh Humphries]

I have a question about Any types in proto3. I'm guessing they aren't quite
fully-fleshed out since I haven't found a lot in the way of doc or real
support in the Java and Go implementations.

My question: can you use enums or scalar types as values of an Any field?
If so, how do they look?

For scalar types, I can understand if the answer is to use the wrapper
well-known types instead. *But* there doesn't appear to be a wrapper for
enum values. There is a well-known type named google.protobuf.EnumValue,
however it is part of the "schema description" well-known types and doesn't
carry enough data for wrapping an enum value (mainly, it's missing a
reference to its enum type, e.g. a type URL).

It would be nice if the Any message's type_url field could be a URL that
returned an enum description, not just a message description. But the docs
state that the contents at that URL are an encoded google.protobuf.Type,
which describes a message and does not support enum types. Should it
instead have a single one-of field that can be an enum or message
description?

Similarly, the type_url field for google.protobuf.Field is in a similar
conundrum. The doc states it is *"t**he field type URL, without the scheme,
for message or enumeration types"*, but nothing really talks about
enumeration types. The naive assumption is that the URL contents are an
enum description (e.g. an encoded google.protobuf.Enum). But if that's the
case, I could try to put an enum type into an Any field and indicate that
same URL. I then get into trouble because I'd be looking to parse a
google.protobuf.Type, in order to understand the Any message contents, but
actually get back the bytes for a google.protobuf.Enum.

That doesn't seem sound.


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: why protobuf optional field does not take null

[Josh Humphries]

In a past life, we used a protoc plugin that just added these setOrClear*
to every builder for each non-repeated field (as well as getOrNull* methods
for every message). The standard generated java code has insertion points
that make this a fairly trivial plugin to write. We also experimented with
adding extra accessors/setters using Guava's Optional (and would probably
use Java 8's Optional today), but that just exploded the API too much.

The generated code is already so voluminous! Adding more and more cruft to
every generated proto wasn't awesome. However, I do agree that the
setOrClear* class of methods was the most useful. Without them, you can't
use method-chaining when building a message where one of the inputs you're
conveying is nullable.

// Kind of stinky:
MyMessage.Builder b = MyMessage.newBuilder()
  .setFoo(foo)
  .setBar(bar)
  .setBaz(baz);
if (frobnitz != null) {
  b.setFrobnitz(frobnitz)
}
return b.build();


// Better:
return MyMessage.newBuilder()
  .setFoo(foo)
  .setBar(bar)
  .setBaz(baz)
  .setOrClearFrobnitz(frobnitz)
  .build();



*Josh Humphries*
jh...@bluegosling.com

On Mon, Mar 20, 2017 at 7:53 AM, Subin Sebastian <subinsebast...@gmail.com>
wrote:

> 8 years later, still the only solution I have is,
>
> *class MyImprovedClass extends MyClass { // MyClass is protoc generated*
> *public void setOrClearFoo(String value) {*
> *if(value != null) setFoo(value);*
> *else clearFoo();*
> *}*
> *}*
>
> On Tuesday, December 1, 2009 at 7:04:26 AM UTC+5:30, Kenton Varda wrote:
>>
>> I'm still open to the idea of adding setOrClearFoo() methods that clear
>> the value if given null -- but only if it can be shown that this will not
>> significantly increase the size of compiled .class files.
>>
>> On Mon, Nov 30, 2009 at 3:42 PM, Henner Zeller <henner...@googlemail.com>
>> wrote:
>>
>>> Hi,
>>> On Mon, Nov 30, 2009 at 15:16, dp <decimus...@gmail.com> wrote:
>>> > Is this still true? Or is there some way to tell PB that a field can
>>> > potentially have null values?
>>>
>>> No change here.
>>>
>>> Problem is that 'null' and 'cleared' are semantically different things.
>>> Sometimes in the software world they're used to mean the same thing
>>> but protocol buffers shouldn't follow that, because that notion is not
>>> universal if you think platform independently. In C++ for instance,
>>> you won't deal with a 'NULL' string (unlike Java, no pointers, pardon,
>>> references are used there).
>>> Note, that protocol buffers have as well the notion of 'default
>>> values'. So if you set a field to 'null' but meaning to clear it, then
>>> accessing that field will return the default value. This is confusing
>>> at best.
>>>
>>> Given this and the platform independent notion I think it is better to
>>> be explicit and not have 'null' magically meaning clearing a field.
>>>
>>> > If not, any suggestions on how to deal with fields that could
>>> > potentially have null values? (Besides the obvious - don't set them,
>>> > d'oh)
>>> >
>>> > *Asking 'cuz I have a fairly big class (~100 fields) and testing if
>>> > each field is null before setting them in the PB Builder object is
>>> > somewhat of a pain.
>>>
>>> A one-line if (foo != null) b.setFoo(foo) else b.clearFoo(); might be
>>> a bit cluttersome sometimes. OTOH, it is pretty straightforward and
>>> quick to read.
>>>
>>> Maybe you can hack up some dynamic proxy that wraps around the setters
>>> (haven't done Java for some time, probably this only works with
>>> interfaces). Or alternatively you chaneg the higher level logic to
>>> just _not_ set a field instead of setting it to 'null' when it really
>>> means that it doesn't want to set it ...
>>>
>>> -h
>>>
>>> >
>>> > On Oct 29, 10:29 am, Kenton Varda <ken...@google.com> wrote:
>>> >> Protocol buffers has no concept of null.  Fields cannot be set to
>>> null.  You
>>> >> can *clear* a field, like:
>>> >>
>>> >>   builder.clearParamCountry();
>>> >>
>>> >> This sets the field back to its default value (the empty string,
>>> unless you
>>> >> declared some other default).  Also, hasParamCountry() will return
>>> false
>>> >> until you set it to something else.
>>> >>
>>> >> But you cannot set any field to null -- this will throw a
>>> >> NullPointerException.
>

Re: [protobuf] serializing new proto message with old client (Java)

[Josh Humphries]

Are you using syntax = "proto3" in your file? I believe the docs state that
proto3 does not require unknown fields to be preserved when a message is
de-serialized and re-serialized. So this behavior, IIRC, only works for
messages generated from proto2 files.

----
*Josh Humphries*
jh...@bluegosling.com

On Sun, Mar 5, 2017 at 2:10 PM, jkg589ih <shra...@gmail.com> wrote:

> Hi,
>
> Developers guide says:
>
> "Messages created by your new code can be parsed by your old code: old
> binaries simply ignore the new field when parsing. However, the unknown
> fields are not discarded, and if the message is later serialized, the
> unknown fields are serialized along with it – so if the message is passed
> on to new code, the new fields are still available."
>
>
> I'm interested in exactly this scenario - old client gets a message from
> the new client, changes something, and sends it back.
>
> I gave it a try, and what I saw is that new proto fields are stored in
> unknownFields. But when I do build(), these fields are discarded.
>
> So when the message is serialized, the new fields are gone. Am I doing it
> wrong ?
>
>
> I found that I could simply serialize the message, and then separately
> serialize the unknownFieldSet, so they are concatenated to the message.
>
> But this doesn't feel like the right way. Is it ?
>
>
> I also found this: https://github.com/bivas/protobuf-java-format/pull/26,
> but last activity was a year ago.
>
>
> What's the right way of having the old client serialize a message
> including fields it doesn't understand ?
>
>
> Thanks,
>
> Alex
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] go protobuf: "rich" descriptors

['Josh Humphries' via Protocol Buffers]

The Java
<https://github.com/google/protobuf/blob/7f3e23707122f31ebfa58b7280bd56cbe77cb44e/java/core/src/main/java/com/google/protobuf/Descriptors.java>
 and C++
<https://github.com/google/protobuf/blob/7f3e23707122f31ebfa58b7280bd56cbe77cb44e/src/google/protobuf/descriptor.h>
protobuf
implementations both provide "rich" descriptors: library types that wrap
the generated DescriptorProto messages to make them much easier to work
with and, generally, more useful.

I've implemented something similar for Go
<https://github.com/jhump/protoreflect/blob/4df185295ba66e94f4fd8e8f60f6a34be0cf875b/desc/descriptor.go>.
My intended use is to be use them alongside the GRPC reflection
<https://github.com/jhump/protoreflect/blob/4df185295ba66e94f4fd8e8f60f6a34be0cf875b/grpcreflect/clientreflection.go>
API
(which, from a Go client, isn't hugely useful without a better/more usable
form of the descriptors).

Is this something that would be welcome in the Go protobuf implementation?
I am happy to open a pull request and contribute it. But I wasn't sure if
the maintainers would prefer it remain a third-party library instead of
integrating it into the core library.


*Josh Humphries*
jh...@bluegosling.com

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] extending field and message options

[Josh Humphries]

All fully-qualified symbols must be a unique for an invocation of protoc.
In the generated code, you can refer to the generated extension (just like
referring to generated messages, enums, and services), and if their
fully-qualified names weren't unique, it would result in compilation errors
in the generated code.


*Josh Humphries*
jh...@bluegosling.com

On Mon, Mar 13, 2017 at 4:15 PM, Arpit Baldeva <abald...@gmail.com> wrote:

> Hi,
>
> I have my proto file like following.
>
> package example;
>
> message ExFieldOptions
> {
>   map<string, string> meta_data = 1;
> }
>
> extend google.protobuf.FieldOptions
> {
> ExFieldOptions exOptions = 4245;
> }
>
> extend google.protobuf.MessageOptions
> {
> ExFieldOptions exOptions = 4245;
> }
>
> When I run it through ProtoC (for C++ code generation), I run into an error
>
> "exOptions" is already defined in "example".
>
> Is this a bug? Or is it required that extension name need to be unique
> across options?
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to resolve a custom option in a protocol buffer FileDescriptor

[Josh Humphries]

When you parse the serialized descriptor proto, you must supply an
ExtensionRegistry and make sure that custom option is registered therein.


*Josh Humphries*
jh...@bluegosling.com

On Tue, Mar 7, 2017 at 11:02 AM, Bill Smith <william.m.sm...@gmail.com>
wrote:

> I'm using protocol buffers 2.5 with Java. I have a proto file that defines
> a custom option. Another proto file uses that custom option. If I persist
> the corresponding FileDescriptorProto's and then read them and convert
> them to FileDescriptors, the reference to the custom option is manifested
> as unknown field. How do I cause that custom option to be resolved
> correctly?
>
> Here's the code. I have two .proto files. protobuf-options.proto looks
> like this:
>
> package options;
> import "google/protobuf/descriptor.proto";
>
> option java_package = "com.example.proto";
> option java_outer_classname = "Options";
>
> extend google.protobuf.FieldOptions {
> optional bool scrub = 5;}
>
> The imported google/protobuf/descriptor.proto is exactly the
> descriptor.proto that ships with Protocol Buffers 2.5.
>
> example.proto looks like this:
>
> package example;
> option java_package = "com.example.protos";
> option java_outer_classname = "ExampleProtos";
> option optimize_for = SPEED;
> option java_generic_services = false;
> import "protobuf-options.proto";
>
> message M {
> optional int32 field1 = 1;
> optional string field2 = 2 [(options.scrub) = true];}
>
> As you can see, field2 references the custom option defined by
> protobuf-options.proto.
>
> The following code writes a binary-encoded version of all three protos to
> /tmp:
>
> package com.example;
> import com.google.protobuf.ByteString;import 
> com.google.protobuf.DescriptorProtos.FileDescriptorProto;import 
> com.google.protobuf.Descriptors.FileDescriptor;import 
> com.example.protos.ExampleProtos;
> import java.io.FileOutputStream;import java.io.OutputStream;
> /**
>  *
>  */public class PersistFDs {
> public void persist(final FileDescriptor fileDescriptor) throws Exception 
> {
> System.out.println("persisting "+fileDescriptor.getName());
> try (final OutputStream outputStream = new 
> FileOutputStream("/tmp/"+fileDescriptor.getName())) {
> final FileDescriptorProto fileDescriptorProto = 
> fileDescriptor.toProto();
> final ByteString byteString = fileDescriptorProto.toByteString();
> byteString.writeTo(outputStream);
> }
> for (final FileDescriptor dependency : 
> fileDescriptor.getDependencies()) {
> persist(dependency);
> }
> }
> public static void main(String[] args) throws Exception {
> final PersistFDs self = new PersistFDs();
> self.persist(ExampleProtos.getDescriptor());
> }}
>
> Finally, the following code loads those those protos from /tmp, converts
> them back into FileDescriptors, and then checks for the custom option on
> field2:
>
> package com.example;
> import com.google.protobuf.ByteString;import 
> com.google.protobuf.DescriptorProtos.FileDescriptorProto;import 
> com.google.protobuf.Descriptors.FieldDescriptor;import 
> com.google.protobuf.Descriptors.FileDescriptor;import 
> com.google.protobuf.UnknownFieldSet.Field;
> import java.io.FileInputStream;import java.io.InputStream;
> /**
>  *
>  */public class LoadFDs {
> public FileDescriptorProto loadProto(final String filePath) throws 
> Exception {
> try (final InputStream inputStream = new FileInputStream(filePath)) {
> final ByteString byteString = ByteString.readFrom(inputStream);
> final FileDescriptorProto result = 
> FileDescriptorProto.parseFrom(byteString);
>
> return result;
> }
> }
>
> public static void main(final String[] args) throws Exception {
> final LoadFDs self = new LoadFDs();
>
> final FileDescriptorProto descriptorFDProto = 
> self.loadProto("/tmp/google/protobuf/descriptor.proto");
> final FileDescriptorProto optionsFDProto = 
> self.loadProto("/tmp/protobuf-options.proto");
> final FileDescriptorProto fakeBoxcarFDProto = 
> self.loadProto("/tmp/example.proto");
>
> final FileDescriptor fD = FileDescriptor.buildFrom(descriptorFDProto, 
> new FileDescriptor[0]);
> final FileDescriptor optionsFD = 
> FileDescriptor.buildFrom(optionsFDProto, new FileDescriptor[] { fD });
> final FileDescriptor fakeBoxcarFD = 
> FileDescriptor.buildFrom(fakeBoxcarFDProto, new FileDescriptor[] { option

Re: [protobuf] What are well-known types?

[Josh Humphries]

What do you think about creating issues for them on the protobuf github
project and then include links to the design docs?

Also, the addition of boxed types seems to be back-pedaling a bit on the
decision to make zero and absent indistinguishable for primitive types.
This seems to be a concession that the proto2 behavior was useful, but at
the expense of on-the-wire bloat for present values.



*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

On Tue, Feb 23, 2016 at 6:34 PM, 'Feng Xiao' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> On Tue, Feb 23, 2016 at 12:55 PM, Zellyn <zel...@gmail.com> wrote:
>
>> There are increasing numbers of references to "well-known" types in
>> protos. For instance, I see changes in the Go implementation to support
>> them.
>> There were passing references in release notes in this group.
>>
>> However, the main protobuf site includes no narrative explanation that I
>> can find.
>>
>> The idea of a few well-known types to represent "Boxed" values since
>> proto3 removes the ability to null out fields makes sense, but the only
>> documentation I could find, in the reference section of the protobuf site
>> at
>> https://developers.google.com/protocol-buffers/docs/reference/google.protobuf,
>> includes all sorts of things like Struct, Method, Mixin, etc. that are
>> entirely unclear.
>>
>> Is there a conversation happening somewhere that I'm missing, or is it
>> Google-internal but not documented outside yet?
>>
> Sorry, the documentation for proto3 is not complete yet. We are still
> working on improving it. There are a couple of internal design docs for
> well-known types, but they are not yet ready to be included in the public
> developer guide.
>
>
>>
>> Thanks,
>>
>> Zellyn
>>
>> ps - the reintroduction of message types for primitives rather undermines
>> my belief in the arguments for removing optional fields in proto3 in the
>> first place. I'd like to give the benefit of the doubt to the folks
>> designing proto3: is the thinking articulated clearly somewhere?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Unhelpful compiler warning when a file has no syntax declaration

[Josh Humphries]

The protoc compiler emits a warning message whenever it encounters a proto
file that has no syntax level declared. But the warning doesn't indicate
the filename. I'd like our builds to be warning-free. But with a make task
that builds *lots* of protos at once, it's a total nuisance to find the
offending file. The protoc compiler really should just indicate the name of
the file in the warning.

I opened a pull request a few days ago to fix this:
https://github.com/google/protobuf/pull/1217
It hasn't gotten any attention.

What's the best policy for trying to submit changes? Should I start a
discussion on this group first, to give visibility to it? Or is there some
time window I should allow (few days?) before expecting any
attention/comments on a pull request?

Thanks in advance.


*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: [grpc-io] Re: GRPC and Ruby, Proto language level 2

[Josh Humphries]

I think that would be great. Thanks, Tim!


*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

On Fri, Jan 15, 2016 at 11:49 AM, Tim Emiola <temi...@google.com> wrote:

> Hi Eric,
>
> We never actually published it - I'll clean up the patch I was using and
> push it to a branch on my github fork of beefcake.  Will that do ?
>
> Tim
>
>
> On Tuesday, January 12, 2016 at 4:06:09 AM UTC-8, Eric Zundel Ayers wrote:
>>
>> Hi Josh (Haberman),
>>
>> Do you have any pointers for us in the patches for the beefcake gem?  We
>> were hoping we might be able to use that with our development while we
>> worked on the protoc patches.
>>
>> We tried using the stock beefcake gem and didn't get very far with our
>> internal proto definitions.   The README for the current open source
>> version says that it doesn't support imports, which is a pretty fundamental
>> restriction!
>>
>> -Eric.
>>
>> On Sunday, January 10, 2016 at 5:44:52 PM UTC-5, Josh Haberman wrote:
>>>
>>> Hi Josh,
>>>
>>> On Friday, January 8, 2016 at 3:17:11 PM UTC-8, Josh Humphries wrote:
>>>>
>>>> I am actively pursuing moving a proprietary RPC mechanism to GRPC. And
>>>> the languages currently needed are Ruby, Go, and Java.
>>>>
>>>> I didn't realize before trying a few things today that the newly added
>>>> languages/platforms in proto3 (e.g. Ruby) cannot generate code via protoc
>>>> for language level 2. This looks like it could be a serious blocker for
>>>> what I and my team want to do.
>>>>
>>>> Is this stance, to not support language level 2, a hard stance? We
>>>> cannot (nor do we even want to) move all of our protos to language level 3.
>>>> We use extensions.
>>>>
>>>> *If we wanted to contribute a patch to protoc to support language level
>>>> 2 in Ruby, would it be accepted?*
>>>>
>>>
>>> Yes most definitely! We have wanted to add this for a while, we just
>>> hadn't gotten to it yet.
>>>
>>> This would involve (as you say):
>>>
>>> - augmenting the schema DSL so you can define proto2 types in addition
>>> to proto3 ones.
>>> - adding field presence for proto2 primitive types (with hasbits)
>>> - implementing proto2 enum semantics
>>> - adding extension support
>>>
>>> I would love to talk over what the implementation would look like on
>>> this. If you're down, we can talk more details.
>>>
>>> Thanks,
>>> Josh
>>>
>>>> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/ea0ac25e-52d7-41d5-8e6b-ec65b266c03a%40googlegroups.com
> <https://groups.google.com/d/msgid/grpc-io/ea0ac25e-52d7-41d5-8e6b-ec65b266c03a%40googlegroups.com?utm_medium=email_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] GRPC and Ruby, Proto language level 2

[Josh Humphries]

I am actively pursuing moving a proprietary RPC mechanism to GRPC. And the
languages currently needed are Ruby, Go, and Java.

I didn't realize before trying a few things today that the newly added
languages/platforms in proto3 (e.g. Ruby) cannot generate code via protoc
for language level 2. This looks like it could be a serious blocker for
what I and my team want to do.

Is this stance, to not support language level 2, a hard stance? We cannot
(nor do we even want to) move all of our protos to language level 3. We use
extensions.

*If we wanted to contribute a patch to protoc to support language level 2
in Ruby, would it be accepted?*

We could likely make due with the changes in semantics for default values
and unknown fields/enum values. So we could just run the existing
language-level-3 code-gen against language level 2 source files. But we'd
need to implement some mechanism to support extensions.

If such a patch would never be accepted, would it be feasible for us to use
a 3rd party proto gem for Ruby with GRPC? I'm quite familiar with the Java
implementation of GRPC and know that it has a layer that is independent
from protos (which, in Java, enables the use of Nano instead of the
standard Java runtime). This could be used to support different mechanisms
for marshalling data, including even alternate IDLs and formats (thrift,
JSON, etc).

Is the C implementation (and Ruby bindings) architected this way, too? If
so, it seems like it should be possible to graft non-protoc generated
protos onto GRPC.


*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Issue 491 in protobuf: Optional field throws NullPointerException when not set values.

[Josh Humphries]

As a work-around, you can implement a simple protoc plugin that will
generate additional null-friendly methods into the classes already created
by the Java codegen built into protoc.

That's actually what we've done at Square. As you point out, the
null-unfriendliness is particularly burdensome with builders, for cases
where you want to use method-chaining but also conditionally set or clear a
field.

Our generated messages include getOrNull*() accessor fields (so they return
null if unset instead of the field's default value) and the builders have
setOrClear*() setter methods (that clear the field when null is passed in).




*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

On Mon, Jun 8, 2015 at 4:04 AM, Andreas V andreas.v...@gmail.com wrote:

 I know its an old thread but i use protobuf3 with Java and it has the same
 NPE behavior with .set(null-reference). Is their any chance to change this?
 an option to generating java classes would be nice. It is absolutly
 annoying to null-check on every .set method. This will lead to more errors
 in code and expand it inadequate due to chained-setters are not possible. :(

 Am Mittwoch, 29. Mai 2013 23:50:59 UTC+2 schrieb prot...@googlecode.com:


 Comment #6 on issue 491 by xiaof...@google.com: Optional field throws
 NullPointerException when not set values.
 http://code.google.com/p/protobuf/issues/detail?id=491

 We don't have the intention to change to the current behavior. Passing
 null
 is likely to be a programming error. Allowing it might do more harm than

 good.

 --
 You received this message because this project is configured to send all

 issue notifications to this address.
 You may adjust your notification preferences at:
 https://code.google.com/hosting/settings

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Proto 3... release date?

[Josh Humphries]

The latest version I saw, IIRC, was still an alpha (alpha 2). What is the
planned release date for protobuf v3.0?


*Josh Humphries*
Manager, Shared Systems  |  Platform Engineering
Atlanta, GA  |  678-400-4867
*Square* (www.squareup.com)

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.