Re: [protobuf] Proposal: a mechanism to deal with sensitive/redacted fields in string output
On Fri, May 10, 2019 at 6:06 PM Adam Cozzette wrote: > I asked for feedback about this proposal within Google and unfortunately > it sounds like there's not a lot of support for accepting this kind of > change. The general feedback I got was that it's best to simply avoid > printing out any protos at all if they might contain sensitive information. > This kind of feature might provide a false sense of security and encourage > developers to print out protos that haven't necessarily been fully > annotated with the sensitive field option. There was some agreement that in > Java it is particularly easy to print stringified protos by accident, but > it seems that ideally we would want to disable that behavior entirely > rather than redacting particular fields. > For what it's worth, when discussing this before, some folks on the Protobuf Team mentioned that the parts of Google that deal with financial transactions actually have something similar to our proposal. Or at least something that accomplishes the same goal. > I gather that Square is already relying on this functionality in its > internal protobuf fork, so I would say if it helps we could probably at > least try to refactor things to minimize the complexity of maintaining that > behavior difference. > That would be super-helpful. I'll have to catch up on the current state of protobuf library code and figure out how to allow convenient interception. Zellyn -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAMQ7dq5jRg9TT0uMfiOreD0F1K4d1EXtUEqP9SkE5aC1jt9UKA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Proposal: a mechanism to deal with sensitive/redacted fields in string output
Hey there, just checking in from Paternity Leave. Last I heard, the Protobuf team was not opposed to the idea, but thought it would be relatively invasive and thus unlikely to get accepted. I believe that it will hardly be invasive at all, so I think the most-likely-to-succeed course of action is to provide a related set of minimal patches to (a) protoc, (b) the C protobuf implementation, (c) the Java implementation and (d) the Go implementation, supporting sensitive fields. Unfortunately I just haven't found the time to do all that. Zellyn On Fri, Apr 19, 2019 at 5:12 PM Josh Humphries wrote: > I don't think there has been any movement on this, but I'd like to ping > the thread again. > > I am still a strong proponent of standard metadata in the proto source and > descriptor to indicate sensitive things. I also think it's a truly wise > idea to use that information to trigger auto-redaction of sensitive data > when serializing message in certain contexts (such as "stringification", > like when emitting the data into a debug log message). > > At FullStory, we've built some interesting tools for using protobufs and > gRPC. Most recent is a web UI for gRPC: > https://github.com/fullstorydev/grpcui. I would *love* to be able to add > features to some of these tools that can be aware of sensitive data in > messages that they handle and even give the data special treatment in some > ways. For the web UI, there's a few interesting things that a client might > choose to do in the face of sensitive data. For one, if that page includes > an analytics/recording tool, it could make sure the elements in the DOM > that may contain sensitive data are excluded from analysis/recording. > Another: it could refuse to send an RPC over an insecure connection if any > of the message's fields are marked as sensitive. > > Outside of the web UI, especially in a world with GDPR, there are all > kinds of tools that could be built that care deeply about whether data is > sensitive or not. For example, static analysis/linters that make sure > sensitive data is treated in a suitably sensitive manner. Without a > standard way to denote that in the proto, open-source tools in this vein > become more complicated to use. If open-source tools must provide their own > custom option (as an example), and then someone wants to use multiple such > tools on their codebase, they end up having to redundantly define multiple > options on each sensitive field. (Admittedly, this is a manufactured > hypothetical.) > > Anyhow, where do things stand with this enhancement request? Is there > anything I can do to stoke the fire and get some attention on it? > > > *Josh Humphries* > jh...@bluegosling.com > > > On Wed, Aug 22, 2018 at 10:01 AM Zellyn wrote: > >> Apologies for the long delay, but I got radically reassigned at work, so >> I haven't had much time to work on this. But it keeps niggling at me, >> because I hate our internal protobuf forks so much. >> >> Here is the proposal: Proto Proposal: a “sensitive” field option >> <https://docs.google.com/document/d/18WI8zN7rk6R0jXW1iC8LDYz7LJ0OrUOTKMGD7nyEnFs> >> . >> >> Zellyn >> >> On Wednesday, February 22, 2017 at 12:10:14 PM UTC-5, Adam Cozzette wrote: >>> >>> Hi Zellyn, this sounds like a reasonable idea. As the next step could >>> you perhaps write up a short proposal with more details on what exactly it >>> would mean for a field to be redacted? To me it seems like the important >>> thing would be to make sure it's clear how redacted fields are supposed to >>> be behave in each situation (i.e. when they should be dropped or not), so >>> that there's no uncertainty about when they're dropped and when they're >>> preserved. (For example, we might say that they're never shown when a proto >>> is implicitly stringified but maybe preserved in all other situations?) We >>> might also need to be careful to get this right for all languages early; >>> even if there's some language where we don't care about redaction for now, >>> it will be hard to change later without making a breaking change. >>> >>> On Thu, Feb 16, 2017 at 1:45 PM, zellyn via Protocol Buffers < >>> prot...@googlegroups.com> wrote: >>> >>>> There are many ways that protocol buffers might be stringified into >>>> logs, accidentally or on purpose, printed in stack traces, etc. The >>>> built-in behavior stringifies the entire protobuf recursively, including >>>> all field data. >>>> >>>> At Square, we deal with payments, and often have data of varying >>>> sensitivity in protobuf field
Re: [protobuf] Re: Question about json_name and protoc
Yeah, it's precisely the fact that protoc populates that field regardless of whether it was specified that I consider bad. https://github.com/protocolbuffers/protobuf/issues/5587 I'm probably way too late to get it fixed, though: it looks like 3.1.0 introduced that behavior :-( Zellyn On Wednesday, January 16, 2019 at 12:23:17 PM UTC-5, Josh Humphries wrote: > > Hey, Zellyn, > I don't think it's possible for protoc-gen-go to do what you're asking. > IIRC, protoc itself always populates this field in the field descriptor. So > I don't think a plugin can actually make this distinction -- at least not > without a change to protoc. And the downside to a change to protoc is that > it would require all code-gen plugins (and/or runtime libraries that > implement the JSON encoding) to have redundant logic for computing the > canonical JSON name from the proto name. > > > *Josh Humphries* > jh...@bluegosling.com > > > On Wed, Jan 16, 2019 at 10:10 AM Zellyn > > wrote: > >> A quick followup: a reason that including a json name for every field in >> the descriptor is bad: it makes it impossible to tell whether an explicit >> json_name was specified or not. So it would be impossible to give jsonpb a >> mode (which it needs, imho) where it uses the protobuf field names as-is, >> unless an explicit json_name was specified. >> >> Or am I understanding it wrong? >> >> Zellyn >> >> >> On Tuesday, January 15, 2019 at 12:47:31 PM UTC-5, Zellyn wrote: >>> >>> Hey folks, >>> >>> We're trying to update the version of protoc we use for compiling our Go >>> protobufs. It appears that at some point between 3.0.0 and 3.6.0, protoc >>> started populating the `json_name` field option for *every* field, >>> whether the option was specified or not. Is that expected? If so, could >>> anyone help me track down the commit that did it? >>> >>> Thanks, >>> >>> Zellyn >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com . >> To post to this group, send email to prot...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Question about json_name and protoc
A quick followup: a reason that including a json name for every field in the descriptor is bad: it makes it impossible to tell whether an explicit json_name was specified or not. So it would be impossible to give jsonpb a mode (which it needs, imho) where it uses the protobuf field names as-is, unless an explicit json_name was specified. Or am I understanding it wrong? Zellyn On Tuesday, January 15, 2019 at 12:47:31 PM UTC-5, Zellyn wrote: > > Hey folks, > > We're trying to update the version of protoc we use for compiling our Go > protobufs. It appears that at some point between 3.0.0 and 3.6.0, protoc > started populating the `json_name` field option for *every* field, > whether the option was specified or not. Is that expected? If so, could > anyone help me track down the commit that did it? > > Thanks, > > Zellyn > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Question about json_name and protoc
Hey folks, We're trying to update the version of protoc we use for compiling our Go protobufs. It appears that at some point between 3.0.0 and 3.6.0, protoc started populating the `json_name` field option for *every* field, whether the option was specified or not. Is that expected? If so, could anyone help me track down the commit that did it? Thanks, Zellyn -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Proposal: a mechanism to deal with sensitive/redacted fields in string output
Apologies for the long delay, but I got radically reassigned at work, so I haven't had much time to work on this. But it keeps niggling at me, because I hate our internal protobuf forks so much. Here is the proposal: Proto Proposal: a “sensitive” field option <https://docs.google.com/document/d/18WI8zN7rk6R0jXW1iC8LDYz7LJ0OrUOTKMGD7nyEnFs> . Zellyn On Wednesday, February 22, 2017 at 12:10:14 PM UTC-5, Adam Cozzette wrote: > > Hi Zellyn, this sounds like a reasonable idea. As the next step could you > perhaps write up a short proposal with more details on what exactly it > would mean for a field to be redacted? To me it seems like the important > thing would be to make sure it's clear how redacted fields are supposed to > be behave in each situation (i.e. when they should be dropped or not), so > that there's no uncertainty about when they're dropped and when they're > preserved. (For example, we might say that they're never shown when a proto > is implicitly stringified but maybe preserved in all other situations?) We > might also need to be careful to get this right for all languages early; > even if there's some language where we don't care about redaction for now, > it will be hard to change later without making a breaking change. > > On Thu, Feb 16, 2017 at 1:45 PM, zellyn via Protocol Buffers < > prot...@googlegroups.com > wrote: > >> There are many ways that protocol buffers might be stringified into logs, >> accidentally or on purpose, printed in stack traces, etc. The built-in >> behavior stringifies the entire protobuf recursively, including all field >> data. >> >> At Square, we deal with payments, and often have data of varying >> sensitivity in protobuf fields, which we'd like to be elided from >> stringified output. >> >> We use an internal fork of protoc to handle a custom field option, >> "redacted", and have also patched the stringification code to print >> "[REDACTED]" for those fields. We do the same in Go, and in the C >> implementation (for Ruby). >> >> Last year, we chatted with the protobuf team, and they were sympathetic >> to our use case (in fact, they mentioned that the part of Google that deals >> with payments has something similar internally: I think that's where the >> "sensitive" name came from). I'd like to get that discussion rolling again. >> >> We'd like to see one of the following happen (in decreasing order of >> awesomeness for us): >> >>- upstreaming of the "redacted" field option, and modification of the >>runtimes to elide redacted fields when stringifying >>- introduction of generic interception points to selectively override >>default stringification behavior in Java, Go, and Ruby (at least). >>- addition of a "SerializeToString" or equivalent method, and removal >>of default full-stringification behavior of the toString (Java), String >>(Go), etc. - that way you only serialize on purpose >> - many tests rely on string comparison, even though nobody is >> supposed to rely on it being stable - perhaps the default behavior >> could >> compute a hash? >> >> Josh Humpries (who now works at Fullstory) created a proposal >> <https://github.com/google/protobuf/issues/1160> a while back, but it >> didn't go anywhere. I reached out to the protobuf team, and Damien Neil >> suggested that this group was the appropriate place to propose such changes. >> >> Bikeshed away! >> >> Zellyn >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com . >> To post to this group, send email to prot...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Proposal: a mechanism to deal with sensitive/redacted fields in string output
There are many ways that protocol buffers might be stringified into logs, accidentally or on purpose, printed in stack traces, etc. The built-in behavior stringifies the entire protobuf recursively, including all field data. At Square, we deal with payments, and often have data of varying sensitivity in protobuf fields, which we'd like to be elided from stringified output. We use an internal fork of protoc to handle a custom field option, "redacted", and have also patched the stringification code to print "[REDACTED]" for those fields. We do the same in Go, and in the C implementation (for Ruby). Last year, we chatted with the protobuf team, and they were sympathetic to our use case (in fact, they mentioned that the part of Google that deals with payments has something similar internally: I think that's where the "sensitive" name came from). I'd like to get that discussion rolling again. We'd like to see one of the following happen (in decreasing order of awesomeness for us): - upstreaming of the "redacted" field option, and modification of the runtimes to elide redacted fields when stringifying - introduction of generic interception points to selectively override default stringification behavior in Java, Go, and Ruby (at least). - addition of a "SerializeToString" or equivalent method, and removal of default full-stringification behavior of the toString (Java), String (Go), etc. - that way you only serialize on purpose - many tests rely on string comparison, even though nobody is supposed to rely on it being stable - perhaps the default behavior could compute a hash? Josh Humpries (who now works at Fullstory) created a proposal <https://github.com/google/protobuf/issues/1160> a while back, but it didn't go anywhere. I reached out to the protobuf team, and Damien Neil suggested that this group was the appropriate place to propose such changes. Bikeshed away! Zellyn -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Question about Go protobufs and import_prefix
Note that I'm not actually generating protobufs into a vendor folder; what I described has the same problems with or without vendoring. On Tuesday, July 12, 2016 at 4:33:33 PM UTC-4, Damien Neil wrote: > > Simpler than maintaining a fork, though. > > The more convenient alternative would be for protoc to understand Go > vendor directories, but that would require putting some Go-specific logic > in protoc itself. > > On Tue, Jul 12, 2016 at 8:00 AM, Zellyn Hunter <zel...@gmail.com > > wrote: > >> Yeah I can do anything with a wrapper. But to the extent that our >> concerns and structure are normal, it would be a shame to need a wrapper. >> >> On Tue, Jul 12, 2016, 8:54 AM Ross Light <li...@google.com > >> wrote: >> >>> Zellyn and I talked a little bit at Gophercon about this. I think some >>> kind of wrapper is in order for his use case: it may actually be a filter >>> between protoc and protoc-gen-go. >>> >>> On Tue, Jul 12, 2016, 8:49 AM Damien Neil <dn...@google.com >>> > wrote: >>> >>>> If I'm following this correctly, your core problem is that protoc >>>> doesn't understand vendored paths as used by the go tool. For example: >>>> >>>> - You have a proto file located in src/square/up/protos/square.proto. >>>> - square.proto imports " >>>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto". >>>> - timestamp.proto is actually located in "src/square/up/vendor/ >>>> github.com/.../timestamp.proto". >>>> >>>> You can't run "protoc --go_out=. square/up/protos/square.proto", >>>> because protoc won't be able to locate the vendored copy of >>>> timestamp.proto. If it did, everything would work correctly without >>>> setting >>>> an import_prefix. >>>> >>>> It would be nice for Go users if protoc did understand vendored paths. >>>> Failing that, however, I think you can get what you want with a very small >>>> wrapper around protoc. The wrapper can copy a proto and all its >>>> dependencies into a temp directory (pulling dependencies from vendored >>>> paths as necessary), run protoc there, and copy the result back. e.g., in >>>> the case of the above example: >>>> >>>> $ mkdir -p /tmp/x/square/up/protos >>>> $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos >>>> $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp >>>> $ cp square/up/vendor/ >>>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto >>>> github.com/golang/protobuf/ptypes/timestamp >>>> $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto >>>> $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos >>>> >>>> The generated .go files will reference non-vendored paths, which the Go >>>> compiler will resolve to the correct vendored directory. >>>> >>>> Does that seem like it would work for you? >>>> >>>> On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter <zel...@gmail.com >>>> > wrote: >>>> >>>>> Usually not too tricky. The problem is that protos and most other >>>>> programming languages prohibit circular imports at the file level, and Go >>>>> does it at the package level. The fact that protoc and the other >>>>> languages >>>>> are okay with a topology ensures there is a valid Go repackaging that >>>>> will >>>>> work. >>>>> >>>>> Zellyn >>>>> >>>>> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers < >>>>> prot...@googlegroups.com > wrote: >>>>> >>>>>> I don't know the background of the Go import system or go_package >>>>>> option. However this statement concerns me a little: >>>>>> >>>>>> > We have to use go_package to reorganize things that are fine in >>>>>> Java/Ruby, but would be circular package imports in Go. >>>>>> >>>>>> Is this implying that certain .proto files would generate invalid Go >>>>>> code until you manually insert some go_package statements? How tricky is >>>>>> it >>>>>> to do this manual untangling? >>>>>> >>>>>> >>>>>> On Friday, July 1, 2016
Re: [protobuf] Question about Go protobufs and import_prefix
Yeah I can do anything with a wrapper. But to the extent that our concerns and structure are normal, it would be a shame to need a wrapper. On Tue, Jul 12, 2016, 8:54 AM Ross Light <li...@google.com> wrote: > Zellyn and I talked a little bit at Gophercon about this. I think some > kind of wrapper is in order for his use case: it may actually be a filter > between protoc and protoc-gen-go. > > On Tue, Jul 12, 2016, 8:49 AM Damien Neil <dn...@google.com> wrote: > >> If I'm following this correctly, your core problem is that protoc doesn't >> understand vendored paths as used by the go tool. For example: >> >> - You have a proto file located in src/square/up/protos/square.proto. >> - square.proto imports " >> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto". >> - timestamp.proto is actually located in "src/square/up/vendor/ >> github.com/.../timestamp.proto". >> >> You can't run "protoc --go_out=. square/up/protos/square.proto", because >> protoc won't be able to locate the vendored copy of timestamp.proto. If it >> did, everything would work correctly without setting an import_prefix. >> >> It would be nice for Go users if protoc did understand vendored paths. >> Failing that, however, I think you can get what you want with a very small >> wrapper around protoc. The wrapper can copy a proto and all its >> dependencies into a temp directory (pulling dependencies from vendored >> paths as necessary), run protoc there, and copy the result back. e.g., in >> the case of the above example: >> >> $ mkdir -p /tmp/x/square/up/protos >> $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos >> $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp >> $ cp square/up/vendor/ >> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto >> github.com/golang/protobuf/ptypes/timestamp >> $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto >> $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos >> >> The generated .go files will reference non-vendored paths, which the Go >> compiler will resolve to the correct vendored directory. >> >> Does that seem like it would work for you? >> >> On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter <zel...@gmail.com> wrote: >> >>> Usually not too tricky. The problem is that protos and most other >>> programming languages prohibit circular imports at the file level, and Go >>> does it at the package level. The fact that protoc and the other languages >>> are okay with a topology ensures there is a valid Go repackaging that will >>> work. >>> >>> Zellyn >>> >>> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers < >>> protobuf@googlegroups.com> wrote: >>> >>>> I don't know the background of the Go import system or go_package >>>> option. However this statement concerns me a little: >>>> >>>> > We have to use go_package to reorganize things that are fine in >>>> Java/Ruby, but would be circular package imports in Go. >>>> >>>> Is this implying that certain .proto files would generate invalid Go >>>> code until you manually insert some go_package statements? How tricky is it >>>> to do this manual untangling? >>>> >>>> >>>> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote: >>>>> >>>>> Oh wow. Thanks for thinking about it so carefully! >>>>> >>>>> I should mention: before you could actually specify full package paths >>>>> to the Go proto plugin, earlier versions of our wrapper ( >>>>> https://github.com/square/goprotowrap) just forcibly manhandled >>>>> things into the shape we wanted by explicitly specifying -M parameter >>>>> import rewrites for *every single* import, and renaming the output >>>>> files into place. >>>>> >>>>> So it's definitely possible to shape things as you want. However, I >>>>> would prefer to find a solution that works with the official plugin, >>>>> because I don't believe our setup is remarkable or unusual: as GRPC sees >>>>> an >>>>> increase in use as a cross-language RPC mechanism, everyone else is going >>>>> to struggle with the same things. >>>>> >>>>> Zellyn >>>>> >>>>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light <li...@google.com> wrote: >>>>&g
Re: [protobuf] Question about Go protobufs and import_prefix
Usually not too tricky. The problem is that protos and most other programming languages prohibit circular imports at the file level, and Go does it at the package level. The fact that protoc and the other languages are okay with a topology ensures there is a valid Go repackaging that will work. Zellyn On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers < protobuf@googlegroups.com> wrote: > I don't know the background of the Go import system or go_package option. > However this statement concerns me a little: > > > We have to use go_package to reorganize things that are fine in > Java/Ruby, but would be circular package imports in Go. > > Is this implying that certain .proto files would generate invalid Go code > until you manually insert some go_package statements? How tricky is it to > do this manual untangling? > > > On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote: >> >> Oh wow. Thanks for thinking about it so carefully! >> >> I should mention: before you could actually specify full package paths to >> the Go proto plugin, earlier versions of our wrapper ( >> https://github.com/square/goprotowrap) just forcibly manhandled things >> into the shape we wanted by explicitly specifying -M parameter import >> rewrites for *every single* import, and renaming the output files into >> place. >> >> So it's definitely possible to shape things as you want. However, I would >> prefer to find a solution that works with the official plugin, because I >> don't believe our setup is remarkable or unusual: as GRPC sees an increase >> in use as a cross-language RPC mechanism, everyone else is going to >> struggle with the same things. >> >> Zellyn >> >> On Fri, Jul 1, 2016 at 7:32 PM Ross Light <li...@google.com> wrote: >> >>> Sorry for the delayed response! I've written about 3-4 draft replies >>> over this week that I've discarded because they don't meet your >>> requirements. I'll think about it more over the long weekend and get back >>> to you. >>> >>> Cheers, >>> -Ross >>> >>> >>> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo <jie...@google.com> wrote: >>> >>>> +Ross who is working on Go protobuf. >>>> >>>> Ross, do you have any comments or suggestions? >>>> >>>> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn <zel...@gmail.com> wrote: >>>> >>>>> Hi folks, >>>>> >>>>> Apologies in advance for the complex email. It takes a bit of >>>>> explaining to set up what we're having trouble with. >>>>> >>>>> I would like to lay out how we generate protos in Go (using our >>>>> unfortunately forked version of the protoc plugin), and ask for >>>>> suggestions >>>>> as to how to accomplish the same thing with the unforked version. In the >>>>> process, I wish to ask questions about the post-vendor-experiment utility >>>>> of the current behavior of import_prefix, hopefully generating more >>>>> discussion than I managed to here >>>>> <https://github.com/golang/protobuf/issues/181>. >>>>> >>>>> Background: >>>>> >>>>>- We have protos spread around all over the place: in our Go >>>>>monorepo, in our Java monorepo, and in miscellaneous Ruby repos. >>>>>- We collect them all into a single directory before generating Go >>>>>code. Call it $COLLECT_DIR >>>>>- We have to use go_package to reorganize things that are fine in >>>>>Java/Ruby, but would be circular package imports in Go. >>>>>- For historical reasons, our Go repo is checked out to >>>>>$GOPATH/src/square/up. I know it's a little weird, but it's logically >>>>>equivalent to any other location. Pretend "square/up" is "square3" if >>>>> you >>>>>like… :-) >>>>>- Our monorepo's vendor folder lives at >>>>>$GOPATH/src/square/up/vendor/... >>>>>- We want to generate protos into $GOPATH/src/square/up/protos to >>>>>keep them all in one place. >>>>>- Most of our protos are in package "squareup.*", but we have some >>>>>third-party protos that have their own packages. Or none (eg. >>>>>nanopb >>>>><https://github.com/PIlin/nanopb/blob/master/generator/nanopb.proto> >>>>&
Re: [protobuf] Question about Go protobufs and import_prefix
Oh wow. Thanks for thinking about it so carefully! I should mention: before you could actually specify full package paths to the Go proto plugin, earlier versions of our wrapper ( https://github.com/square/goprotowrap) just forcibly manhandled things into the shape we wanted by explicitly specifying -M parameter import rewrites for *every single* import, and renaming the output files into place. So it's definitely possible to shape things as you want. However, I would prefer to find a solution that works with the official plugin, because I don't believe our setup is remarkable or unusual: as GRPC sees an increase in use as a cross-language RPC mechanism, everyone else is going to struggle with the same things. Zellyn On Fri, Jul 1, 2016 at 7:32 PM Ross Light <li...@google.com> wrote: > Sorry for the delayed response! I've written about 3-4 draft replies over > this week that I've discarded because they don't meet your requirements. > I'll think about it more over the long weekend and get back to you. > > Cheers, > -Ross > > > On Fri, Jul 1, 2016 at 3:11 PM Jie Luo <jie...@google.com> wrote: > >> +Ross who is working on Go protobuf. >> >> Ross, do you have any comments or suggestions? >> >> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn <zel...@gmail.com> wrote: >> >>> Hi folks, >>> >>> Apologies in advance for the complex email. It takes a bit of explaining >>> to set up what we're having trouble with. >>> >>> I would like to lay out how we generate protos in Go (using our >>> unfortunately forked version of the protoc plugin), and ask for suggestions >>> as to how to accomplish the same thing with the unforked version. In the >>> process, I wish to ask questions about the post-vendor-experiment utility >>> of the current behavior of import_prefix, hopefully generating more >>> discussion than I managed to here >>> <https://github.com/golang/protobuf/issues/181>. >>> >>> Background: >>> >>>- We have protos spread around all over the place: in our Go >>>monorepo, in our Java monorepo, and in miscellaneous Ruby repos. >>>- We collect them all into a single directory before generating Go >>>code. Call it $COLLECT_DIR >>>- We have to use go_package to reorganize things that are fine in >>>Java/Ruby, but would be circular package imports in Go. >>>- For historical reasons, our Go repo is checked out to >>>$GOPATH/src/square/up. I know it's a little weird, but it's logically >>>equivalent to any other location. Pretend "square/up" is "square3" if you >>>like… :-) >>>- Our monorepo's vendor folder lives at >>>$GOPATH/src/square/up/vendor/... >>>- We want to generate protos into $GOPATH/src/square/up/protos to >>>keep them all in one place. >>>- Most of our protos are in package "squareup.*", but we have some >>>third-party protos that have their own packages. Or none (eg. nanopb >>><https://github.com/PIlin/nanopb/blob/master/generator/nanopb.proto> >>>). >>>- Right now we also have copies of WKT protos in $COLLECT_DIR, but >>>I'd love to just import them from their vendor folder locations. >>> >>> *What we do now: f**ull go_package including "protos" folder in >>> protobufs* >>> >>> Add full package and path go_package to every protobuf: >>> eg. For a file with package "squareup.testing", we set option go_package >>> = "square/up/squareup/protos/testing". >>> (We haven't actually added these declarations everywhere yet: this is >>> what our forked plugin does: if go_package doesn't specify package path, we >>> use square/up/protos + slash-separated-package. But it's about the same >>> thing. Our fork was created long before go_package gained the ability to >>> actually specify output package path) >>> >>> Cons: >>> >>>- We generate into the output directory of $GOPATH/src, which is >>>terrible: we could accidentally generate stuff into any arbitrary go >>>package in our GOPATH. :-( >>>- We're writing an implementation detail (where we currently store >>>protos) into all the proto files >>> >>> *What I would like to do:* >>> >>> Generate into an output directory of $GOPATH/src/square/up/protos, and >>> use the import_prefix option to ensure that things get imported from the >>> right place. >>> >>>
[protobuf] Re: Default Values vs Missing Values
protos are also gaining "Well-known Types", some of which are "boxed" (Message) versions of the primitive types: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf I believe the actual docs on well-known types are currently Google-internal :-( Zellyn On Saturday, March 26, 2016 at 2:43:00 PM UTC-4, Yoav H wrote: > > Hi, > > I wanted ask regarding the decision to populate fields with default > values, even if they do not appear in the encoded message. > If I want to send a "patch" message, where I want to update just the > provided fields, how can I do that with protobuf (without adding IsXXXSet > for every field)? > > Why not add another type, representing a default value? > So the schematics would be, if the field is missing, it is null, and if > the field exists, but with this "missing value" type, it will get the > default value? > > Thanks, > Yoav. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] What are well-known types?
There are increasing numbers of references to "well-known" types in protos. For instance, I see changes in the Go implementation to support them. There were passing references in release notes in this group. However, the main protobuf site includes no narrative explanation that I can find. The idea of a few well-known types to represent "Boxed" values since proto3 removes the ability to null out fields makes sense, but the only documentation I could find, in the reference section of the protobuf site at https://developers.google.com/protocol-buffers/docs/reference/google.protobuf, includes all sorts of things like Struct, Method, Mixin, etc. that are entirely unclear. Is there a conversation happening somewhere that I'm missing, or is it Google-internal but not documented outside yet? Thanks, Zellyn ps - the reintroduction of message types for primitives rather undermines my belief in the arguments for removing optional fields in proto3 in the first place. I'd like to give the benefit of the doubt to the folks designing proto3: is the thinking articulated clearly somewhere? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] protoc-gen-go and internal/grpc
Any particular reason the grpc plugin for protoc-gen-go is in an internal/ subdirectory? We import it because we want to include the grpc plugin along with our custom plugins, but with the move to go 1.5, the import will break. I was considering adding a grpc or include_grpc package under protoc-gen-go/ that could be imported just to link the grpc plugin, but that seems silly: would it be reasonable to elide the /internal/ subdirectory? Zellyn -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
