Oh wow. Thanks for thinking about it so carefully! I should mention: before you could actually specify full package paths to the Go proto plugin, earlier versions of our wrapper ( https://github.com/square/goprotowrap) just forcibly manhandled things into the shape we wanted by explicitly specifying -M parameter import rewrites for *every single* import, and renaming the output files into place.
So it's definitely possible to shape things as you want. However, I would prefer to find a solution that works with the official plugin, because I don't believe our setup is remarkable or unusual: as GRPC sees an increase in use as a cross-language RPC mechanism, everyone else is going to struggle with the same things. Zellyn On Fri, Jul 1, 2016 at 7:32 PM Ross Light <[email protected]> wrote: > Sorry for the delayed response! I've written about 3-4 draft replies over > this week that I've discarded because they don't meet your requirements. > I'll think about it more over the long weekend and get back to you. > > Cheers, > -Ross > > > On Fri, Jul 1, 2016 at 3:11 PM Jie Luo <[email protected]> wrote: > >> +Ross who is working on Go protobuf. >> >> Ross, do you have any comments or suggestions? >> >> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn <[email protected]> wrote: >> >>> Hi folks, >>> >>> Apologies in advance for the complex email. It takes a bit of explaining >>> to set up what we're having trouble with. >>> >>> I would like to lay out how we generate protos in Go (using our >>> unfortunately forked version of the protoc plugin), and ask for suggestions >>> as to how to accomplish the same thing with the unforked version. In the >>> process, I wish to ask questions about the post-vendor-experiment utility >>> of the current behavior of import_prefix, hopefully generating more >>> discussion than I managed to here >>> <https://github.com/golang/protobuf/issues/181>. >>> >>> Background: >>> >>> - We have protos spread around all over the place: in our Go >>> monorepo, in our Java monorepo, and in miscellaneous Ruby repos. >>> - We collect them all into a single directory before generating Go >>> code. Call it $COLLECT_DIR >>> - We have to use go_package to reorganize things that are fine in >>> Java/Ruby, but would be circular package imports in Go. >>> - For historical reasons, our Go repo is checked out to >>> $GOPATH/src/square/up. I know it's a little weird, but it's logically >>> equivalent to any other location. Pretend "square/up" is "square3" if you >>> like… :-) >>> - Our monorepo's vendor folder lives at >>> $GOPATH/src/square/up/vendor/... >>> - We want to generate protos into $GOPATH/src/square/up/protos to >>> keep them all in one place. >>> - Most of our protos are in package "squareup.*", but we have some >>> third-party protos that have their own packages. Or none (eg. nanopb >>> <https://github.com/PIlin/nanopb/blob/master/generator/nanopb.proto> >>> ). >>> - Right now we also have copies of WKT protos in $COLLECT_DIR, but >>> I'd love to just import them from their vendor folder locations. >>> >>> *What we do now: f**ull go_package including "protos" folder in >>> protobufs* >>> >>> Add full package and path go_package to every protobuf: >>> eg. For a file with package "squareup.testing", we set option go_package >>> = "square/up/squareup/protos/testing". >>> (We haven't actually added these declarations everywhere yet: this is >>> what our forked plugin does: if go_package doesn't specify package path, we >>> use square/up/protos + slash-separated-package. But it's about the same >>> thing. Our fork was created long before go_package gained the ability to >>> actually specify output package path) >>> >>> Cons: >>> >>> - We generate into the output directory of $GOPATH/src, which is >>> terrible: we could accidentally generate stuff into any arbitrary go >>> package in our GOPATH. :-( >>> - We're writing an implementation detail (where we currently store >>> protos) into all the proto files >>> >>> *What I would like to do:* >>> >>> Generate into an output directory of $GOPATH/src/square/up/protos, and >>> use the import_prefix option to ensure that things get imported from the >>> right place. >>> >>> *Why that doesn't work:* >>> >>> - import_prefix is added to *every* import, including grpc packages, >>> the context package, the proto runtime package, etc. This made sense when >>> the prefix was intended to be something like >>> "square/up/Godep/_workspace/" >>> (eg. vendoring by import-path-rewriting), but doesn't make sense anymore. >>> - import_prefix is added to imports for ALL protos, including things >>> like well-known-type protos that should just live under >>> $GOPATH/src/square/up/vendor/github.com/golang/protobuf/proto >>> >>> *What I'd like to get out of this:* >>> Either: >>> >>> - a reworking of the way import_prefix works >>> - a new parameter with slightly different semantics: adds an import >>> prefix only for protos, and only for protos in a particular list of >>> folders: >>> - eg. proto_import_prefix_by_dir=$COLLECT_DIR:square/up/protos >>> - an explanation of how I'm Doing it Wrong - I'm happy to change >>> things. While there's a lot of existing code, Go protos have caused me so >>> much pain, I'm willing to invest pretty significant effort. >>> >>> *Why I'd like to defork* >>> >>> Our fork was created before go_package could rearrange output packages ( >>> commit >>> <https://github.com/golang/protobuf/commit/c75fbf01dc6cb73649c4cd4326182c3e44aa9dbb>). >>> With the recent increase in grpc-related volatility of the protobuf >>> package, it's increasingly painful to maintain our fork. And since grpc >>> updates are often tied to proto updates, I can't simply avoid updating it. >>> Plus all the usual reasons maintaining a fork is terrible :-) >>> >>> Comments, suggestions, and help welcome. >>> >>> Zellyn >>> >>> cc dsymonds, matloob at Google, Josh at Square. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
