Hi folks,

Apologies in advance for the complex email. It takes a bit of explaining to 
set up what we're having trouble with.

I would like to lay out how we generate protos in Go (using our 
unfortunately forked version of the protoc plugin), and ask for suggestions 
as to how to accomplish the same thing with the unforked version. In the 
process, I wish to ask questions about the post-vendor-experiment utility 
of the current behavior of import_prefix, hopefully generating more 
discussion than I managed to here 


   - We have protos spread around all over the place: in our Go monorepo, 
   in our Java monorepo, and in miscellaneous Ruby repos.
   - We collect them all into a single directory before generating Go code. 
   Call it $COLLECT_DIR
   - We have to use go_package to reorganize things that are fine in 
   Java/Ruby, but would be circular package imports in Go.
   - For historical reasons, our Go repo is checked out to 
   $GOPATH/src/square/up. I know it's a little weird, but it's logically 
   equivalent to any other location. Pretend "square/up" is "square3" if you 
   likeā€¦ :-)
   - Our monorepo's vendor folder lives at $GOPATH/src/square/up/vendor/...
   - We want to generate protos into $GOPATH/src/square/up/protos to keep 
   them all in one place.
   - Most of our protos are in package "squareup.*", but we have some 
   third-party protos that have their own packages. Or none (eg. nanopb 
   - Right now we also have copies of WKT protos in $COLLECT_DIR, but I'd 
   love to just import them from their vendor folder locations.

*What we do now: f**ull go_package including "protos" folder in protobufs*

Add full package and path go_package to every protobuf:
eg. For a file with package "squareup.testing", we set option go_package = 
(We haven't actually added these declarations everywhere yet: this is what 
our forked plugin does: if go_package doesn't specify package path, we use 
square/up/protos + slash-separated-package. But it's about the same thing. 
Our fork was created long before go_package gained the ability to actually 
specify output package path)


   - We generate into the output directory of $GOPATH/src, which is 
   terrible: we could accidentally generate stuff into any arbitrary go 
   package in our GOPATH. :-(
   - We're writing an implementation detail (where we currently store 
   protos) into all the proto files

*What I would like to do:*

Generate into an output directory of $GOPATH/src/square/up/protos, and use 
the import_prefix option to ensure that things get imported from the right 

*Why that doesn't work:*

   - import_prefix is added to *every* import, including grpc packages, the 
   context package, the proto runtime package, etc. This made sense when the 
   prefix was intended to be something like "square/up/Godep/_workspace/" (eg. 
   vendoring by import-path-rewriting), but doesn't make sense anymore.
   - import_prefix is added to imports for ALL protos, including things 
   like well-known-type protos that should just live under 

*What I'd like to get out of this:*

   - a reworking of the way import_prefix works
   - a new parameter with slightly different semantics: adds an import 
   prefix only for protos, and only for protos in a particular list of folders:
      - eg. proto_import_prefix_by_dir=$COLLECT_DIR:square/up/protos
   - an explanation of how I'm Doing it Wrong - I'm happy to change things. 
   While there's a lot of existing code, Go protos have caused me so much 
   pain, I'm willing to invest pretty significant effort.

*Why I'd like to defork*

Our fork was created before go_package could rearrange output packages (
With the recent increase in grpc-related volatility of the protobuf 
package, it's increasingly painful to maintain our fork. And since grpc 
updates are often tied to proto updates, I can't simply avoid updating it. 
Plus all the usual reasons maintaining a fork is terrible :-)

Comments, suggestions, and help welcome.


cc dsymonds, matloob at Google, Josh at Square.

You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Reply via email to