Re: [protobuf] Question about Go protobufs and import_prefix

[Zellyn]

Note that I'm not actually generating protobufs into a vendor folder; what 
I described has the same problems with or without vendoring.

On Tuesday, July 12, 2016 at 4:33:33 PM UTC-4, Damien Neil wrote:
>
> Simpler than maintaining a fork, though.
>
> The more convenient alternative would be for protoc to understand Go 
> vendor directories, but that would require putting some Go-specific logic 
> in protoc itself.
>
> On Tue, Jul 12, 2016 at 8:00 AM, Zellyn Hunter  > wrote:
>
>> Yeah I can do anything with a wrapper. But to the extent that our 
>> concerns and structure are normal, it would be a shame to need a wrapper.
>>
>> On Tue, Jul 12, 2016, 8:54 AM Ross Light  
>> wrote:
>>
>>> Zellyn and I talked a little bit at Gophercon about this. I think some 
>>> kind of wrapper is in order for his use case: it may actually be a filter 
>>> between protoc and protoc-gen-go.
>>>
>>> On Tue, Jul 12, 2016, 8:49 AM Damien Neil >> > wrote:
>>>
 If I'm following this correctly, your core problem is that protoc 
 doesn't understand vendored paths as used by the go tool. For example:

 - You have a proto file located in src/square/up/protos/square.proto.
 - square.proto imports "
 github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
 - timestamp.proto is actually located in "src/square/up/vendor/
 github.com/.../timestamp.proto".

 You can't run "protoc --go_out=. square/up/protos/square.proto", 
 because protoc won't be able to locate the vendored copy of 
 timestamp.proto. If it did, everything would work correctly without 
 setting 
 an import_prefix.

 It would be nice for Go users if protoc did understand vendored paths. 
 Failing that, however, I think you can get what you want with a very small 
 wrapper around protoc. The wrapper can copy a proto and all its 
 dependencies into a temp directory (pulling dependencies from vendored 
 paths as necessary), run protoc there, and copy the result back. e.g., in 
 the case of the above example:

 $ mkdir -p /tmp/x/square/up/protos
 $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
 $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
 $ cp square/up/vendor/
 github.com/golang/protobuf/ptypes/timestamp/timestamp.proto 
 github.com/golang/protobuf/ptypes/timestamp
 $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
 $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

 The generated .go files will reference non-vendored paths, which the Go 
 compiler will resolve to the correct vendored directory.

 Does that seem like it would work for you?

 On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter  wrote:

> Usually not too tricky. The problem is that protos and most other 
> programming languages prohibit circular imports at the file level, and Go 
> does it at the package level. The fact that protoc and the other 
> languages 
> are okay with a topology ensures there is a valid Go repackaging that 
> will 
> work.
>
> Zellyn
>
> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
> prot...@googlegroups.com > wrote:
>
>> I don't know the background of the Go import system or go_package 
>> option. However this statement concerns me a little:
>>
>> > We have to use go_package to reorganize things that are fine in 
>> Java/Ruby, but would be circular package imports in Go.
>>
>> Is this implying that certain .proto files would generate invalid Go 
>> code until you manually insert some go_package statements? How tricky is 
>> it 
>> to do this manual untangling?
>>
>>
>> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>>>
>>> Oh wow. Thanks for thinking about it so carefully!
>>>
>>> I should mention: before you could actually specify full package 
>>> paths to the Go proto plugin, earlier versions of our wrapper (
>>> https://github.com/square/goprotowrap) just forcibly manhandled 
>>> things into the shape we wanted by explicitly specifying -M parameter 
>>> import rewrites for *every single* import, and renaming the output 
>>> files into place.
>>>
>>> So it's definitely possible to shape things as you want. However, I 
>>> would prefer to find a solution that works with the official plugin, 
>>> because I don't believe our setup is remarkable or unusual: as GRPC 
>>> sees an 
>>> increase in use as a cross-language RPC mechanism, everyone else is 
>>> going 
>>> to struggle with the same things.
>>>
>>> Zellyn
>>>
>>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light >> > wrote:
>>>
 Sorry for the delayed response!  I've written about 3-4 draft 

Re: [protobuf] Question about Go protobufs and import_prefix

['Ross Light' via Protocol Buffers]

Zellyn and I talked a little bit at Gophercon about this. I think some kind
of wrapper is in order for his use case: it may actually be a filter
between protoc and protoc-gen-go.

On Tue, Jul 12, 2016, 8:49 AM Damien Neil  wrote:

> If I'm following this correctly, your core problem is that protoc doesn't
> understand vendored paths as used by the go tool. For example:
>
> - You have a proto file located in src/square/up/protos/square.proto.
> - square.proto imports "
> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
> - timestamp.proto is actually located in "src/square/up/vendor/
> github.com/.../timestamp.proto".
>
> You can't run "protoc --go_out=. square/up/protos/square.proto", because
> protoc won't be able to locate the vendored copy of timestamp.proto. If it
> did, everything would work correctly without setting an import_prefix.
>
> It would be nice for Go users if protoc did understand vendored paths.
> Failing that, however, I think you can get what you want with a very small
> wrapper around protoc. The wrapper can copy a proto and all its
> dependencies into a temp directory (pulling dependencies from vendored
> paths as necessary), run protoc there, and copy the result back. e.g., in
> the case of the above example:
>
> $ mkdir -p /tmp/x/square/up/protos
> $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
> $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
> $ cp square/up/vendor/
> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto
> github.com/golang/protobuf/ptypes/timestamp
> $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
> $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos
>
> The generated .go files will reference non-vendored paths, which the Go
> compiler will resolve to the correct vendored directory.
>
> Does that seem like it would work for you?
>
> On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter  wrote:
>
>> Usually not too tricky. The problem is that protos and most other
>> programming languages prohibit circular imports at the file level, and Go
>> does it at the package level. The fact that protoc and the other languages
>> are okay with a topology ensures there is a valid Go repackaging that will
>> work.
>>
>> Zellyn
>>
>> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
>> protobuf@googlegroups.com> wrote:
>>
>>> I don't know the background of the Go import system or go_package
>>> option. However this statement concerns me a little:
>>>
>>> > We have to use go_package to reorganize things that are fine in
>>> Java/Ruby, but would be circular package imports in Go.
>>>
>>> Is this implying that certain .proto files would generate invalid Go
>>> code until you manually insert some go_package statements? How tricky is it
>>> to do this manual untangling?
>>>
>>>
>>> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:

 Oh wow. Thanks for thinking about it so carefully!

 I should mention: before you could actually specify full package paths
 to the Go proto plugin, earlier versions of our wrapper (
 https://github.com/square/goprotowrap) just forcibly manhandled things
 into the shape we wanted by explicitly specifying -M parameter import
 rewrites for *every single* import, and renaming the output files into
 place.

 So it's definitely possible to shape things as you want. However, I
 would prefer to find a solution that works with the official plugin,
 because I don't believe our setup is remarkable or unusual: as GRPC sees an
 increase in use as a cross-language RPC mechanism, everyone else is going
 to struggle with the same things.

 Zellyn

 On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:

> Sorry for the delayed response!  I've written about 3-4 draft replies
> over this week that I've discarded because they don't meet your
> requirements.  I'll think about it more over the long weekend and get back
> to you.
>
> Cheers,
> -Ross
>
>
> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>
>> +Ross who is working on Go protobuf.
>>
>> Ross, do you have any comments or suggestions?
>>
>> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:
>>
>>> Hi folks,
>>>
>>> Apologies in advance for the complex email. It takes a bit of
>>> explaining to set up what we're having trouble with.
>>>
>>> I would like to lay out how we generate protos in Go (using our
>>> unfortunately forked version of the protoc plugin), and ask for 
>>> suggestions
>>> as to how to accomplish the same thing with the unforked version. In the
>>> process, I wish to ask questions about the post-vendor-experiment 
>>> utility
>>> of the current behavior of import_prefix, hopefully generating more
>>> discussion than I managed to 

Re: [protobuf] Question about Go protobufs and import_prefix

['Damien Neil' via Protocol Buffers]

If I'm following this correctly, your core problem is that protoc doesn't
understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.
- square.proto imports "
github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
- timestamp.proto is actually located in "src/square/up/vendor/
github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because
protoc won't be able to locate the vendored copy of timestamp.proto. If it
did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths.
Failing that, however, I think you can get what you want with a very small
wrapper around protoc. The wrapper can copy a proto and all its
dependencies into a temp directory (pulling dependencies from vendored
paths as necessary), run protoc there, and copy the result back. e.g., in
the case of the above example:

$ mkdir -p /tmp/x/square/up/protos
$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
$ cp square/up/vendor/
github.com/golang/protobuf/ptypes/timestamp/timestamp.proto
github.com/golang/protobuf/ptypes/timestamp
$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go
compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?

On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter  wrote:

> Usually not too tricky. The problem is that protos and most other
> programming languages prohibit circular imports at the file level, and Go
> does it at the package level. The fact that protoc and the other languages
> are okay with a topology ensures there is a valid Go repackaging that will
> work.
>
> Zellyn
>
> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
> protobuf@googlegroups.com> wrote:
>
>> I don't know the background of the Go import system or go_package option.
>> However this statement concerns me a little:
>>
>> > We have to use go_package to reorganize things that are fine in
>> Java/Ruby, but would be circular package imports in Go.
>>
>> Is this implying that certain .proto files would generate invalid Go code
>> until you manually insert some go_package statements? How tricky is it to
>> do this manual untangling?
>>
>>
>> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>>>
>>> Oh wow. Thanks for thinking about it so carefully!
>>>
>>> I should mention: before you could actually specify full package paths
>>> to the Go proto plugin, earlier versions of our wrapper (
>>> https://github.com/square/goprotowrap) just forcibly manhandled things
>>> into the shape we wanted by explicitly specifying -M parameter import
>>> rewrites for *every single* import, and renaming the output files into
>>> place.
>>>
>>> So it's definitely possible to shape things as you want. However, I
>>> would prefer to find a solution that works with the official plugin,
>>> because I don't believe our setup is remarkable or unusual: as GRPC sees an
>>> increase in use as a cross-language RPC mechanism, everyone else is going
>>> to struggle with the same things.
>>>
>>> Zellyn
>>>
>>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:
>>>
 Sorry for the delayed response!  I've written about 3-4 draft replies
 over this week that I've discarded because they don't meet your
 requirements.  I'll think about it more over the long weekend and get back
 to you.

 Cheers,
 -Ross


 On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:

> +Ross who is working on Go protobuf.
>
> Ross, do you have any comments or suggestions?
>
> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:
>
>> Hi folks,
>>
>> Apologies in advance for the complex email. It takes a bit of
>> explaining to set up what we're having trouble with.
>>
>> I would like to lay out how we generate protos in Go (using our
>> unfortunately forked version of the protoc plugin), and ask for 
>> suggestions
>> as to how to accomplish the same thing with the unforked version. In the
>> process, I wish to ask questions about the post-vendor-experiment utility
>> of the current behavior of import_prefix, hopefully generating more
>> discussion than I managed to here
>> .
>>
>> Background:
>>
>>- We have protos spread around all over the place: in our Go
>>monorepo, in our Java monorepo, and in miscellaneous Ruby repos.
>>- We collect them all into a single directory before generating
>>Go code. Call it $COLLECT_DIR
>>- We have to use go_package to 

Re: [protobuf] Question about Go protobufs and import_prefix

['Damien Neil' via Protocol Buffers]

Simpler than maintaining a fork, though.

The more convenient alternative would be for protoc to understand Go vendor
directories, but that would require putting some Go-specific logic in
protoc itself.

On Tue, Jul 12, 2016 at 8:00 AM, Zellyn Hunter  wrote:

> Yeah I can do anything with a wrapper. But to the extent that our concerns
> and structure are normal, it would be a shame to need a wrapper.
>
> On Tue, Jul 12, 2016, 8:54 AM Ross Light  wrote:
>
>> Zellyn and I talked a little bit at Gophercon about this. I think some
>> kind of wrapper is in order for his use case: it may actually be a filter
>> between protoc and protoc-gen-go.
>>
>> On Tue, Jul 12, 2016, 8:49 AM Damien Neil  wrote:
>>
>>> If I'm following this correctly, your core problem is that protoc
>>> doesn't understand vendored paths as used by the go tool. For example:
>>>
>>> - You have a proto file located in src/square/up/protos/square.proto.
>>> - square.proto imports "
>>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
>>> - timestamp.proto is actually located in "src/square/up/vendor/
>>> github.com/.../timestamp.proto".
>>>
>>> You can't run "protoc --go_out=. square/up/protos/square.proto", because
>>> protoc won't be able to locate the vendored copy of timestamp.proto. If it
>>> did, everything would work correctly without setting an import_prefix.
>>>
>>> It would be nice for Go users if protoc did understand vendored paths.
>>> Failing that, however, I think you can get what you want with a very small
>>> wrapper around protoc. The wrapper can copy a proto and all its
>>> dependencies into a temp directory (pulling dependencies from vendored
>>> paths as necessary), run protoc there, and copy the result back. e.g., in
>>> the case of the above example:
>>>
>>> $ mkdir -p /tmp/x/square/up/protos
>>> $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
>>> $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
>>> $ cp square/up/vendor/
>>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto
>>> github.com/golang/protobuf/ptypes/timestamp
>>> $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
>>> $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos
>>>
>>> The generated .go files will reference non-vendored paths, which the Go
>>> compiler will resolve to the correct vendored directory.
>>>
>>> Does that seem like it would work for you?
>>>
>>> On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter 
>>> wrote:
>>>
 Usually not too tricky. The problem is that protos and most other
 programming languages prohibit circular imports at the file level, and Go
 does it at the package level. The fact that protoc and the other languages
 are okay with a topology ensures there is a valid Go repackaging that will
 work.

 Zellyn

 On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
 protobuf@googlegroups.com> wrote:

> I don't know the background of the Go import system or go_package
> option. However this statement concerns me a little:
>
> > We have to use go_package to reorganize things that are fine in
> Java/Ruby, but would be circular package imports in Go.
>
> Is this implying that certain .proto files would generate invalid Go
> code until you manually insert some go_package statements? How tricky is 
> it
> to do this manual untangling?
>
>
> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>>
>> Oh wow. Thanks for thinking about it so carefully!
>>
>> I should mention: before you could actually specify full package
>> paths to the Go proto plugin, earlier versions of our wrapper (
>> https://github.com/square/goprotowrap) just forcibly manhandled
>> things into the shape we wanted by explicitly specifying -M parameter
>> import rewrites for *every single* import, and renaming the output
>> files into place.
>>
>> So it's definitely possible to shape things as you want. However, I
>> would prefer to find a solution that works with the official plugin,
>> because I don't believe our setup is remarkable or unusual: as GRPC sees 
>> an
>> increase in use as a cross-language RPC mechanism, everyone else is going
>> to struggle with the same things.
>>
>> Zellyn
>>
>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:
>>
>>> Sorry for the delayed response!  I've written about 3-4 draft
>>> replies over this week that I've discarded because they don't meet your
>>> requirements.  I'll think about it more over the long weekend and get 
>>> back
>>> to you.
>>>
>>> Cheers,
>>> -Ross
>>>
>>>
>>> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>>>
 +Ross who is working on Go protobuf.

 Ross, do you have any 

Re: [protobuf] Question about Go protobufs and import_prefix

['Damien Neil' via Protocol Buffers]

(Apologies if you get two copies of this mail; resending to the list 
because my first try bounced.)

If I'm following this correctly, your core problem is that protoc doesn't 
understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.
- square.proto imports 
"github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
- timestamp.proto is actually located in 
"src/square/up/vendor/github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because 
protoc won't be able to locate the vendored copy of timestamp.proto. If it 
did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths. 
Failing that, however, I think you can get what you want with a very small 
wrapper around protoc. The wrapper can copy a proto and all its 
dependencies into a temp directory (pulling dependencies from vendored 
paths as necessary), run protoc there, and copy the result back. e.g., in 
the case of the above example:

$ mkdir -p /tmp/x/square/up/protos
$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
$ cp 
square/up/vendor/github.com/golang/protobuf/ptypes/timestamp/timestamp.proto 
github.com/golang/protobuf/ptypes/timestamp
$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go 
compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?


On Sunday, July 10, 2016 at 12:32:26 PM UTC-7, Zellyn wrote:
>
> Usually not too tricky. The problem is that protos and most other 
> programming languages prohibit circular imports at the file level, and Go 
> does it at the package level. The fact that protoc and the other languages 
> are okay with a topology ensures there is a valid Go repackaging that will 
> work.
>
> Zellyn
>
> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
> prot...@googlegroups.com > wrote:
>
>> I don't know the background of the Go import system or go_package option. 
>> However this statement concerns me a little:
>>
>> > We have to use go_package to reorganize things that are fine in 
>> Java/Ruby, but would be circular package imports in Go.
>>
>> Is this implying that certain .proto files would generate invalid Go code 
>> until you manually insert some go_package statements? How tricky is it to 
>> do this manual untangling?
>>
>>
>> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>>>
>>> Oh wow. Thanks for thinking about it so carefully!
>>>
>>> I should mention: before you could actually specify full package paths 
>>> to the Go proto plugin, earlier versions of our wrapper (
>>> https://github.com/square/goprotowrap) just forcibly manhandled things 
>>> into the shape we wanted by explicitly specifying -M parameter import 
>>> rewrites for *every single* import, and renaming the output files into 
>>> place.
>>>
>>> So it's definitely possible to shape things as you want. However, I 
>>> would prefer to find a solution that works with the official plugin, 
>>> because I don't believe our setup is remarkable or unusual: as GRPC sees an 
>>> increase in use as a cross-language RPC mechanism, everyone else is going 
>>> to struggle with the same things.
>>>
>>> Zellyn
>>>
>>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light >> > wrote:
>>>
 Sorry for the delayed response!  I've written about 3-4 draft replies 
 over this week that I've discarded because they don't meet your 
 requirements.  I'll think about it more over the long weekend and get back 
 to you.

 Cheers,
 -Ross


 On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  
 wrote:

> +Ross who is working on Go protobuf.
>
> Ross, do you have any comments or suggestions?
>
> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  > wrote:
>
>> Hi folks,
>>
>> Apologies in advance for the complex email. It takes a bit of 
>> explaining to set up what we're having trouble with.
>>
>> I would like to lay out how we generate protos in Go (using our 
>> unfortunately forked version of the protoc plugin), and ask for 
>> suggestions 
>> as to how to accomplish the same thing with the unforked version. In the 
>> process, I wish to ask questions about the post-vendor-experiment 
>> utility 
>> of the current behavior of import_prefix, hopefully generating more 
>> discussion than I managed to here 
>> .
>>
>> Background:
>>
>>- We have protos spread around all over the place: in our Go 
>>monorepo, in our Java monorepo, and in miscellaneous Ruby 

Re: [protobuf] Question about Go protobufs and import_prefix

[Zellyn Hunter]

Yeah I can do anything with a wrapper. But to the extent that our concerns
and structure are normal, it would be a shame to need a wrapper.

On Tue, Jul 12, 2016, 8:54 AM Ross Light  wrote:

> Zellyn and I talked a little bit at Gophercon about this. I think some
> kind of wrapper is in order for his use case: it may actually be a filter
> between protoc and protoc-gen-go.
>
> On Tue, Jul 12, 2016, 8:49 AM Damien Neil  wrote:
>
>> If I'm following this correctly, your core problem is that protoc doesn't
>> understand vendored paths as used by the go tool. For example:
>>
>> - You have a proto file located in src/square/up/protos/square.proto.
>> - square.proto imports "
>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".
>> - timestamp.proto is actually located in "src/square/up/vendor/
>> github.com/.../timestamp.proto".
>>
>> You can't run "protoc --go_out=. square/up/protos/square.proto", because
>> protoc won't be able to locate the vendored copy of timestamp.proto. If it
>> did, everything would work correctly without setting an import_prefix.
>>
>> It would be nice for Go users if protoc did understand vendored paths.
>> Failing that, however, I think you can get what you want with a very small
>> wrapper around protoc. The wrapper can copy a proto and all its
>> dependencies into a temp directory (pulling dependencies from vendored
>> paths as necessary), run protoc there, and copy the result back. e.g., in
>> the case of the above example:
>>
>> $ mkdir -p /tmp/x/square/up/protos
>> $ cp src/square/up/protos/square.proto /tmp/x/square/up/protos
>> $ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp
>> $ cp square/up/vendor/
>> github.com/golang/protobuf/ptypes/timestamp/timestamp.proto
>> github.com/golang/protobuf/ptypes/timestamp
>> $ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto
>> $ cp /tmp/x/square/up/protos/square.pb.go square/up/protos
>>
>> The generated .go files will reference non-vendored paths, which the Go
>> compiler will resolve to the correct vendored directory.
>>
>> Does that seem like it would work for you?
>>
>> On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter  wrote:
>>
>>> Usually not too tricky. The problem is that protos and most other
>>> programming languages prohibit circular imports at the file level, and Go
>>> does it at the package level. The fact that protoc and the other languages
>>> are okay with a topology ensures there is a valid Go repackaging that will
>>> work.
>>>
>>> Zellyn
>>>
>>> On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
>>> protobuf@googlegroups.com> wrote:
>>>
 I don't know the background of the Go import system or go_package
 option. However this statement concerns me a little:

 > We have to use go_package to reorganize things that are fine in
 Java/Ruby, but would be circular package imports in Go.

 Is this implying that certain .proto files would generate invalid Go
 code until you manually insert some go_package statements? How tricky is it
 to do this manual untangling?


 On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>
> Oh wow. Thanks for thinking about it so carefully!
>
> I should mention: before you could actually specify full package paths
> to the Go proto plugin, earlier versions of our wrapper (
> https://github.com/square/goprotowrap) just forcibly manhandled
> things into the shape we wanted by explicitly specifying -M parameter
> import rewrites for *every single* import, and renaming the output
> files into place.
>
> So it's definitely possible to shape things as you want. However, I
> would prefer to find a solution that works with the official plugin,
> because I don't believe our setup is remarkable or unusual: as GRPC sees 
> an
> increase in use as a cross-language RPC mechanism, everyone else is going
> to struggle with the same things.
>
> Zellyn
>
> On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:
>
>> Sorry for the delayed response!  I've written about 3-4 draft replies
>> over this week that I've discarded because they don't meet your
>> requirements.  I'll think about it more over the long weekend and get 
>> back
>> to you.
>>
>> Cheers,
>> -Ross
>>
>>
>> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>>
>>> +Ross who is working on Go protobuf.
>>>
>>> Ross, do you have any comments or suggestions?
>>>
>>> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:
>>>
 Hi folks,

 Apologies in advance for the complex email. It takes a bit of
 explaining to set up what we're having trouble with.

 I would like to lay out how we generate protos in Go (using our
 unfortunately forked version of 

Re: [protobuf] Question about Go protobufs and import_prefix

[Zellyn Hunter]

Usually not too tricky. The problem is that protos and most other
programming languages prohibit circular imports at the file level, and Go
does it at the package level. The fact that protoc and the other languages
are okay with a topology ensures there is a valid Go repackaging that will
work.

Zellyn

On Fri, Jul 8, 2016, 6:52 PM 'Josh Haberman' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> I don't know the background of the Go import system or go_package option.
> However this statement concerns me a little:
>
> > We have to use go_package to reorganize things that are fine in
> Java/Ruby, but would be circular package imports in Go.
>
> Is this implying that certain .proto files would generate invalid Go code
> until you manually insert some go_package statements? How tricky is it to
> do this manual untangling?
>
>
> On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>>
>> Oh wow. Thanks for thinking about it so carefully!
>>
>> I should mention: before you could actually specify full package paths to
>> the Go proto plugin, earlier versions of our wrapper (
>> https://github.com/square/goprotowrap) just forcibly manhandled things
>> into the shape we wanted by explicitly specifying -M parameter import
>> rewrites for *every single* import, and renaming the output files into
>> place.
>>
>> So it's definitely possible to shape things as you want. However, I would
>> prefer to find a solution that works with the official plugin, because I
>> don't believe our setup is remarkable or unusual: as GRPC sees an increase
>> in use as a cross-language RPC mechanism, everyone else is going to
>> struggle with the same things.
>>
>> Zellyn
>>
>> On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:
>>
>>> Sorry for the delayed response!  I've written about 3-4 draft replies
>>> over this week that I've discarded because they don't meet your
>>> requirements.  I'll think about it more over the long weekend and get back
>>> to you.
>>>
>>> Cheers,
>>> -Ross
>>>
>>>
>>> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>>>
 +Ross who is working on Go protobuf.

 Ross, do you have any comments or suggestions?

 On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:

> Hi folks,
>
> Apologies in advance for the complex email. It takes a bit of
> explaining to set up what we're having trouble with.
>
> I would like to lay out how we generate protos in Go (using our
> unfortunately forked version of the protoc plugin), and ask for 
> suggestions
> as to how to accomplish the same thing with the unforked version. In the
> process, I wish to ask questions about the post-vendor-experiment utility
> of the current behavior of import_prefix, hopefully generating more
> discussion than I managed to here
> .
>
> Background:
>
>- We have protos spread around all over the place: in our Go
>monorepo, in our Java monorepo, and in miscellaneous Ruby repos.
>- We collect them all into a single directory before generating Go
>code. Call it $COLLECT_DIR
>- We have to use go_package to reorganize things that are fine in
>Java/Ruby, but would be circular package imports in Go.
>- For historical reasons, our Go repo is checked out to
>$GOPATH/src/square/up. I know it's a little weird, but it's logically
>equivalent to any other location. Pretend "square/up" is "square3" if 
> you
>like… :-)
>- Our monorepo's vendor folder lives at
>$GOPATH/src/square/up/vendor/...
>- We want to generate protos into $GOPATH/src/square/up/protos to
>keep them all in one place.
>- Most of our protos are in package "squareup.*", but we have some
>third-party protos that have their own packages. Or none (eg.
>nanopb
>
>).
>- Right now we also have copies of WKT protos in $COLLECT_DIR, but
>I'd love to just import them from their vendor folder locations.
>
> *What we do now: f**ull go_package including "protos" folder in
> protobufs*
>
> Add full package and path go_package to every protobuf:
> eg. For a file with package "squareup.testing", we set option
> go_package = "square/up/squareup/protos/testing".
> (We haven't actually added these declarations everywhere yet: this is
> what our forked plugin does: if go_package doesn't specify package path, 
> we
> use square/up/protos + slash-separated-package. But it's about the same
> thing. Our fork was created long before go_package gained the ability to
> actually specify output package path)
>
> Cons:
>
>- We generate into the output directory of $GOPATH/src, which is
>terrible: we could 

Re: [protobuf] Question about Go protobufs and import_prefix

['Josh Haberman' via Protocol Buffers]

I don't know the background of the Go import system or go_package option. 
However this statement concerns me a little:

> We have to use go_package to reorganize things that are fine in 
Java/Ruby, but would be circular package imports in Go.

Is this implying that certain .proto files would generate invalid Go code 
until you manually insert some go_package statements? How tricky is it to 
do this manual untangling?

On Friday, July 1, 2016 at 7:53:03 PM UTC-7, Zellyn wrote:
>
> Oh wow. Thanks for thinking about it so carefully!
>
> I should mention: before you could actually specify full package paths to 
> the Go proto plugin, earlier versions of our wrapper (
> https://github.com/square/goprotowrap) just forcibly manhandled things 
> into the shape we wanted by explicitly specifying -M parameter import 
> rewrites for *every single* import, and renaming the output files into 
> place.
>
> So it's definitely possible to shape things as you want. However, I would 
> prefer to find a solution that works with the official plugin, because I 
> don't believe our setup is remarkable or unusual: as GRPC sees an increase 
> in use as a cross-language RPC mechanism, everyone else is going to 
> struggle with the same things.
>
> Zellyn
>
> On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:
>
>> Sorry for the delayed response!  I've written about 3-4 draft replies 
>> over this week that I've discarded because they don't meet your 
>> requirements.  I'll think about it more over the long weekend and get back 
>> to you.
>>
>> Cheers,
>> -Ross
>>
>>
>> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>>
>>> +Ross who is working on Go protobuf.
>>>
>>> Ross, do you have any comments or suggestions?
>>>
>>> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:
>>>
 Hi folks,

 Apologies in advance for the complex email. It takes a bit of 
 explaining to set up what we're having trouble with.

 I would like to lay out how we generate protos in Go (using our 
 unfortunately forked version of the protoc plugin), and ask for 
 suggestions 
 as to how to accomplish the same thing with the unforked version. In the 
 process, I wish to ask questions about the post-vendor-experiment utility 
 of the current behavior of import_prefix, hopefully generating more 
 discussion than I managed to here 
 .

 Background:

- We have protos spread around all over the place: in our Go 
monorepo, in our Java monorepo, and in miscellaneous Ruby repos.
- We collect them all into a single directory before generating Go 
code. Call it $COLLECT_DIR
- We have to use go_package to reorganize things that are fine in 
Java/Ruby, but would be circular package imports in Go.
- For historical reasons, our Go repo is checked out to 
$GOPATH/src/square/up. I know it's a little weird, but it's logically 
equivalent to any other location. Pretend "square/up" is "square3" if 
 you 
like… :-)
- Our monorepo's vendor folder lives at 
$GOPATH/src/square/up/vendor/...
- We want to generate protos into $GOPATH/src/square/up/protos to 
keep them all in one place.
- Most of our protos are in package "squareup.*", but we have some 
third-party protos that have their own packages. Or none (eg. nanopb 

).
- Right now we also have copies of WKT protos in $COLLECT_DIR, but 
I'd love to just import them from their vendor folder locations.

 *What we do now: f**ull go_package including "protos" folder in 
 protobufs*

 Add full package and path go_package to every protobuf:
 eg. For a file with package "squareup.testing", we set option 
 go_package = "square/up/squareup/protos/testing".
 (We haven't actually added these declarations everywhere yet: this is 
 what our forked plugin does: if go_package doesn't specify package path, 
 we 
 use square/up/protos + slash-separated-package. But it's about the same 
 thing. Our fork was created long before go_package gained the ability to 
 actually specify output package path)

 Cons:

- We generate into the output directory of $GOPATH/src, which is 
terrible: we could accidentally generate stuff into any arbitrary go 
package in our GOPATH. :-(
- We're writing an implementation detail (where we currently store 
protos) into all the proto files

 *What I would like to do:*

 Generate into an output directory of $GOPATH/src/square/up/protos, and 
 use the import_prefix option to ensure that things get imported from the 
 right place.

 *Why that doesn't work:*

- import_prefix is added to 

Re: [protobuf] Question about Go protobufs and import_prefix

[Zellyn Hunter]

Oh wow. Thanks for thinking about it so carefully!

I should mention: before you could actually specify full package paths to
the Go proto plugin, earlier versions of our wrapper (
https://github.com/square/goprotowrap) just forcibly manhandled things into
the shape we wanted by explicitly specifying -M parameter import rewrites
for *every single* import, and renaming the output files into place.

So it's definitely possible to shape things as you want. However, I would
prefer to find a solution that works with the official plugin, because I
don't believe our setup is remarkable or unusual: as GRPC sees an increase
in use as a cross-language RPC mechanism, everyone else is going to
struggle with the same things.

Zellyn

On Fri, Jul 1, 2016 at 7:32 PM Ross Light  wrote:

> Sorry for the delayed response!  I've written about 3-4 draft replies over
> this week that I've discarded because they don't meet your requirements.
> I'll think about it more over the long weekend and get back to you.
>
> Cheers,
> -Ross
>
>
> On Fri, Jul 1, 2016 at 3:11 PM Jie Luo  wrote:
>
>> +Ross who is working on Go protobuf.
>>
>> Ross, do you have any comments or suggestions?
>>
>> On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:
>>
>>> Hi folks,
>>>
>>> Apologies in advance for the complex email. It takes a bit of explaining
>>> to set up what we're having trouble with.
>>>
>>> I would like to lay out how we generate protos in Go (using our
>>> unfortunately forked version of the protoc plugin), and ask for suggestions
>>> as to how to accomplish the same thing with the unforked version. In the
>>> process, I wish to ask questions about the post-vendor-experiment utility
>>> of the current behavior of import_prefix, hopefully generating more
>>> discussion than I managed to here
>>> .
>>>
>>> Background:
>>>
>>>- We have protos spread around all over the place: in our Go
>>>monorepo, in our Java monorepo, and in miscellaneous Ruby repos.
>>>- We collect them all into a single directory before generating Go
>>>code. Call it $COLLECT_DIR
>>>- We have to use go_package to reorganize things that are fine in
>>>Java/Ruby, but would be circular package imports in Go.
>>>- For historical reasons, our Go repo is checked out to
>>>$GOPATH/src/square/up. I know it's a little weird, but it's logically
>>>equivalent to any other location. Pretend "square/up" is "square3" if you
>>>like… :-)
>>>- Our monorepo's vendor folder lives at
>>>$GOPATH/src/square/up/vendor/...
>>>- We want to generate protos into $GOPATH/src/square/up/protos to
>>>keep them all in one place.
>>>- Most of our protos are in package "squareup.*", but we have some
>>>third-party protos that have their own packages. Or none (eg. nanopb
>>>
>>>).
>>>- Right now we also have copies of WKT protos in $COLLECT_DIR, but
>>>I'd love to just import them from their vendor folder locations.
>>>
>>> *What we do now: f**ull go_package including "protos" folder in
>>> protobufs*
>>>
>>> Add full package and path go_package to every protobuf:
>>> eg. For a file with package "squareup.testing", we set option go_package
>>> = "square/up/squareup/protos/testing".
>>> (We haven't actually added these declarations everywhere yet: this is
>>> what our forked plugin does: if go_package doesn't specify package path, we
>>> use square/up/protos + slash-separated-package. But it's about the same
>>> thing. Our fork was created long before go_package gained the ability to
>>> actually specify output package path)
>>>
>>> Cons:
>>>
>>>- We generate into the output directory of $GOPATH/src, which is
>>>terrible: we could accidentally generate stuff into any arbitrary go
>>>package in our GOPATH. :-(
>>>- We're writing an implementation detail (where we currently store
>>>protos) into all the proto files
>>>
>>> *What I would like to do:*
>>>
>>> Generate into an output directory of $GOPATH/src/square/up/protos, and
>>> use the import_prefix option to ensure that things get imported from the
>>> right place.
>>>
>>> *Why that doesn't work:*
>>>
>>>- import_prefix is added to *every* import, including grpc packages,
>>>the context package, the proto runtime package, etc. This made sense when
>>>the prefix was intended to be something like 
>>> "square/up/Godep/_workspace/"
>>>(eg. vendoring by import-path-rewriting), but doesn't make sense anymore.
>>>- import_prefix is added to imports for ALL protos, including things
>>>like well-known-type protos that should just live under
>>>$GOPATH/src/square/up/vendor/github.com/golang/protobuf/proto
>>>
>>> *What I'd like to get out of this:*
>>> Either:
>>>
>>>- a reworking of the way import_prefix works
>>>- a new parameter with slightly different 

Re: [protobuf] Question about Go protobufs and import_prefix

['Jie Luo' via Protocol Buffers]

+Ross who is working on Go protobuf.

Ross, do you have any comments or suggestions?

On Fri, Jun 24, 2016 at 8:35 AM, Zellyn  wrote:

> Hi folks,
>
> Apologies in advance for the complex email. It takes a bit of explaining
> to set up what we're having trouble with.
>
> I would like to lay out how we generate protos in Go (using our
> unfortunately forked version of the protoc plugin), and ask for suggestions
> as to how to accomplish the same thing with the unforked version. In the
> process, I wish to ask questions about the post-vendor-experiment utility
> of the current behavior of import_prefix, hopefully generating more
> discussion than I managed to here
> .
>
> Background:
>
>- We have protos spread around all over the place: in our Go monorepo,
>in our Java monorepo, and in miscellaneous Ruby repos.
>- We collect them all into a single directory before generating Go
>code. Call it $COLLECT_DIR
>- We have to use go_package to reorganize things that are fine in
>Java/Ruby, but would be circular package imports in Go.
>- For historical reasons, our Go repo is checked out to
>$GOPATH/src/square/up. I know it's a little weird, but it's logically
>equivalent to any other location. Pretend "square/up" is "square3" if you
>like… :-)
>- Our monorepo's vendor folder lives at
>$GOPATH/src/square/up/vendor/...
>- We want to generate protos into $GOPATH/src/square/up/protos to keep
>them all in one place.
>- Most of our protos are in package "squareup.*", but we have some
>third-party protos that have their own packages. Or none (eg. nanopb
>).
>- Right now we also have copies of WKT protos in $COLLECT_DIR, but I'd
>love to just import them from their vendor folder locations.
>
> *What we do now: f**ull go_package including "protos" folder in protobufs*
>
> Add full package and path go_package to every protobuf:
> eg. For a file with package "squareup.testing", we set option go_package =
> "square/up/squareup/protos/testing".
> (We haven't actually added these declarations everywhere yet: this is what
> our forked plugin does: if go_package doesn't specify package path, we use
> square/up/protos + slash-separated-package. But it's about the same thing.
> Our fork was created long before go_package gained the ability to actually
> specify output package path)
>
> Cons:
>
>- We generate into the output directory of $GOPATH/src, which is
>terrible: we could accidentally generate stuff into any arbitrary go
>package in our GOPATH. :-(
>- We're writing an implementation detail (where we currently store
>protos) into all the proto files
>
> *What I would like to do:*
>
> Generate into an output directory of $GOPATH/src/square/up/protos, and use
> the import_prefix option to ensure that things get imported from the right
> place.
>
> *Why that doesn't work:*
>
>- import_prefix is added to *every* import, including grpc packages,
>the context package, the proto runtime package, etc. This made sense when
>the prefix was intended to be something like "square/up/Godep/_workspace/"
>(eg. vendoring by import-path-rewriting), but doesn't make sense anymore.
>- import_prefix is added to imports for ALL protos, including things
>like well-known-type protos that should just live under
>$GOPATH/src/square/up/vendor/github.com/golang/protobuf/proto
>
> *What I'd like to get out of this:*
> Either:
>
>- a reworking of the way import_prefix works
>- a new parameter with slightly different semantics: adds an import
>prefix only for protos, and only for protos in a particular list of 
> folders:
>   - eg. proto_import_prefix_by_dir=$COLLECT_DIR:square/up/protos
>- an explanation of how I'm Doing it Wrong - I'm happy to change
>things. While there's a lot of existing code, Go protos have caused me so
>much pain, I'm willing to invest pretty significant effort.
>
> *Why I'd like to defork*
>
> Our fork was created before go_package could rearrange output packages (
> commit
> ).
> With the recent increase in grpc-related volatility of the protobuf
> package, it's increasingly painful to maintain our fork. And since grpc
> updates are often tied to proto updates, I can't simply avoid updating it.
> Plus all the usual reasons maintaining a fork is terrible :-)
>
> Comments, suggestions, and help welcome.
>
> Zellyn
>
> cc dsymonds, matloob at Google, Josh at Square.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit