Surely these issues already exist in gogo-protobuf? How are they handled there?
On Thu, Feb 1, 2018, 08:28 'Jisi Liu' via golang-nuts < golang-nuts@googlegroups.com> wrote: > This is not Java specific. I just used Java as an example. For most > protobuf implementations, there is a contract between the runtime and > generated code which is not meant to be public. e.g. In go-protobuf, > runtime expects the generated classes define some internal fields, like > XXX_unrecognized, and XXX_sizecache. > > If we are moving from table-driven to codegen, then I guess there will be > more contracts between the runtime and generated code, and potentially > introduce contracts also between the containing message and embedded > message fields - you probably want to extract duplicate methods into > runtime to save code size, and the generate code needs to deal with > submessages itself. This is my primary concern. > > In a single repo development model, someone can change such contract as > long as they change the protoc-gen and the runtime library at the same > time. However, in open source, such change can cause dependency-hell > <https://en.wikipedia.org/wiki/Dependency_hell>issues as the different > versions of generated protobufs are no-longer compatible with each other > and/or with the runtime. With a more tightly coupled model, we are more > prone to such issue. > > One way to solve this is to make the runtime support all the previous > versions of the contract indefinitely and be always backward compatible. > Then users can switch to the newest runtime to support all the old > libraries. We are following this approach in protobuf-java 3.x, which > causes lots of dev overhead. I'm a bit worried that we would need to do the > same for Go, if we go with the code-gen approach. > > > On Wed, Jan 31, 2018 at 10:25 PM Walter Schulze <awalterschu...@gmail.com> > wrote: > >> Could we perhaps get a code snippet example that non java programmers can >> follow. >> >> PS >> >> When I previously referred to java programmers as real software >> developers, I didn't add some much needed context. >> >> Sometimes in this java dominated world I personally don't feel like a >> real software developer. >> >> On Wed, 31 Jan 2018, 21:44 liujisi via golang-nuts, < >> golang-nuts@googlegroups.com> wrote: >> >>> >>> >>> On Wednesday, January 31, 2018 at 11:20:31 AM UTC-8, Walter Schulze >>> wrote: >>> >>>> Hi JT please see my inline replies. >>>> >>>> On Wed, 31 Jan 2018 at 19:05 <thebroke...@gmail.com> wrote: >>>> >>>>> Thank you, Walter, for your support. >>>>> >>>>> > gogo/protobuf is disappointed that golang/protobuf still thinks that >>>>> runtime reflection is an efficient way of serializing structures. >>>>> >>>>> The table-driven implementation avoids reflect in the fast and common >>>>> path. Instead, are you referring to the fact that we don't perform >>>>> full-code generation of Marshal/Unmarshal like what gogo/protobuf does? We >>>>> are aware that full-code generation will often out-perform the >>>>> table-driven >>>>> approach we took. However, full code-generation drastically bloats the >>>>> binary size when you have many proto messages linked in. Keeping the >>>>> binary >>>>> size smaller was an important design decision for us and seemed to be a >>>>> better default. >>>>> >>>> >>>> Yes, I was referring to the speed of code generation over runtime >>>> reflection. >>>> What I struggle to understand is why the optimize_for file option that >>>> is part of proto 2 and 3 is not considered by golang/protobuf as a way to >>>> specify when code generation should be used over runtime reflection. >>>> This seems to work for most other languages, including Java, which I >>>> heard is quite popular among real software developers. >>>> >>> >>> Note that the code-generation approach used by other languages (mostly >>> C++/Java) has its own problem. Mostly because of the tight coupling among >>> generated code, runtime and embedded sub messages (generated by a different >>> party using a different version of protoc). These problem don't exist >>> inside of google as we use a single repo build system, but it cause >>> significant issues in opensource. For instance, Hadoop is still shipping >>> protobuf v2.5 generated code which is incompatible with later version of >>> protobufs. All the projects using Hadoop are then version locked to v2.5, >>> as an upgrade in any project (including Hadoop itself) will break the >>> build. Version upgrade can only happen when all the transitive dependency >>> closure upgrade together atomically. >>> >>> We solve this problem in protobuf-java v3.0+ by introducing ABI >>> backward/forward compatibility guarantees on generated code and runtime. >>> However, this introduced lots of overhead on code maintenance, reduced >>> development velocity and limited the change we could do. We are now >>> solving the issue by introduce table driven to Java. The recent benchmark >>> result showed performance on par for android platforms, and hopefully we >>> can release the new implementation in a few months. >>> >>> For Go, if we are going to introduce full generated code, I'd strongly >>> recommend considering those complications. Major version bump is also >>> expensive for protobufs as all the dependency libraries would have to bump >>> their major version too. >>> >>> >>>> >>>> >>>>> >>>>> We are open to considering an option that allows user to specify >>>>> full-code generation for select messages. >>>>> >>>> >>>> This is exactly what gogo/protobuf allows users to do. >>>> Using protobuf extensions gogo/protobuf allows the user to specify per >>>> message or file whether they want to generate marshalers, unmarshalers, >>>> etc. >>>> A user can also create a vanity binary to generate these methods if you >>>> do not wish to use extensions and want to enforce a specific style across >>>> and organization. >>>> >>>> >>>>> >>>>> > gogo/protobuf is still open to being merged back into >>>>> golang/protobuf and has been since its inception 5 years ago. >>>>> >>>>> That is good to hear. I have not yet gone through all of gogo/protobuf >>>>> to determine what it would to merge, or what should be merged. This will >>>>> be >>>>> future work. >>>>> >>>> >>>> gogo/protobuf is also be open to only being partly merged. >>>> >>>> One other major advantage of gogo/protobuf is generating the structures >>>> you want to use, by allowing you to modify the generated structure using >>>> protobuf extensions like customtype. >>>> This way you can avoid copying between the protobuf generated structure >>>> and a user defined go structure that you actually want to use. >>>> This is a huge speed and safety gain and probably the most important >>>> feature of gogo/protobuf. >>>> proto3 has addressed the biggest concern by allowing the generation of >>>> fields without pointers, but there are other cases as well, including >>>> casttype, customname for generating more lintable code and even not >>>> generating the structure at all, for ultimate customization. >>>> I would hope that merging some of these ideas will also be on the table. >>>> >>>> Looking forward to working together for a change >>>> Please let me know how I can help >>>> >>>> Skeptically hopeful about a new era for protobufs in Go >>>> Walter Schulze >>>> >>>> >>> >>>>> JT >>>>> >>>>> On Wednesday, January 31, 2018 at 7:13:38 AM UTC-8, Walter Schulze >>>>> wrote: >>>>>> >>>>>> gogo/protobuf is happy to be acknowledged by Google as an entity in >>>>>> the golang protobuf space. >>>>>> gogo/protobuf welcomes golang/protobuf to the community and is >>>>>> extremely happy to see this kind of transparency. >>>>>> >>>>>> gogo/protobuf will also merge these changes and as usual try to stay >>>>>> as close as possible to golang/protobuf, >>>>>> including also following the same version tagging. >>>>>> >>>>>> gogo/protobuf is disappointed that golang/protobuf still thinks that >>>>>> runtime reflection is an efficient way of serializing structures. >>>>>> >>>>>> go Green go GoGoProtobuf >>>>>> >>>>>> PS >>>>>> >>>>>> gogo/protobuf is still open to being merged back into golang/protobuf >>>>>> and has been since its inception 5 years ago. >>>>>> gogo/protobuf feels for its users, especially those that are not >>>>>> acknowledged by grpc-gateway and grpc-go, >>>>>> and forced to employ work arounds, to preserve their missions of >>>>>> safety and efficiency. >>>>>> It knows that its existence is not something that anyone prefers, and >>>>>> it welcomes death, >>>>>> but only if it can preserve its legacy of fast serailization and >>>>>> generating the structures you want to use. >>>>>> >>>>>> >>>>>> On Tuesday, 30 January 2018 23:44:37 UTC+1, joe...@google.com wrote: >>>>>>> >>>>>>> Done. I tagged v1.0.0. When we perform the merge in the future, it >>>>>>> will be tagged as v1.1.0. >>>>>>> >>>>>>> On Tuesday, January 30, 2018 at 9:37:23 AM UTC-8, Alexey >>>>>>> Palazhchenko wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Can you please add tags to the repository before that? SemVer or >>>>>>>> even tags with _any_ semantic would greatly help to rollback to the >>>>>>>> latest >>>>>>>> working version when things break. >>>>>>>> >>>>>>>> –-– >>>>>>>> Alexey «AlekSi» Palazhchenko >>>>>>>> >>>>>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "golang-nuts" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe. >>>>> >>>> To unsubscribe from this group and all its topics, send an email to >>>>> golang-nuts...@googlegroups.com. >>>> >>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "golang-nuts" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe. >>> >> To unsubscribe from this group and all its topics, send an email to >>> golang-nuts+unsubscr...@googlegroups.com. >> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
