On Tue, Jun 4, 2024 at 9:59 PM Reinhard Tartler <[email protected]> wrote: > > Adding debian-go@, I think this topic deserves a wider audience > > sorry for the duplicate mail, I messed up the copy for debian-go@, so this is > a re-send. > > > On Mon, Jun 3, 2024 at 2:31 PM Reinhard Tartler <[email protected]> wrote: >> >> Basically notary insists on golang-github-golang-protobuf-1-3-dev, whereas >> golang-github-grpc-ecosystem-grpc-gateway.v2-dev (which is coming via the >> opentelemetry deps) is expecting golang-github-golang-protobuf-1-5-dev. >> >> How can we resolve this situation? > > > so, I noticed that we have at least 12 packages in Debian that have > alternative dependency declaration in the form of > `golang-github-golang-protobuf-1-3-dev | > golang-github-golang-protobuf-1-5-dev`. Most (if not all of them) contain > generated *.pb.go files. I think I may have been under the false assumption > that code generated with protoc-gen-go-1-3 would not be compatible with > golang-github-golang-protobuf-1-5-dev. But is code generated with > protoc-gen-go-1-5 actually compatible with > golang-github-golang-protobuf-1-3-dev, that is, can it be linked with > golang/protobuf @v1.3? > > However, in that case the alternative "Depends" on those 12+ packages still > confuses me. Let me explain. > > As far as I can tell, there are several implementations of protobuf, each > coming with their own code generator and runtime library. Let me put this > down the four implementations of protobuf in Debian in the following table: > > source package | generator | notes > -------------- | --------- | ----- > golang-gogoprotobuf | gogoprotobuf | more performant, > deprecated upstream > golang-github-golang-protobuf-1-3 | protoc-gen-go-1-3 | implements protobuf > APIv1 > golang-github-golang-protobuf-1-5 | protoc-gen-go-1-5 | is a version of APIv1 > implemented "in terms of APIv2" > golang-google-protobuf | protoc-gen-go | APIv2, supersedes the > above > > The above information is mostly taken out of my understanding from > https://go.dev/blog/protobuf-apiv2 > > The alternative dependencies in the 12+ package seem to imply that the > generated code in those packages can be freely used with the runtimes that > come with either golang/[email protected] or @v1.5. But is that actually a safe > assumption? I'm particularly concerned that code generated with > protoc-gen-go-1-5 is unlikely to work with the library that comes with > golang-github-golang-protobuf-1-3. I can see the other way around (code > generated with protoc-gen-go-1-3 working with golang/[email protected]), but that > sounds like something we should set a policy for. >
I'm not sure about that either. But I only heard the runtime break with gogoprobuf, not the google ones (as your next paragraph says). I assume if the packages can be compiled, then they are safe. > I totally see how code generated with gogoprotobuf not being compatible with > anything else, leading to issues such as #975431 / > https://github.com/containerd/ttrpc/issues/62#issuecomment-686768932. So this > is effectively prevented by the "Conflicts" relationship in > golang-github-golang-protobuf-1-5-dev: > Yes they are not compatible. But for the packaging, there are no conflicts around gogoprotobuf packages. You probably misread golang-goprotobuf-dev as golang-gogoprotobuf-dev. (These names are very similar...). golang-goprotobuf-dev is a transitional package which ties the runtime dev package and the code generator together. > Package: golang-github-golang-protobuf-1-5-dev > Source: golang-github-golang-protobuf-1-5 > Version: 1.5.4-1 > Conflicts: golang-github-golang-protobuf-1-3-dev, golang-goprotobuf-dev > > Similarly, we have another relationship here: > > Package: golang-github-golang-protobuf-1-3-dev > Source: golang-github-golang-protobuf-1-3 > Version: 1.3.5-4 > Conflicts: golang-github-golang-protobuf-1-5-dev > > (Isn't there a missing Conflicts relationship on golang-goprotobuf-dev? -- is > that intentional or an oversight?) > > In a similar vein, I can see that code generated by protoc-gen-go might cause > similar crashes when used with the runtime from > golang-github-golang-protobuf-1-3 (or possibly > golang-github-golang-protobuf-1-5, but I'm not sure). Note that neither > package declares any conflicts to golang-google-protobuf-dev (which comes > from google/protobuf). Is that intentional or an oversight? If the pb.go files are generated by protoc-gen-go (from src:golang-google-protobuf), they can't be consumed by golang-github-golang-protobuf-1-3-dev or golang-github-gogo-protobuf-dev. The application will fail at compile time, so it would just be noticed as FTBFS. > > > Does my rambling above make sense? What conclusions do we want to take from > here? > > - It seems to me that we probably want to move packages away from > src:golang-gogoprotobuf as much as possible, in particular because of the > interoperability concerns. Is that agreeable? > Yes. This package is dead, and we also should convince upstream to move away from it. > - It also seems to me that we probably want to avoid having both > golang-github-golang-protobuf-1-3 and golang-github-golang-protobuf-1-5 at > the same time. Both implement APIv1, and as such should be source-code > compatible. We currently seem to have 19 packages build-depending on > golang-github-golang-protobuf-1-3-dev. Can we lift-and-shift all of them in > one transition to @v1.5? > I think so. The runtime library golang-github-golang-protobuf-1-5-dev should be compatible with golang-github-golang-protobuf-1-3-dev. Current approach of listing them as alternative dependency seems to be working more gracefully. I probably should not introduce golang-github-golang-protobuf-1-3-dev at first, but only keep the protoc-gen-go-1-3 (the pb.go generator) around. > - As for packaging docker 2.4 (cf. #1033839, which spun off this > conversation), which of the four library implementations above should it be > linking against? -- We can savely rule out gogoprotobuf (it's deprecated), > and I guess google-protobuf as well, because I don't think all dependencies > have moved to the APIv2. That leaves two options: golang-protobuf @v1.3 and > @v1.5. Given the compatibility concerns above, I guess @v1.5 is the better > choice. That means that at least the 'notary' package needs to be updated (as > Arnaud points separately) to pick up protobuf v1.5. > I think the notary code doesn't need to change. Just switching the golang-github-golang-protobuf-1-*-dev in Depends could work. As the 1.5 should be compatible with 1.3. > - Currently, packages that use grpc (that is, build-depend on > golang-google-grpc-dev or protoc-gen-go-grpc), contain code that is generated > with golang/protobuf @v1.5. Wouldn't we expect applications that use grpc to > (possibly silently) break when linked against any other library than protobuf > @v1.5? Would it help to reflect this in package relationships (i.e., add > appropriate Conflicts or Build-Conflicts relationships)? > protoc-gen-go-grpc is a new binary package, which is not used by any package yet. In theory, if the pb.go files are generated by protoc-gen-go-1-5 or protoc-gen-go (the generator without version suffix is from src:golang-google-protobuf), and the application still uses golang-github-gogo-protobuf-dev as runtime library, than it will break. >From my observation for some upstream projects over these years, I think the breakage only happens when the pb.go files from golang-google-genproto-dev are generated by the new generators. For example, the containerd project pins the version at https://github.com/containerd/containerd/blob/74a86c30b3fcbb06802f4e3d95f49d0a127d4298/go.mod#L143 but they have bumped protobuf runtime as well as grpc runtime. > > Please do share your thoughts and opinions, I'd find the whole topic rather > confusing and would like to help solve it, and not make it worse. -- Shengjing Zhu
