Sounds good! As this is pretty performance sensitive, I think we'd want to benchmark any changes to this code before we merge it. Let me know if I can help there..
> You mentioned there are others for Cortex @tom.wil...@gmail.com? Bringing this up at the next Cortex community is probably the best way: https://github.com/cortexproject/cortex#community-meetings Cheers! Tom On Wed, Apr 28, 2021 at 5:39 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > This surprised me a little, and sounds awesome - but I can't find any more >> information about it. Does v1.4.0 generate code for the serialisation / >> deserialisation function or still rely on Golang reflection? Does it allow >> for the neat tricks to get rid of pointers and all the allocations? >> Anything I can read about this would be super useful. > > > Hmm, I'm not sure on all the specifics, but here's what I've found so far: > * Golang reflection has largely been replaced by the protoreflect > <https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect> > package (which *does not *use Golang reflection under the hood) > * SerDe uses the protoreflect package via dedicated modules for different > formats (json > <https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/encoding/protojson>, > text > <https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/encoding/prototext>, > wire > <https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/encoding/protowire> > ) > * Not sure about the specific tricks/ if the overuse of pointers + > allocations are still present, but there is now a first-class lib for > creating compiler plugins > <https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen> that > might be what you're looking for? Looks like there are many plugins that > use it already, judging by the imports > <https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen?tab=importedby> > > Off the top of my head, I would think checking in with the Cortex and >> Thanos projects is probably a good idea ... >> > > +1 sounds good! Are there specific people I should ping on this list? Or > open up issues in their repos? You mentioned there are others for Cortex @ > tom.wil...@gmail.com? > > On Wed, Apr 28, 2021 at 8:05 AM Tom Wilkie <tom.wil...@gmail.com> wrote: > >> > Along with that, many of the performance boosts that are provided by >> gogo have been addressed in the official Go lib, golang/protobuf >> <https://github.com/golang/protobuf>, as of v1.4.0 >> <https://blog.golang.org/protobuf-apiv2>. >> >> This surprised me a little, and sounds awesome - but I can't find any >> more information about it. Does v1.4.0 generate code for the serialisation >> / deserialisation function or still rely on Golang reflection? Does it >> allow for the neat tricks to get rid of pointers and all the allocations? >> Anything I can read about this would be super useful. >> >> > Off the top of my head, I would think checking in with the Cortex and >> Thanos projects is probably a good idea, I know both have a good amount of >> optimizations optimizing allocations, so it would be good to check that >> these would still be possible. >> >> From a Cortex PoV, we have copies of these protos so I don't think this >> would be too much of a problem, but I'd defer to people who know better >> than me. >> >> Cheers >> >> Tom >> >> On Wed, Apr 28, 2021 at 10:07 AM Frederic Branczyk <fbranc...@gmail.com> >> wrote: >> >>> I'd be very happy with this. One consideration that we should take >>> (certainly not blocking this but should keep in mind) is how this would >>> affect immediate downstream users. Off the top of my head, I would think >>> checking in with the Cortex and Thanos projects is probably a good idea, I >>> know both have a good amount of optimizations optimizing allocations, so it >>> would be good to check that these would still be possible. >>> >>> On Tue, 27 Apr 2021 at 22:51, Austin Cawley-Edwards < >>> austin.caw...@gmail.com> wrote: >>> >>>> >>>> Hi all, >>>> >>>> The protobuf library used in Prometheus, gogo/protobuf >>>> <https://github.com/gogo/protobuf>, is no longer actively maintained >>>> (activity largely stopped pre-2020, looking for new ownership >>>> <https://github.com/gogo/protobuf/issues/691>). Along with that, many >>>> of the performance boosts that are provided by gogo have been addressed in >>>> the official Go lib, golang/protobuf >>>> <https://github.com/golang/protobuf>, as of v1.4.0 >>>> <https://blog.golang.org/protobuf-apiv2>. >>>> >>>> Many projects that used gogo/protobuf have since switched to the >>>> official lib (ex: Istio <https://github.com/istio/istio/pull/17132>, >>>> Envoyproxy <https://github.com/envoyproxy/go-control-plane/issues/213>), >>>> largely for eco-system compatibility reasons now that performance is not a >>>> factor. The gogo-compiled protobufs are not compatible with any >>>> golang-compiled protobufs, and vice-versa. This makes consuming external >>>> protobuf APIs impossible unless they also maintain a gogo version, which is >>>> not common. >>>> >>>> I'm wondering if anyone has done work in this area recently, and if the >>>> community agrees it's a net benefit switching to the official >>>> golang/protobuf implementation. >>>> >>>> *What this would mean for Prometheus* >>>> >>>> Looking at the history of protobuf in Prometheus, it seems like both >>>> golang/protobuf and gogo/protobuf were until the end of 2017, when it then >>>> made sense to consolidate onto gogo (#3346 >>>> <https://github.com/prometheus/prometheus/issues/3346>) (#3394 >>>> <https://github.com/prometheus/prometheus/pull/3394>). >>>> >>>> As noted in the above issues, the official golang/protobuf package is >>>> still present in vendored files, so it is just the Prometheus protos that >>>> would need to be updated. The build procedures (mainly >>>> scripts/genproto.sh >>>> <https://github.com/prometheus/prometheus/blob/75e505babb9bbcefd8849e945814d35c7ce97e9f/scripts/genproto.sh>) >>>> have not changed much since the 2017 shift, so the work would be mainly >>>> adjusting this back to use golang/protobuf and recompiling the `prompb` >>>> package. >>>> >>>> Does anyone know of other necessary changes/ complications that I'm >>>> missing? >>>> >>>> Thanks all for your time, >>>> Austin >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to prometheus-developers+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-developers/38caa51d-e88a-489b-a045-54144cd1a03fn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-developers/38caa51d-e88a-489b-a045-54144cd1a03fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-developers+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umy%2BhjN10r9q_MOwZ2wHL-0MEDZWw_2SCUPW%2B%3D8xpKvdQw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umy%2BhjN10r9q_MOwZ2wHL-0MEDZWw_2SCUPW%2B%3D8xpKvdQw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAB58Z13%3DKnyuMzeyZY0P5sDdvorZviSMqwnDos3ekWqsOTgRJw%40mail.gmail.com.
