Hey all, Trying to bring this thread back to life and give some updates since it petered out.
At Kuma[0], we've gone down the suggested `file_sd` route by building a custom binary that runs in a Prometheus sidecar to do the polling and writing of the sd file, which is then shared w/ Prometheus via volume mounts[1]. This works very well once it is set up correctly, but has proven difficult for many users to get up and running as it involves a good deal of low-level, non-prometheus config. It's even more difficult if they are not on the team managing Prometheus infra. For those reasons, implementing another generic SD would help purely with adoption, as configuring a few lines of yaml is a dramatically lower starting place compared to configuring a sidecar with shared volume mounts. I understand and agree with the general hesitation here, especially when it comes to defining entirely new "generic" SDs, and even the hesitation to adopt a widely-used protocol like envoy's xDS, but with the primary goal of increasing adoption, what do you all think about exposing a grpc_sd that uses the same format as the file_sd? I'm sure much of the implementation could be shared and it would remove the burden of setting up the network hop and file sharing themselves. Lastly, in terms of continued support, we at Kuma are committed to our Prometheus integration and are happy to contribute and maintain this SD with the Promtheus community. Other related issues: - #6484 [2], gRPC proposal - #7919 [3], Kuma SD proposal Thanks for hearing this out, Austin [0]: https://kuma.io/ [1]: https://kuma.io/docs/0.5.0/policies/traffic-metrics/#configure-dataplane-discovery-by-prometheus-2 [2]: https://github.com/prometheus/prometheus/issues/6484 [3]: https://github.com/prometheus/prometheus/issues/7919 On Friday, December 20, 2019 at 6:55:08 PM UTC-5 [email protected] wrote: > On Fri, 20 Dec 2019 at 22:57, Yaroslav Skopets <[email protected]> wrote: > >> Thank you! >> >> Do you why the discussion around `http_sd` stop ? >> > > That particular one looks like it started going into the weeds of > inventing a brand new SD, and petered out. There's enough of them out there > without us creating more. > > More generally, why should we have two generic SDs? That kinda defeats the > purpose of having a generic integration, which is to have one thing that > all the other use cases can be handled via. > If you want to add a hop that goes over http it's not exactly difficult to > run curl in a cronjob or have a sidecar. > > Brian > > >> On Fri, Dec 20, 2019 at 11:19 PM Daniel Swarbrick < >> [email protected]> wrote: >> >>> You might want to read this earlier thread first: >>> https://groups.google.com/d/topic/prometheus-developers/3B3jBsErK5M/discussion >>> >>> I would certainly be in favour of some generic gRPC-based SD method, >>> such that people could implement their own gRPC services, exposing scrape >>> targets from whatever funky / proprietary information sources they use. As >>> I wrote in the aforementioned thread, I ended up implementing a bare >>> minimum fake Consul service, providing only the API endpoints used by >>> Prometheus, as an SD mechanism for hosts stored in an enterprise database >>> (Postgres, in my case). It has worked flawlessly for over 18 months. >>> >>> I would of course much rather not have to fake other products' APIs >>> though. >>> >>> On Wednesday, December 18, 2019 at 11:34:36 PM UTC+1, Yaroslav Skopets >>> wrote: >>>> >>>> Proposal >>>> >>>> - In the context of a Service Mesh, there is always a notion of a >>>> Control Plane that coordinates telemetry among other things >>>> - It would be perfect for overall user experience if a Control >>>> Plane could also become a source of scrape targets (specifically, for >>>> collecting metrics out of side-car proxies) >>>> - As a way to achieve that, we would like to propose to introduce a >>>> gRPC interface for SD that Control Planes could implement >>>> - The choice of gRPC will enable effective server-side streaming of >>>> SD information, and it will remove the requirement on users to think >>>> upfront about refresh intervals they cannot choose reliably anyway >>>> - There is already a well-defined, battle-tested and widely used >>>> protocol for delivering discovery information - Envoy xDS API >>>> <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol> >>>> - Based on all the above, we would like to ask if Prometheus >>>> community is open for such a contribution >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Best regards, >> Yaroslav Skopets >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/8894963b-f578-4898-a3ff-d0b963a3e3a4n%40googlegroups.com.
