Hey Julien + Frederic, Thanks for the quick replies.
I'm gathering details on the DNS SD option, but we have tried that option without success. I'll post back when I have a full answer and am happy to update the issue for completeness. I agree with Frederic on the Prometheus Operator point and think that even if there was a way to use the Prometheus Operator to solve this, we'd still be reliant on users to have a specific Prometheus installation. I think what you've said about the gRPC drawbacks are very reasonable, and the HTTP approach sounds like a good, simple alternative. The restrictions on such a sd also make total sense -- is there a full list of those restrictions somewhere/ could you expand on them if not? My only worry is that we'd end up trying to invent a new HTTP sd, which seems to be what killed these threads in the past. Thank you! Austin On Sat, Oct 10, 2020 at 2:23 AM Frederic Branczyk <[email protected]> wrote: > > If you have the Prometheus Operator, you have a my understanding is that > > you can make your "generic sd" by creating Probe CRD's (e.g. from within > > the kuma controllers). Did you consider that option too? If you are not > > in kube then you don't have shared mountpoints issues. > > Just for clarification, Prometheus Operator Probes are only meant for > automating the setup and specialties around making blackbox probes more > accessible. They are not a general-purpose way to scrape targets. > > On Sat, 10 Oct 2020 at 01:09, Julien Pivotto <[email protected]> > wrote: > >> On 09 Oct 14:48, Austin Cawley-Edwards wrote: >> > Hey all, >> > >> > Trying to bring this thread back to life and give some updates since it >> > petered out. >> > >> > At Kuma[0], we've gone down the suggested `file_sd` route by building a >> > custom binary that runs in a Prometheus sidecar to do the polling and >> > writing of the sd file, which is then shared w/ Prometheus via volume >> > mounts[1]. This works very well once it is set up correctly, but has >> proven >> > difficult for many users to get up and running as it involves a good >> deal >> > of low-level, non-prometheus config. It's even more difficult if they >> are >> > not on the team managing Prometheus infra. >> > >> > For those reasons, implementing another generic SD would help purely >> with >> > adoption, as configuring a few lines of yaml is a dramatically lower >> > starting place compared to configuring a sidecar with shared volume >> mounts. >> > I understand and agree with the general hesitation here, especially >> when it >> > comes to defining entirely new "generic" SDs, and even the hesitation >> to >> > adopt a widely-used protocol like envoy's xDS, but with the primary >> goal of >> > increasing adoption, what do you all think about exposing a grpc_sd >> that >> > uses the same format as the file_sd? I'm sure much of the >> implementation >> > could be shared and it would remove the burden of setting up the >> network >> > hop and file sharing themselves. >> > >> > Lastly, in terms of continued support, we at Kuma are committed to our >> > Prometheus integration and are happy to contribute and maintain this SD >> > with the Promtheus community. >> > >> > Other related issues: >> > - #6484 [2], gRPC proposal >> > - #7919 [3], Kuma SD proposal >> >> As you might have seen, we are slowly getting new SD's into prometheus >> core. We had 4 new SD's in the last weeks. >> >> When the issue for Kuma arrived in te bug tracker, there were some >> leftover questions, like did you try the DNS-SD? Then the issue was >> directly closed. >> >> If you have the Prometheus Operator, you have a my understanding is that >> you can make your "generic sd" by creating Probe CRD's (e.g. from within >> the kuma controllers). Did you consider that option too? If you are not >> in kube then you don't have shared mountpoints issues. >> >> Should we have a generic SD, I would not like that much GRPC as would >> bring many dependencies, that we already had issues with. Additionally, >> we have more chances to use a more-established standard like HTTP (with >> long poll). Especially since the communications only need to be one-way. >> >> I also think that such SD's should be limited to what they do, and that >> it should be less flexible than the current file_sd. I think that any >> generic remote file_sd should have constraints, such as all the labels >> should be prefixes with __meta unless the __address__ label. We should >> try to enforce on those SD's the same rules we have kept for the SD's >> that are implemented now. >> >> > >> > Thanks for hearing this out, >> > Austin >> > >> > [0]: https://kuma.io/ >> > [1]: >> > >> https://kuma.io/docs/0.5.0/policies/traffic-metrics/#configure-dataplane-discovery-by-prometheus-2 >> > [2]: https://github.com/prometheus/prometheus/issues/6484 >> > [3]: https://github.com/prometheus/prometheus/issues/7919 >> > On Friday, December 20, 2019 at 6:55:08 PM UTC-5 >> > [email protected] wrote: >> > >> > > On Fri, 20 Dec 2019 at 22:57, Yaroslav Skopets <[email protected]> >> wrote: >> > > >> > >> Thank you! >> > >> >> > >> Do you why the discussion around `http_sd` stop ? >> > >> >> > > >> > > That particular one looks like it started going into the weeds of >> > > inventing a brand new SD, and petered out. There's enough of them out >> there >> > > without us creating more. >> > > >> > > More generally, why should we have two generic SDs? That kinda >> defeats the >> > > purpose of having a generic integration, which is to have one thing >> that >> > > all the other use cases can be handled via. >> > > If you want to add a hop that goes over http it's not exactly >> difficult to >> > > run curl in a cronjob or have a sidecar. >> > > >> > > Brian >> > > >> > > >> > >> On Fri, Dec 20, 2019 at 11:19 PM Daniel Swarbrick < >> > >> [email protected]> wrote: >> > >> >> > >>> You might want to read this earlier thread first: >> > >>> >> https://groups.google.com/d/topic/prometheus-developers/3B3jBsErK5M/discussion >> > >>> >> > >>> I would certainly be in favour of some generic gRPC-based SD >> method, >> > >>> such that people could implement their own gRPC services, exposing >> scrape >> > >>> targets from whatever funky / proprietary information sources they >> use. As >> > >>> I wrote in the aforementioned thread, I ended up implementing a >> bare >> > >>> minimum fake Consul service, providing only the API endpoints used >> by >> > >>> Prometheus, as an SD mechanism for hosts stored in an enterprise >> database >> > >>> (Postgres, in my case). It has worked flawlessly for over 18 months. >> > >>> >> > >>> I would of course much rather not have to fake other products' APIs >> > >>> though. >> > >>> >> > >>> On Wednesday, December 18, 2019 at 11:34:36 PM UTC+1, Yaroslav >> Skopets >> > >>> wrote: >> > >>>> >> > >>>> Proposal >> > >>>> >> > >>>> - In the context of a Service Mesh, there is always a notion of >> a >> > >>>> Control Plane that coordinates telemetry among other things >> > >>>> - It would be perfect for overall user experience if a Control >> > >>>> Plane could also become a source of scrape targets >> (specifically, for >> > >>>> collecting metrics out of side-car proxies) >> > >>>> - As a way to achieve that, we would like to propose to >> introduce a >> > >>>> gRPC interface for SD that Control Planes could implement >> > >>>> - The choice of gRPC will enable effective server-side >> streaming of >> > >>>> SD information, and it will remove the requirement on users to >> think >> > >>>> upfront about refresh intervals they cannot choose reliably >> anyway >> > >>>> - There is already a well-defined, battle-tested and widely >> used >> > >>>> protocol for delivering discovery information - Envoy xDS API >> > >>>> < >> https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol> >> > >>>> - Based on all the above, we would like to ask if Prometheus >> > >>>> community is open for such a contribution >> > >>>> >> > >>>> -- >> > >>> You received this message because you are subscribed to the Google >> > >>> Groups "Prometheus Developers" group. >> > >>> To unsubscribe from this group and stop receiving emails from it, >> send >> > >>> an email to [email protected]. >> > >>> To view this discussion on the web visit >> > >>> >> https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com >> > >>> < >> https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com?utm_medium=email&utm_source=footer >> > >> > >>> . >> > >>> >> > >> >> > >> >> > >> -- >> > >> Best regards, >> > >> Yaroslav Skopets >> > >> >> > >> -- >> > >> You received this message because you are subscribed to the Google >> Groups >> > >> "Prometheus Developers" group. >> > >> To unsubscribe from this group and stop receiving emails from it, >> send an >> > >> email to [email protected]. >> > >> >> > > To view this discussion on the web visit >> > >> >> https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com >> > >> < >> https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com?utm_medium=email&utm_source=footer >> > >> > >> . >> > >> >> > > >> > > >> > > -- >> > > Brian Brazil >> > > www.robustperception.io >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/8894963b-f578-4898-a3ff-d0b963a3e3a4n%40googlegroups.com >> . >> >> >> -- >> Julien Pivotto >> @roidelapluie >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/20201009230855.GA1267035%40oxygen >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CALGL%2BUA2mQmp-P57ZDcybNP0F51k4uxn5QeH_8ORhziHapUpNQ%40mail.gmail.com.
