Hey all, I've gathered up some research about our efforts with the DNS SD option, please let me know if I've overlooked something.
Since the DNS SD[1] only supports basic A, AAAA, and SRV records and a small set of meta labels, the amount of information that can be transparently stored and parsed is limited. Of the three available meta labels, the only field that would make sense to use for holding information is `__meta_dns_name`. This is limiting when many pieces of info are necessary, as they'd have to be encoded and parsed with some decently complex relabling steps. In terms of a service mesh like Kuma, there are quite a lot of labels that are valuable to an end user, for example: the mesh name, the dataplane name, the service name, user-defined tags about the service, etc. These would get quite complex to encode and parse in a single (or even if spread across the three) label supported by the DNS SD mechanism. This limitation is likely to affect anyone trying to integrate with an extisiting Prometheus installation without, outside of a natively supported environment. Providing a more generic external SD might also be useful as an alternative to adding new environment-specific SDs where these limitations are hit. Anyways, I'll update the original ticket -- please let me know what you think @julien and if you'd be willing to further discuss what extra constraints would be necessary. Thanks so much, Austin [1]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config On Monday, October 12, 2020 at 6:15:22 PM UTC-4 Austin Cawley-Edwards wrote: > Hey Julien + Frederic, > > Thanks for the quick replies. > > I'm gathering details on the DNS SD option, but we have tried that option > without success. I'll post back when I have a full answer and am happy to > update the issue for completeness. > > I agree with Frederic on the Prometheus Operator point and think that even > if there was a way to use the Prometheus Operator to solve this, we'd still > be reliant on users to have a specific Prometheus installation. > > I think what you've said about the gRPC drawbacks are very reasonable, and > the HTTP approach sounds like a good, simple alternative. The restrictions > on such a sd also make total sense -- is there a full list of those > restrictions somewhere/ could you expand on them if not? My only worry is > that we'd end up trying to invent a new HTTP sd, which seems to be what > killed these threads in the past. > > Thank you! > Austin > > On Sat, Oct 10, 2020 at 2:23 AM Frederic Branczyk <[email protected]> > wrote: > >> > If you have the Prometheus Operator, you have a my understanding is that >> > you can make your "generic sd" by creating Probe CRD's (e.g. from within >> > the kuma controllers). Did you consider that option too? If you are not >> > in kube then you don't have shared mountpoints issues. >> >> Just for clarification, Prometheus Operator Probes are only meant for >> automating the setup and specialties around making blackbox probes more >> accessible. They are not a general-purpose way to scrape targets. >> >> On Sat, 10 Oct 2020 at 01:09, Julien Pivotto <[email protected]> >> wrote: >> >>> On 09 Oct 14:48, Austin Cawley-Edwards wrote: >>> > Hey all, >>> > >>> > Trying to bring this thread back to life and give some updates since >>> it >>> > petered out. >>> > >>> > At Kuma[0], we've gone down the suggested `file_sd` route by building >>> a >>> > custom binary that runs in a Prometheus sidecar to do the polling and >>> > writing of the sd file, which is then shared w/ Prometheus via volume >>> > mounts[1]. This works very well once it is set up correctly, but has >>> proven >>> > difficult for many users to get up and running as it involves a good >>> deal >>> > of low-level, non-prometheus config. It's even more difficult if they >>> are >>> > not on the team managing Prometheus infra. >>> > >>> > For those reasons, implementing another generic SD would help purely >>> with >>> > adoption, as configuring a few lines of yaml is a dramatically lower >>> > starting place compared to configuring a sidecar with shared volume >>> mounts. >>> > I understand and agree with the general hesitation here, especially >>> when it >>> > comes to defining entirely new "generic" SDs, and even the hesitation >>> to >>> > adopt a widely-used protocol like envoy's xDS, but with the primary >>> goal of >>> > increasing adoption, what do you all think about exposing a grpc_sd >>> that >>> > uses the same format as the file_sd? I'm sure much of the >>> implementation >>> > could be shared and it would remove the burden of setting up the >>> network >>> > hop and file sharing themselves. >>> > >>> > Lastly, in terms of continued support, we at Kuma are committed to our >>> > Prometheus integration and are happy to contribute and maintain this >>> SD >>> > with the Promtheus community. >>> > >>> > Other related issues: >>> > - #6484 [2], gRPC proposal >>> > - #7919 [3], Kuma SD proposal >>> >>> As you might have seen, we are slowly getting new SD's into prometheus >>> core. We had 4 new SD's in the last weeks. >>> >>> When the issue for Kuma arrived in te bug tracker, there were some >>> leftover questions, like did you try the DNS-SD? Then the issue was >>> directly closed. >>> >>> If you have the Prometheus Operator, you have a my understanding is that >>> you can make your "generic sd" by creating Probe CRD's (e.g. from within >>> the kuma controllers). Did you consider that option too? If you are not >>> in kube then you don't have shared mountpoints issues. >>> >>> Should we have a generic SD, I would not like that much GRPC as would >>> bring many dependencies, that we already had issues with. Additionally, >>> we have more chances to use a more-established standard like HTTP (with >>> long poll). Especially since the communications only need to be one-way. >>> >>> I also think that such SD's should be limited to what they do, and that >>> it should be less flexible than the current file_sd. I think that any >>> generic remote file_sd should have constraints, such as all the labels >>> should be prefixes with __meta unless the __address__ label. We should >>> try to enforce on those SD's the same rules we have kept for the SD's >>> that are implemented now. >>> >>> > >>> > Thanks for hearing this out, >>> > Austin >>> > >>> > [0]: https://kuma.io/ >>> > [1]: >>> > >>> https://kuma.io/docs/0.5.0/policies/traffic-metrics/#configure-dataplane-discovery-by-prometheus-2 >>> > [2]: https://github.com/prometheus/prometheus/issues/6484 >>> > [3]: https://github.com/prometheus/prometheus/issues/7919 >>> > On Friday, December 20, 2019 at 6:55:08 PM UTC-5 >>> > [email protected] wrote: >>> > >>> > > On Fri, 20 Dec 2019 at 22:57, Yaroslav Skopets <[email protected]> >>> wrote: >>> > > >>> > >> Thank you! >>> > >> >>> > >> Do you why the discussion around `http_sd` stop ? >>> > >> >>> > > >>> > > That particular one looks like it started going into the weeds of >>> > > inventing a brand new SD, and petered out. There's enough of them >>> out there >>> > > without us creating more. >>> > > >>> > > More generally, why should we have two generic SDs? That kinda >>> defeats the >>> > > purpose of having a generic integration, which is to have one thing >>> that >>> > > all the other use cases can be handled via. >>> > > If you want to add a hop that goes over http it's not exactly >>> difficult to >>> > > run curl in a cronjob or have a sidecar. >>> > > >>> > > Brian >>> > > >>> > > >>> > >> On Fri, Dec 20, 2019 at 11:19 PM Daniel Swarbrick < >>> > >> [email protected]> wrote: >>> > >> >>> > >>> You might want to read this earlier thread first: >>> > >>> >>> https://groups.google.com/d/topic/prometheus-developers/3B3jBsErK5M/discussion >>> > >>> >>> > >>> I would certainly be in favour of some generic gRPC-based SD >>> method, >>> > >>> such that people could implement their own gRPC services, exposing >>> scrape >>> > >>> targets from whatever funky / proprietary information sources they >>> use. As >>> > >>> I wrote in the aforementioned thread, I ended up implementing a >>> bare >>> > >>> minimum fake Consul service, providing only the API endpoints used >>> by >>> > >>> Prometheus, as an SD mechanism for hosts stored in an enterprise >>> database >>> > >>> (Postgres, in my case). It has worked flawlessly for over 18 >>> months. >>> > >>> >>> > >>> I would of course much rather not have to fake other products' >>> APIs >>> > >>> though. >>> > >>> >>> > >>> On Wednesday, December 18, 2019 at 11:34:36 PM UTC+1, Yaroslav >>> Skopets >>> > >>> wrote: >>> > >>>> >>> > >>>> Proposal >>> > >>>> >>> > >>>> - In the context of a Service Mesh, there is always a notion >>> of a >>> > >>>> Control Plane that coordinates telemetry among other things >>> > >>>> - It would be perfect for overall user experience if a Control >>> > >>>> Plane could also become a source of scrape targets >>> (specifically, for >>> > >>>> collecting metrics out of side-car proxies) >>> > >>>> - As a way to achieve that, we would like to propose to >>> introduce a >>> > >>>> gRPC interface for SD that Control Planes could implement >>> > >>>> - The choice of gRPC will enable effective server-side >>> streaming of >>> > >>>> SD information, and it will remove the requirement on users to >>> think >>> > >>>> upfront about refresh intervals they cannot choose reliably >>> anyway >>> > >>>> - There is already a well-defined, battle-tested and widely >>> used >>> > >>>> protocol for delivering discovery information - Envoy xDS API >>> > >>>> < >>> https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol> >>> > >>>> - Based on all the above, we would like to ask if Prometheus >>> > >>>> community is open for such a contribution >>> > >>>> >>> > >>>> -- >>> > >>> You received this message because you are subscribed to the Google >>> > >>> Groups "Prometheus Developers" group. >>> > >>> To unsubscribe from this group and stop receiving emails from it, >>> send >>> > >>> an email to [email protected]. >>> > >>> To view this discussion on the web visit >>> > >>> >>> https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com >>> >>> > >>> < >>> https://groups.google.com/d/msgid/prometheus-developers/cb86c927-8642-45b0-bf33-00d14e09aec7%40googlegroups.com?utm_medium=email&utm_source=footer >>> > >>> > >>> . >>> > >>> >>> > >> >>> > >> >>> > >> -- >>> > >> Best regards, >>> > >> Yaroslav Skopets >>> > >> >>> > >> -- >>> > >> You received this message because you are subscribed to the Google >>> Groups >>> > >> "Prometheus Developers" group. >>> > >> To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > >> email to [email protected]. >>> > >> >>> > > To view this discussion on the web visit >>> > >> >>> https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com >>> >>> > >> < >>> https://groups.google.com/d/msgid/prometheus-developers/CAJsMHLKWheCeGT_MiEGjrxbPpR8_t-nFmCgu4Vps5OE66-mekQ%40mail.gmail.com?utm_medium=email&utm_source=footer >>> > >>> > >> . >>> > >> >>> > > >>> > > >>> > > -- >>> > > Brian Brazil >>> > > www.robustperception.io >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/8894963b-f578-4898-a3ff-d0b963a3e3a4n%40googlegroups.com >>> . >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/20201009230855.GA1267035%40oxygen >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/79d85438-039b-4ab7-9638-86f3db987d70n%40googlegroups.com.
