> Well I think we should have a xds package which would have envoy_sd, and > possibly kuma_sd, but stay scoped and tight to what exists _now_. Having > them in a xds package enables us to reuse code.
That could make sense to me, though in the long term seems like it would be more of a burden in the long run that fragments xDS implementations and brings more dependencies into Prometheus. I don't think it would make sense to implement this for envoy without someone asking for it, but it could be done for Kuma where we have an xDS client in the xds package, and then pull in Kuma's xDS API for Monitoring assignments. I'd be happy to update my fork to reflect that if you'd like to see it? Best, Austin On Thursday, February 18, 2021 at 5:01:16 PM UTC-5 Austin Cawley-Edwards wrote: > We could try out EDS, though it seems like a very envoy-specific feature > without a good way to provide non-envoy labels. The only way that seems > possible with EDS is via the LbEndpoint's metadata[1], though that very > specifically not what it is designed for. > > [1]: > https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#config-endpoint-v3-lbendpoint > On Thursday, February 18, 2021 at 4:52:52 PM UTC-5 Austin Cawley-Edwards > wrote: > >> Yeah, that would be nice too but would not solve the original use case of >> this thread, which was to provide a generic option that did not need to >> co-locate data next to Prometheus via a filesystem. I think xDS is still a >> good fit for a generic option, but we would need to define an API. >> >> If we were to just directly use the structure of file_sd, then we're not >> creating anything new, just joining the xDS discovery protocol with an >> existing data structure. >> >> >> What are you thinking? Do you want to open a new thread to see if anyone >> would find an envoy_ds valuable? >> >> Best, >> Austin >> >> On Thursday, February 18, 2021 at 4:45:05 PM UTC-5 Julien Pivotto wrote: >> >>> Yeah my impression was that eDS would fit here, what envoy uses for >>> endpoints. >>> >>> >>> On 18 Feb 13:31, Austin Cawley-Edwards wrote: >>> > Quick follow-up: I think an "envoy discovery" method to scrape >>> > Envoy-specific metrics might be worthwhile, but I don't think it fits >>> in >>> > the same conversation as this one. >>> > >>> > On Thursday, February 18, 2021 at 4:12:39 PM UTC-5 Austin >>> Cawley-Edwards >>> > wrote: >>> > >>> > > Hey Julien, >>> > > >>> > > I think the disconnect here is that xDS is just a protocol used for >>> many >>> > > different Envoy services[1], and does not define any actual data >>> that could >>> > > be mapped to a target itself. It is purely meant as a base that is >>> to be >>> > > extended. Can you clarify what you mean by "current xds api's"? Do >>> you mean >>> > > just discovering the current Envoys in a cluster? Or do you mean >>> trying to >>> > > discover all the various Listeners/ VirtualHosts[2] (which may or >>> may not >>> > > expose a metrics endpoint) from Envoy? Envoy does not implement >>> anything >>> > > about metrics of the underlying services it proxies, i.e. there is >>> no way >>> > > to say "these services are good to scrape" to my knowledge. >>> > > >>> > > For HTTP vs gRPC, I'm not sure why we wouldn't want to support both >>> as the >>> > > xDS protocol defines support for both, but gRPC is more common. >>> > > >>> > > Best, >>> > > Austin >>> > > >>> > > [1]: >>> https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/service >>> > > [2]: >>> > > >>> https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#resource-types >>> >>> > > >>> > > On Thursday, February 18, 2021 at 3:20:18 PM UTC-5 Julien Pivotto >>> wrote: >>> > > >>> > >> On 18 Feb 12:10, Austin Cawley-Edwards wrote: >>> > >> > Hey Julien, >>> > >> > >>> > >> > I've finished up a *rough *proof of concept for what xDS support >>> in >>> > >> > Prometheus could look like [1]. I think it shows the main areas >>> without >>> > >> > going too over-the-top. In summary, the two main pieces to the >>> > >> integration >>> > >> > seem to be an xDS API definition[2] and an xDS client/ discovery >>> that >>> > >> uses >>> > >> > an implementation of this API [3] (HTTP discovery implementation >>> > >> here[4]). >>> > >> > For the xDS API, envoy does not have a native concept of metrics/ >>> > >> > monitoring, so we need to define this data structure -- I've >>> currently >>> > >> > sketched one out (taking from Kuma) that is very close to a >>> > >> targetgroup[5]. >>> > >> > >>> > >> > A few pros: >>> > >> > - support for both HTTP and gRPC API implementations >>> > >> > - xDS protocol is well documented >>> > >> > - versioned API will make modifications possible >>> > >> > >>> > >> > A few cons: >>> > >> > - complexity of the protocol for people who have not used it >>> > >> > - added dependencies on gRPC and envoy + build processes >>> additions for >>> > >> > compiling protobuf >>> > >> > - xDS adds some overhead for implementors of this API, though >>> this >>> > >> seems >>> > >> > like an "advanced" use case where that is ok >>> > >> > >>> > >> > Please let me know what you think and if you think it's feasible >>> to >>> > >> move >>> > >> > forward with xDS or if we should go back to discussions of a >>> simpler >>> > >> HTTP/ >>> > >> > gRPC approach. >>> > >> >>> > >> Thanks, here are my initial thoughts: >>> > >> >>> > >> - We should not *add* an API, but discover everything on the xDS >>> API. >>> > >> The meta labels created would then enable to filter what is needed >>> and >>> > >> what is not. It is not the responsibility of the service discovery. >>> > >> Monitoring is not just about whitebox monitoring, the scope can be >>> > >> much larger. Also, I want something that could be used out of the >>> box >>> > >> with current xds api's. >>> > >> - We need to settle by using grpc or http, we would not support >>> both. >>> > >> Which one is the most used? >>> > >> >>> > >> So overall I don't think there should be any overhead for the >>> > >> implementors of the API, since my understating is that there are >>> out of >>> > >> the box implementations out there, like envoy? What am I missing? >>> > >> >>> > >> > >>> > >> > Best, >>> > >> > Austin >>> > >> > >>> > >> > [1]: >>> > >> >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds >>> > >> > [2]: >>> > >> > >>> > >> >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#xds-api-implementation >>> >>> > >> > [3]: >>> > >> > >>> > >> >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#discovery-scraping >>> >>> > >> > [4]: >>> > >> > >>> > >> >>> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/http_discovery.go >>> >>> > >> > [5]: >>> > >> > >>> > >> >>> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/api/v1alpha1/mads.proto >>> >>> > >> > On Saturday, February 6, 2021 at 7:04:21 PM UTC-5 Austin >>> Cawley-Edwards >>> > >> > wrote: >>> > >> > >>> > >> > > Hey Julien, >>> > >> > > >>> > >> > > Thanks for reaching back out about this. I’ve been on the midst >>> of a >>> > >> job >>> > >> > > transition and have not made any progress, but that is finally >>> fully >>> > >> done >>> > >> > > and I can try to move this along (though it will still be >>> slowly). >>> > >> I’ll get >>> > >> > > back to research and bring it up at the next Kuma community >>> meeting >>> > >> in two >>> > >> > > weeks, and then report back. >>> > >> > > >>> > >> > > Best, >>> > >> > > Austin >>> > >> > > >>> > >> > > On Sat, Feb 6, 2021 at 3:36 PM Julien Pivotto < >>> > >> [email protected]> >>> > >> > > wrote: >>> > >> > > >>> > >> > >> Hello Austin, >>> > >> > >> >>> > >> > >> Is there any progress here? I am really willing to learn more >>> about >>> > >> xds >>> > >> > >> and how it would fit here. >>> > >> > >> >>> > >> > >> On 29 Oct 11:09, Austin Cawley-Edwards wrote: >>> > >> > >> > Perfect, I'll get started collecting that data. Thanks for >>> the >>> > >> > >> direction. >>> > >> > >> > >>> > >> > >> > Best, >>> > >> > >> > Austin >>> > >> > >> > >>> > >> > >> > On Wed, Oct 28, 2020 at 1:07 PM Julien Pivotto < >>> > >> [email protected] >>> > >> > >> > >>> > >> > >> > wrote: >>> > >> > >> > >>> > >> > >> > > On 28 Oct 08:28, Austin Cawley-Edwards wrote: >>> > >> > >> > > > Hey Julien, >>> > >> > >> > > > >>> > >> > >> > > > Following up on if you've had time to think about how we >>> can >>> > >> move >>> > >> > >> > > forward >>> > >> > >> > > > on one of these paths. I haven't been able to find any >>> formal >>> > >> > >> proposals >>> > >> > >> > > for >>> > >> > >> > > > the other recently merged SDs like Eureka besides this >>> > >> comment[1]. >>> > >> > >> Would >>> > >> > >> > > it >>> > >> > >> > > > be valuable for us to work on something similar for >>> envoy's >>> > >> xDS in >>> > >> > >> the >>> > >> > >> > > > meantime? >>> > >> > >> > > >>> > >> > >> > > >>> > >> > >> > > Let's be honest that recent service discoveries are >>> 'small', and >>> > >> that >>> > >> > >> > > big ones are quite old now. >>> > >> > >> > > >>> > >> > >> > > I still need to gather intel on xDS; but let's start by >>> > >> collecting >>> > >> > >> data: >>> > >> > >> > > >>> > >> > >> > > - What would the configuration look like (does it need >>> multiple >>> > >> > >> > > `roles`) >>> > >> > >> > > - Which meta labels could we have exposed >>> > >> > >> > > - How would we set the address >>> > >> > >> > > >>> > >> > >> > > That hopefully should not be too much work and then we can >>> see >>> > >> how/if >>> > >> > >> we >>> > >> > >> > > move forward with this. >>> > >> > >> > > >>> > >> > >> > > > >>> > >> > >> > > > Thanks, >>> > >> > >> > > > Austin >>> > >> > >> > > > >>> > >> > >> > > > [1]: >>> > >> > >> > > > >>> > >> > >> > > >>> > >> > >> >>> > >> >>> https://github.com/prometheus/prometheus/pull/3369#issuecomment-667344846 >>> > >> > >> > > > >>> > >> > >> > > > On Wednesday, October 21, 2020 at 6:49:59 PM UTC-4 >>> Austin >>> > >> > >> Cawley-Edwards >>> > >> > >> > > > wrote: >>> > >> > >> > > > >>> > >> > >> > > > > That makes sense to me, and would be nice to build of >>> an >>> > >> existing >>> > >> > >> > > > > protocol, and agree it sounds like a large effort. I’m >>> > >> pretty new >>> > >> > >> to >>> > >> > >> > > xDS as >>> > >> > >> > > > > well, though we have many people on the Kuma team who >>> are >>> > >> quite >>> > >> > >> > > > > knowledgeable — I’d be happy to set up a call/ bring >>> them in >>> > >> to >>> > >> > >> help >>> > >> > >> > > size >>> > >> > >> > > > > the work if you’d be open to that. >>> > >> > >> > > > > >>> > >> > >> > > > > And understand + agree on the configuration approach >>> for the >>> > >> extra >>> > >> > >> > > layer. >>> > >> > >> > > > > The main difficulty for users is configuring the >>> file_sd + >>> > >> > >> sidecar, >>> > >> > >> > > > > relabel_configs should be fairly straightforward and >>> > >> accessible. >>> > >> > >> > > > > >>> > >> > >> > > > > Austin >>> > >> > >> > > > > >>> > >> > >> > > > > On Wed, Oct 21, 2020 at 6:37 PM Julien Pivotto < >>> > >> > >> > > [email protected]> >>> > >> > >> > > > > wrote: >>> > >> > >> > > > > >>> > >> > >> > > > >> On 21 Oct 15:12, Austin Cawley-Edwards wrote: >>> > >> > >> > > > >> > Hey all, >>> > >> > >> > > > >> > >>> > >> > >> > > > >> > I've gathered up some research about our efforts >>> with the >>> > >> DNS >>> > >> > >> SD >>> > >> > >> > > > >> option, >>> > >> > >> > > > >> > please let me know if I've overlooked something. >>> > >> > >> > > > >> >>> > >> > >> > > > >> I have been looking on my side too - I think that >>> what the >>> > >> option >>> > >> > >> > > that's >>> > >> > >> > > > >> more likely on the table would not to have a >>> "generic" >>> > >> > >> implementation, >>> > >> > >> > > > >> but a xDS protocol service discovery. In that case, >>> it >>> > >> would not >>> > >> > >> be an >>> > >> > >> > > > >> issue to have that in Prometheus (even the gRPC >>> version). >>> > >> > >> > > > >> >>> > >> > >> > > > >> However, the extra layer that kuma is offering in the >>> SD >>> > >> should >>> > >> > >> be >>> > >> > >> > > done >>> > >> > >> > > > >> at the Prometheus configuration level - e.g. by the >>> user, >>> > >> in the >>> > >> > >> > > > >> relabel_configs. >>> > >> > >> > > > >> >>> > >> > >> > > > >> I think I'd like to better understand xDS but it >>> seems that >>> > >> it >>> > >> > >> could >>> > >> > >> > > be >>> > >> > >> > > > >> good option for a service discovery in Prometheus, at >>> least >>> > >> at >>> > >> > >> first >>> > >> > >> > > > >> sight. >>> > >> > >> > > > >> >>> > >> > >> > > > >> We are looking at a huge amount of work here I think. >>> > >> > >> > > > >> >>> > >> > >> > > > >> -- >>> > >> > >> > > > >> Julien Pivotto >>> > >> > >> > > > >> @roidelapluie >>> > >> > >> > > > >> >>> > >> > >> > > > > >>> > >> > >> > > > >>> > >> > >> > > > -- >>> > >> > >> > > > You received this message because you are subscribed to >>> the >>> > >> Google >>> > >> > >> > > Groups "Prometheus Developers" group. >>> > >> > >> > > > To unsubscribe from this group and stop receiving emails >>> from >>> > >> it, >>> > >> > >> send >>> > >> > >> > > an email to [email protected]. >>> > >> > >> > > > To view this discussion on the web visit >>> > >> > >> > > >>> > >> > >> >>> > >> >>> https://groups.google.com/d/msgid/prometheus-developers/89922987-42e4-4807-9956-5590cb2de558n%40googlegroups.com >>> >>> > >> > >> > > . >>> > >> > >> > > >>> > >> > >> > > >>> > >> > >> > > -- >>> > >> > >> > > Julien Pivotto >>> > >> > >> > > @roidelapluie >>> > >> > >> > > >>> > >> > >> > >>> > >> > >> > -- >>> > >> > >> > You received this message because you are subscribed to the >>> Google >>> > >> > >> Groups "Prometheus Developers" group. >>> > >> > >> > To unsubscribe from this group and stop receiving emails >>> from it, >>> > >> send >>> > >> > >> an email to [email protected]. >>> > >> > >> > To view this discussion on the web visit >>> > >> > >> >>> > >> >>> https://groups.google.com/d/msgid/prometheus-developers/CALGL%2BUBExJmaD9rzXWnTJf5h6my3cbYrrh-uQjCD9U9PGoW-Aw%40mail.gmail.com >>> >>> > >> > >> . >>> > >> > >> >>> > >> > >> -- >>> > >> > >> Julien Pivotto >>> > >> > >> @roidelapluie >>> > >> > >> >>> > >> > > >>> > >> > >>> > >> > -- >>> > >> > You received this message because you are subscribed to the >>> Google >>> > >> Groups "Prometheus Developers" group. >>> > >> > To unsubscribe from this group and stop receiving emails from it, >>> send >>> > >> an email to [email protected]. >>> > >> > To view this discussion on the web visit >>> > >> >>> https://groups.google.com/d/msgid/prometheus-developers/7db91c8f-33e2-4253-8cc6-daf596a1d1ban%40googlegroups.com. >>> >>> >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> Julien Pivotto >>> > >> @roidelapluie >>> > >> >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Developers" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-developers/47fdf607-fac5-41bd-94ea-683fe67e38bfn%40googlegroups.com. >>> >>> >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/25b690f4-a24c-42df-b0e5-909a4e55287cn%40googlegroups.com.
