On 18 Feb 14:06, Austin Cawley-Edwards wrote: > > Well I think we should have a xds package which would have envoy_sd, and > > possibly kuma_sd, but stay scoped and tight to what exists _now_. Having > > them in a xds package enables us to reuse code. > > That could make sense to me, though in the long term seems like it would be > more of a burden in the long run that fragments xDS implementations and > brings more dependencies into Prometheus. > > I don't think it would make sense to implement this for envoy without > someone asking for it, but it could be done for Kuma where we have an xDS > client in the xds package, and then pull in Kuma's xDS API for Monitoring > assignments. I'd be happy to update my fork to reflect that if you'd like > to see it?
I don't want to instruct you to do a lot of "free" work given I don't really know where we are heading but I am willing to see what it would look like with existing kuma xds api, over HTTP (grpc is currently quite a pain to update and would cause short-term issues to integrate in Prometheus, so if http is available let's use that). > > Best, > Austin > > On Thursday, February 18, 2021 at 5:01:16 PM UTC-5 Austin Cawley-Edwards > wrote: > > > We could try out EDS, though it seems like a very envoy-specific feature > > without a good way to provide non-envoy labels. The only way that seems > > possible with EDS is via the LbEndpoint's metadata[1], though that very > > specifically not what it is designed for. > > > > [1]: > > https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#config-endpoint-v3-lbendpoint > > On Thursday, February 18, 2021 at 4:52:52 PM UTC-5 Austin Cawley-Edwards > > wrote: > > > >> Yeah, that would be nice too but would not solve the original use case of > >> this thread, which was to provide a generic option that did not need to > >> co-locate data next to Prometheus via a filesystem. I think xDS is still a > >> good fit for a generic option, but we would need to define an API. > >> > >> If we were to just directly use the structure of file_sd, then we're not > >> creating anything new, just joining the xDS discovery protocol with an > >> existing data structure. > >> > >> > >> What are you thinking? Do you want to open a new thread to see if anyone > >> would find an envoy_ds valuable? > >> > >> Best, > >> Austin > >> > >> On Thursday, February 18, 2021 at 4:45:05 PM UTC-5 Julien Pivotto wrote: > >> > >>> Yeah my impression was that eDS would fit here, what envoy uses for > >>> endpoints. > >>> > >>> > >>> On 18 Feb 13:31, Austin Cawley-Edwards wrote: > >>> > Quick follow-up: I think an "envoy discovery" method to scrape > >>> > Envoy-specific metrics might be worthwhile, but I don't think it fits > >>> in > >>> > the same conversation as this one. > >>> > > >>> > On Thursday, February 18, 2021 at 4:12:39 PM UTC-5 Austin > >>> Cawley-Edwards > >>> > wrote: > >>> > > >>> > > Hey Julien, > >>> > > > >>> > > I think the disconnect here is that xDS is just a protocol used for > >>> many > >>> > > different Envoy services[1], and does not define any actual data > >>> that could > >>> > > be mapped to a target itself. It is purely meant as a base that is > >>> to be > >>> > > extended. Can you clarify what you mean by "current xds api's"? Do > >>> you mean > >>> > > just discovering the current Envoys in a cluster? Or do you mean > >>> trying to > >>> > > discover all the various Listeners/ VirtualHosts[2] (which may or > >>> may not > >>> > > expose a metrics endpoint) from Envoy? Envoy does not implement > >>> anything > >>> > > about metrics of the underlying services it proxies, i.e. there is > >>> no way > >>> > > to say "these services are good to scrape" to my knowledge. > >>> > > > >>> > > For HTTP vs gRPC, I'm not sure why we wouldn't want to support both > >>> as the > >>> > > xDS protocol defines support for both, but gRPC is more common. > >>> > > > >>> > > Best, > >>> > > Austin > >>> > > > >>> > > [1]: > >>> https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/service > >>> > > [2]: > >>> > > > >>> https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#resource-types > >>> > >>> > > > >>> > > On Thursday, February 18, 2021 at 3:20:18 PM UTC-5 Julien Pivotto > >>> wrote: > >>> > > > >>> > >> On 18 Feb 12:10, Austin Cawley-Edwards wrote: > >>> > >> > Hey Julien, > >>> > >> > > >>> > >> > I've finished up a *rough *proof of concept for what xDS support > >>> in > >>> > >> > Prometheus could look like [1]. I think it shows the main areas > >>> without > >>> > >> > going too over-the-top. In summary, the two main pieces to the > >>> > >> integration > >>> > >> > seem to be an xDS API definition[2] and an xDS client/ discovery > >>> that > >>> > >> uses > >>> > >> > an implementation of this API [3] (HTTP discovery implementation > >>> > >> here[4]). > >>> > >> > For the xDS API, envoy does not have a native concept of metrics/ > >>> > >> > monitoring, so we need to define this data structure -- I've > >>> currently > >>> > >> > sketched one out (taking from Kuma) that is very close to a > >>> > >> targetgroup[5]. > >>> > >> > > >>> > >> > A few pros: > >>> > >> > - support for both HTTP and gRPC API implementations > >>> > >> > - xDS protocol is well documented > >>> > >> > - versioned API will make modifications possible > >>> > >> > > >>> > >> > A few cons: > >>> > >> > - complexity of the protocol for people who have not used it > >>> > >> > - added dependencies on gRPC and envoy + build processes > >>> additions for > >>> > >> > compiling protobuf > >>> > >> > - xDS adds some overhead for implementors of this API, though > >>> this > >>> > >> seems > >>> > >> > like an "advanced" use case where that is ok > >>> > >> > > >>> > >> > Please let me know what you think and if you think it's feasible > >>> to > >>> > >> move > >>> > >> > forward with xDS or if we should go back to discussions of a > >>> simpler > >>> > >> HTTP/ > >>> > >> > gRPC approach. > >>> > >> > >>> > >> Thanks, here are my initial thoughts: > >>> > >> > >>> > >> - We should not *add* an API, but discover everything on the xDS > >>> API. > >>> > >> The meta labels created would then enable to filter what is needed > >>> and > >>> > >> what is not. It is not the responsibility of the service discovery. > >>> > >> Monitoring is not just about whitebox monitoring, the scope can be > >>> > >> much larger. Also, I want something that could be used out of the > >>> box > >>> > >> with current xds api's. > >>> > >> - We need to settle by using grpc or http, we would not support > >>> both. > >>> > >> Which one is the most used? > >>> > >> > >>> > >> So overall I don't think there should be any overhead for the > >>> > >> implementors of the API, since my understating is that there are > >>> out of > >>> > >> the box implementations out there, like envoy? What am I missing? > >>> > >> > >>> > >> > > >>> > >> > Best, > >>> > >> > Austin > >>> > >> > > >>> > >> > [1]: > >>> > >> > >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds > >>> > >> > [2]: > >>> > >> > > >>> > >> > >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#xds-api-implementation > >>> > >>> > >> > [3]: > >>> > >> > > >>> > >> > >>> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#discovery-scraping > >>> > >>> > >> > [4]: > >>> > >> > > >>> > >> > >>> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/http_discovery.go > >>> > >>> > >> > [5]: > >>> > >> > > >>> > >> > >>> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/api/v1alpha1/mads.proto > >>> > >>> > >> > On Saturday, February 6, 2021 at 7:04:21 PM UTC-5 Austin > >>> Cawley-Edwards > >>> > >> > wrote: > >>> > >> > > >>> > >> > > Hey Julien, > >>> > >> > > > >>> > >> > > Thanks for reaching back out about this. I’ve been on the midst > >>> of a > >>> > >> job > >>> > >> > > transition and have not made any progress, but that is finally > >>> fully > >>> > >> done > >>> > >> > > and I can try to move this along (though it will still be > >>> slowly). > >>> > >> I’ll get > >>> > >> > > back to research and bring it up at the next Kuma community > >>> meeting > >>> > >> in two > >>> > >> > > weeks, and then report back. > >>> > >> > > > >>> > >> > > Best, > >>> > >> > > Austin > >>> > >> > > > >>> > >> > > On Sat, Feb 6, 2021 at 3:36 PM Julien Pivotto < > >>> > >> [email protected]> > >>> > >> > > wrote: > >>> > >> > > > >>> > >> > >> Hello Austin, > >>> > >> > >> > >>> > >> > >> Is there any progress here? I am really willing to learn more > >>> about > >>> > >> xds > >>> > >> > >> and how it would fit here. > >>> > >> > >> > >>> > >> > >> On 29 Oct 11:09, Austin Cawley-Edwards wrote: > >>> > >> > >> > Perfect, I'll get started collecting that data. Thanks for > >>> the > >>> > >> > >> direction. > >>> > >> > >> > > >>> > >> > >> > Best, > >>> > >> > >> > Austin > >>> > >> > >> > > >>> > >> > >> > On Wed, Oct 28, 2020 at 1:07 PM Julien Pivotto < > >>> > >> [email protected] > >>> > >> > >> > > >>> > >> > >> > wrote: > >>> > >> > >> > > >>> > >> > >> > > On 28 Oct 08:28, Austin Cawley-Edwards wrote: > >>> > >> > >> > > > Hey Julien, > >>> > >> > >> > > > > >>> > >> > >> > > > Following up on if you've had time to think about how we > >>> can > >>> > >> move > >>> > >> > >> > > forward > >>> > >> > >> > > > on one of these paths. I haven't been able to find any > >>> formal > >>> > >> > >> proposals > >>> > >> > >> > > for > >>> > >> > >> > > > the other recently merged SDs like Eureka besides this > >>> > >> comment[1]. > >>> > >> > >> Would > >>> > >> > >> > > it > >>> > >> > >> > > > be valuable for us to work on something similar for > >>> envoy's > >>> > >> xDS in > >>> > >> > >> the > >>> > >> > >> > > > meantime? > >>> > >> > >> > > > >>> > >> > >> > > > >>> > >> > >> > > Let's be honest that recent service discoveries are > >>> 'small', and > >>> > >> that > >>> > >> > >> > > big ones are quite old now. > >>> > >> > >> > > > >>> > >> > >> > > I still need to gather intel on xDS; but let's start by > >>> > >> collecting > >>> > >> > >> data: > >>> > >> > >> > > > >>> > >> > >> > > - What would the configuration look like (does it need > >>> multiple > >>> > >> > >> > > `roles`) > >>> > >> > >> > > - Which meta labels could we have exposed > >>> > >> > >> > > - How would we set the address > >>> > >> > >> > > > >>> > >> > >> > > That hopefully should not be too much work and then we can > >>> see > >>> > >> how/if > >>> > >> > >> we > >>> > >> > >> > > move forward with this. > >>> > >> > >> > > > >>> > >> > >> > > > > >>> > >> > >> > > > Thanks, > >>> > >> > >> > > > Austin > >>> > >> > >> > > > > >>> > >> > >> > > > [1]: > >>> > >> > >> > > > > >>> > >> > >> > > > >>> > >> > >> > >>> > >> > >>> https://github.com/prometheus/prometheus/pull/3369#issuecomment-667344846 > >>> > >> > >> > > > > >>> > >> > >> > > > On Wednesday, October 21, 2020 at 6:49:59 PM UTC-4 > >>> Austin > >>> > >> > >> Cawley-Edwards > >>> > >> > >> > > > wrote: > >>> > >> > >> > > > > >>> > >> > >> > > > > That makes sense to me, and would be nice to build of > >>> an > >>> > >> existing > >>> > >> > >> > > > > protocol, and agree it sounds like a large effort. I’m > >>> > >> pretty new > >>> > >> > >> to > >>> > >> > >> > > xDS as > >>> > >> > >> > > > > well, though we have many people on the Kuma team who > >>> are > >>> > >> quite > >>> > >> > >> > > > > knowledgeable — I’d be happy to set up a call/ bring > >>> them in > >>> > >> to > >>> > >> > >> help > >>> > >> > >> > > size > >>> > >> > >> > > > > the work if you’d be open to that. > >>> > >> > >> > > > > > >>> > >> > >> > > > > And understand + agree on the configuration approach > >>> for the > >>> > >> extra > >>> > >> > >> > > layer. > >>> > >> > >> > > > > The main difficulty for users is configuring the > >>> file_sd + > >>> > >> > >> sidecar, > >>> > >> > >> > > > > relabel_configs should be fairly straightforward and > >>> > >> accessible. > >>> > >> > >> > > > > > >>> > >> > >> > > > > Austin > >>> > >> > >> > > > > > >>> > >> > >> > > > > On Wed, Oct 21, 2020 at 6:37 PM Julien Pivotto < > >>> > >> > >> > > [email protected]> > >>> > >> > >> > > > > wrote: > >>> > >> > >> > > > > > >>> > >> > >> > > > >> On 21 Oct 15:12, Austin Cawley-Edwards wrote: > >>> > >> > >> > > > >> > Hey all, > >>> > >> > >> > > > >> > > >>> > >> > >> > > > >> > I've gathered up some research about our efforts > >>> with the > >>> > >> DNS > >>> > >> > >> SD > >>> > >> > >> > > > >> option, > >>> > >> > >> > > > >> > please let me know if I've overlooked something. > >>> > >> > >> > > > >> > >>> > >> > >> > > > >> I have been looking on my side too - I think that > >>> what the > >>> > >> option > >>> > >> > >> > > that's > >>> > >> > >> > > > >> more likely on the table would not to have a > >>> "generic" > >>> > >> > >> implementation, > >>> > >> > >> > > > >> but a xDS protocol service discovery. In that case, > >>> it > >>> > >> would not > >>> > >> > >> be an > >>> > >> > >> > > > >> issue to have that in Prometheus (even the gRPC > >>> version). > >>> > >> > >> > > > >> > >>> > >> > >> > > > >> However, the extra layer that kuma is offering in the > >>> SD > >>> > >> should > >>> > >> > >> be > >>> > >> > >> > > done > >>> > >> > >> > > > >> at the Prometheus configuration level - e.g. by the > >>> user, > >>> > >> in the > >>> > >> > >> > > > >> relabel_configs. > >>> > >> > >> > > > >> > >>> > >> > >> > > > >> I think I'd like to better understand xDS but it > >>> seems that > >>> > >> it > >>> > >> > >> could > >>> > >> > >> > > be > >>> > >> > >> > > > >> good option for a service discovery in Prometheus, at > >>> least > >>> > >> at > >>> > >> > >> first > >>> > >> > >> > > > >> sight. > >>> > >> > >> > > > >> > >>> > >> > >> > > > >> We are looking at a huge amount of work here I think. > >>> > >> > >> > > > >> > >>> > >> > >> > > > >> -- > >>> > >> > >> > > > >> Julien Pivotto > >>> > >> > >> > > > >> @roidelapluie > >>> > >> > >> > > > >> > >>> > >> > >> > > > > > >>> > >> > >> > > > > >>> > >> > >> > > > -- > >>> > >> > >> > > > You received this message because you are subscribed to > >>> the > >>> > >> Google > >>> > >> > >> > > Groups "Prometheus Developers" group. > >>> > >> > >> > > > To unsubscribe from this group and stop receiving emails > >>> from > >>> > >> it, > >>> > >> > >> send > >>> > >> > >> > > an email to [email protected]. > >>> > >> > >> > > > To view this discussion on the web visit > >>> > >> > >> > > > >>> > >> > >> > >>> > >> > >>> https://groups.google.com/d/msgid/prometheus-developers/89922987-42e4-4807-9956-5590cb2de558n%40googlegroups.com > >>> > >>> > >> > >> > > . > >>> > >> > >> > > > >>> > >> > >> > > > >>> > >> > >> > > -- > >>> > >> > >> > > Julien Pivotto > >>> > >> > >> > > @roidelapluie > >>> > >> > >> > > > >>> > >> > >> > > >>> > >> > >> > -- > >>> > >> > >> > You received this message because you are subscribed to the > >>> Google > >>> > >> > >> Groups "Prometheus Developers" group. > >>> > >> > >> > To unsubscribe from this group and stop receiving emails > >>> from it, > >>> > >> send > >>> > >> > >> an email to [email protected]. > >>> > >> > >> > To view this discussion on the web visit > >>> > >> > >> > >>> > >> > >>> https://groups.google.com/d/msgid/prometheus-developers/CALGL%2BUBExJmaD9rzXWnTJf5h6my3cbYrrh-uQjCD9U9PGoW-Aw%40mail.gmail.com > >>> > >>> > >> > >> . > >>> > >> > >> > >>> > >> > >> -- > >>> > >> > >> Julien Pivotto > >>> > >> > >> @roidelapluie > >>> > >> > >> > >>> > >> > > > >>> > >> > > >>> > >> > -- > >>> > >> > You received this message because you are subscribed to the > >>> Google > >>> > >> Groups "Prometheus Developers" group. > >>> > >> > To unsubscribe from this group and stop receiving emails from it, > >>> send > >>> > >> an email to [email protected]. > >>> > >> > To view this discussion on the web visit > >>> > >> > >>> https://groups.google.com/d/msgid/prometheus-developers/7db91c8f-33e2-4253-8cc6-daf596a1d1ban%40googlegroups.com. > >>> > >>> > >>> > >> > >>> > >> > >>> > >> > >>> > >> -- > >>> > >> Julien Pivotto > >>> > >> @roidelapluie > >>> > >> > >>> > > > >>> > > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> Groups "Prometheus Developers" group. > >>> > To unsubscribe from this group and stop receiving emails from it, send > >>> an email to [email protected]. > >>> > To view this discussion on the web visit > >>> https://groups.google.com/d/msgid/prometheus-developers/47fdf607-fac5-41bd-94ea-683fe67e38bfn%40googlegroups.com. > >>> > >>> > >>> > >>> > >>> -- > >>> Julien Pivotto > >>> @roidelapluie > >>> > >> > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/25b690f4-a24c-42df-b0e5-909a4e55287cn%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20210218224035.GA369776%40oxygen.
