We could try out EDS, though it seems like a very envoy-specific feature without a good way to provide non-envoy labels. The only way that seems possible with EDS is via the LbEndpoint's metadata[1], though that very specifically not what it is designed for.
[1]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#config-endpoint-v3-lbendpoint On Thursday, February 18, 2021 at 4:52:52 PM UTC-5 Austin Cawley-Edwards wrote: > Yeah, that would be nice too but would not solve the original use case of > this thread, which was to provide a generic option that did not need to > co-locate data next to Prometheus via a filesystem. I think xDS is still a > good fit for a generic option, but we would need to define an API. > > If we were to just directly use the structure of file_sd, then we're not > creating anything new, just joining the xDS discovery protocol with an > existing data structure. > > > What are you thinking? Do you want to open a new thread to see if anyone > would find an envoy_ds valuable? > > Best, > Austin > > On Thursday, February 18, 2021 at 4:45:05 PM UTC-5 Julien Pivotto wrote: > >> Yeah my impression was that eDS would fit here, what envoy uses for >> endpoints. >> >> >> On 18 Feb 13:31, Austin Cawley-Edwards wrote: >> > Quick follow-up: I think an "envoy discovery" method to scrape >> > Envoy-specific metrics might be worthwhile, but I don't think it fits >> in >> > the same conversation as this one. >> > >> > On Thursday, February 18, 2021 at 4:12:39 PM UTC-5 Austin >> Cawley-Edwards >> > wrote: >> > >> > > Hey Julien, >> > > >> > > I think the disconnect here is that xDS is just a protocol used for >> many >> > > different Envoy services[1], and does not define any actual data that >> could >> > > be mapped to a target itself. It is purely meant as a base that is to >> be >> > > extended. Can you clarify what you mean by "current xds api's"? Do >> you mean >> > > just discovering the current Envoys in a cluster? Or do you mean >> trying to >> > > discover all the various Listeners/ VirtualHosts[2] (which may or may >> not >> > > expose a metrics endpoint) from Envoy? Envoy does not implement >> anything >> > > about metrics of the underlying services it proxies, i.e. there is no >> way >> > > to say "these services are good to scrape" to my knowledge. >> > > >> > > For HTTP vs gRPC, I'm not sure why we wouldn't want to support both >> as the >> > > xDS protocol defines support for both, but gRPC is more common. >> > > >> > > Best, >> > > Austin >> > > >> > > [1]: >> https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/service >> > > [2]: >> > > >> https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#resource-types >> >> > > >> > > On Thursday, February 18, 2021 at 3:20:18 PM UTC-5 Julien Pivotto >> wrote: >> > > >> > >> On 18 Feb 12:10, Austin Cawley-Edwards wrote: >> > >> > Hey Julien, >> > >> > >> > >> > I've finished up a *rough *proof of concept for what xDS support >> in >> > >> > Prometheus could look like [1]. I think it shows the main areas >> without >> > >> > going too over-the-top. In summary, the two main pieces to the >> > >> integration >> > >> > seem to be an xDS API definition[2] and an xDS client/ discovery >> that >> > >> uses >> > >> > an implementation of this API [3] (HTTP discovery implementation >> > >> here[4]). >> > >> > For the xDS API, envoy does not have a native concept of metrics/ >> > >> > monitoring, so we need to define this data structure -- I've >> currently >> > >> > sketched one out (taking from Kuma) that is very close to a >> > >> targetgroup[5]. >> > >> > >> > >> > A few pros: >> > >> > - support for both HTTP and gRPC API implementations >> > >> > - xDS protocol is well documented >> > >> > - versioned API will make modifications possible >> > >> > >> > >> > A few cons: >> > >> > - complexity of the protocol for people who have not used it >> > >> > - added dependencies on gRPC and envoy + build processes additions >> for >> > >> > compiling protobuf >> > >> > - xDS adds some overhead for implementors of this API, though this >> > >> seems >> > >> > like an "advanced" use case where that is ok >> > >> > >> > >> > Please let me know what you think and if you think it's feasible >> to >> > >> move >> > >> > forward with xDS or if we should go back to discussions of a >> simpler >> > >> HTTP/ >> > >> > gRPC approach. >> > >> >> > >> Thanks, here are my initial thoughts: >> > >> >> > >> - We should not *add* an API, but discover everything on the xDS >> API. >> > >> The meta labels created would then enable to filter what is needed >> and >> > >> what is not. It is not the responsibility of the service discovery. >> > >> Monitoring is not just about whitebox monitoring, the scope can be >> > >> much larger. Also, I want something that could be used out of the >> box >> > >> with current xds api's. >> > >> - We need to settle by using grpc or http, we would not support >> both. >> > >> Which one is the most used? >> > >> >> > >> So overall I don't think there should be any overhead for the >> > >> implementors of the API, since my understating is that there are out >> of >> > >> the box implementations out there, like envoy? What am I missing? >> > >> >> > >> > >> > >> > Best, >> > >> > Austin >> > >> > >> > >> > [1]: >> > >> >> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds >> > >> > [2]: >> > >> > >> > >> >> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#xds-api-implementation >> >> > >> > [3]: >> > >> > >> > >> >> https://github.com/austince/prometheus/tree/discovery-xds/discovery/xds#discovery-scraping >> >> > >> > [4]: >> > >> > >> > >> >> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/http_discovery.go >> >> > >> > [5]: >> > >> > >> > >> >> https://github.com/austince/prometheus/blob/discovery-xds/discovery/xds/api/v1alpha1/mads.proto >> >> > >> > On Saturday, February 6, 2021 at 7:04:21 PM UTC-5 Austin >> Cawley-Edwards >> > >> > wrote: >> > >> > >> > >> > > Hey Julien, >> > >> > > >> > >> > > Thanks for reaching back out about this. I’ve been on the midst >> of a >> > >> job >> > >> > > transition and have not made any progress, but that is finally >> fully >> > >> done >> > >> > > and I can try to move this along (though it will still be >> slowly). >> > >> I’ll get >> > >> > > back to research and bring it up at the next Kuma community >> meeting >> > >> in two >> > >> > > weeks, and then report back. >> > >> > > >> > >> > > Best, >> > >> > > Austin >> > >> > > >> > >> > > On Sat, Feb 6, 2021 at 3:36 PM Julien Pivotto < >> > >> [email protected]> >> > >> > > wrote: >> > >> > > >> > >> > >> Hello Austin, >> > >> > >> >> > >> > >> Is there any progress here? I am really willing to learn more >> about >> > >> xds >> > >> > >> and how it would fit here. >> > >> > >> >> > >> > >> On 29 Oct 11:09, Austin Cawley-Edwards wrote: >> > >> > >> > Perfect, I'll get started collecting that data. Thanks for >> the >> > >> > >> direction. >> > >> > >> > >> > >> > >> > Best, >> > >> > >> > Austin >> > >> > >> > >> > >> > >> > On Wed, Oct 28, 2020 at 1:07 PM Julien Pivotto < >> > >> [email protected] >> > >> > >> > >> > >> > >> > wrote: >> > >> > >> > >> > >> > >> > > On 28 Oct 08:28, Austin Cawley-Edwards wrote: >> > >> > >> > > > Hey Julien, >> > >> > >> > > > >> > >> > >> > > > Following up on if you've had time to think about how we >> can >> > >> move >> > >> > >> > > forward >> > >> > >> > > > on one of these paths. I haven't been able to find any >> formal >> > >> > >> proposals >> > >> > >> > > for >> > >> > >> > > > the other recently merged SDs like Eureka besides this >> > >> comment[1]. >> > >> > >> Would >> > >> > >> > > it >> > >> > >> > > > be valuable for us to work on something similar for >> envoy's >> > >> xDS in >> > >> > >> the >> > >> > >> > > > meantime? >> > >> > >> > > >> > >> > >> > > >> > >> > >> > > Let's be honest that recent service discoveries are >> 'small', and >> > >> that >> > >> > >> > > big ones are quite old now. >> > >> > >> > > >> > >> > >> > > I still need to gather intel on xDS; but let's start by >> > >> collecting >> > >> > >> data: >> > >> > >> > > >> > >> > >> > > - What would the configuration look like (does it need >> multiple >> > >> > >> > > `roles`) >> > >> > >> > > - Which meta labels could we have exposed >> > >> > >> > > - How would we set the address >> > >> > >> > > >> > >> > >> > > That hopefully should not be too much work and then we can >> see >> > >> how/if >> > >> > >> we >> > >> > >> > > move forward with this. >> > >> > >> > > >> > >> > >> > > > >> > >> > >> > > > Thanks, >> > >> > >> > > > Austin >> > >> > >> > > > >> > >> > >> > > > [1]: >> > >> > >> > > > >> > >> > >> > > >> > >> > >> >> > >> >> https://github.com/prometheus/prometheus/pull/3369#issuecomment-667344846 >> > >> > >> > > > >> > >> > >> > > > On Wednesday, October 21, 2020 at 6:49:59 PM UTC-4 Austin >> > >> > >> Cawley-Edwards >> > >> > >> > > > wrote: >> > >> > >> > > > >> > >> > >> > > > > That makes sense to me, and would be nice to build of >> an >> > >> existing >> > >> > >> > > > > protocol, and agree it sounds like a large effort. I’m >> > >> pretty new >> > >> > >> to >> > >> > >> > > xDS as >> > >> > >> > > > > well, though we have many people on the Kuma team who >> are >> > >> quite >> > >> > >> > > > > knowledgeable — I’d be happy to set up a call/ bring >> them in >> > >> to >> > >> > >> help >> > >> > >> > > size >> > >> > >> > > > > the work if you’d be open to that. >> > >> > >> > > > > >> > >> > >> > > > > And understand + agree on the configuration approach >> for the >> > >> extra >> > >> > >> > > layer. >> > >> > >> > > > > The main difficulty for users is configuring the >> file_sd + >> > >> > >> sidecar, >> > >> > >> > > > > relabel_configs should be fairly straightforward and >> > >> accessible. >> > >> > >> > > > > >> > >> > >> > > > > Austin >> > >> > >> > > > > >> > >> > >> > > > > On Wed, Oct 21, 2020 at 6:37 PM Julien Pivotto < >> > >> > >> > > [email protected]> >> > >> > >> > > > > wrote: >> > >> > >> > > > > >> > >> > >> > > > >> On 21 Oct 15:12, Austin Cawley-Edwards wrote: >> > >> > >> > > > >> > Hey all, >> > >> > >> > > > >> > >> > >> > >> > > > >> > I've gathered up some research about our efforts >> with the >> > >> DNS >> > >> > >> SD >> > >> > >> > > > >> option, >> > >> > >> > > > >> > please let me know if I've overlooked something. >> > >> > >> > > > >> >> > >> > >> > > > >> I have been looking on my side too - I think that what >> the >> > >> option >> > >> > >> > > that's >> > >> > >> > > > >> more likely on the table would not to have a "generic" >> > >> > >> implementation, >> > >> > >> > > > >> but a xDS protocol service discovery. In that case, it >> > >> would not >> > >> > >> be an >> > >> > >> > > > >> issue to have that in Prometheus (even the gRPC >> version). >> > >> > >> > > > >> >> > >> > >> > > > >> However, the extra layer that kuma is offering in the >> SD >> > >> should >> > >> > >> be >> > >> > >> > > done >> > >> > >> > > > >> at the Prometheus configuration level - e.g. by the >> user, >> > >> in the >> > >> > >> > > > >> relabel_configs. >> > >> > >> > > > >> >> > >> > >> > > > >> I think I'd like to better understand xDS but it seems >> that >> > >> it >> > >> > >> could >> > >> > >> > > be >> > >> > >> > > > >> good option for a service discovery in Prometheus, at >> least >> > >> at >> > >> > >> first >> > >> > >> > > > >> sight. >> > >> > >> > > > >> >> > >> > >> > > > >> We are looking at a huge amount of work here I think. >> > >> > >> > > > >> >> > >> > >> > > > >> -- >> > >> > >> > > > >> Julien Pivotto >> > >> > >> > > > >> @roidelapluie >> > >> > >> > > > >> >> > >> > >> > > > > >> > >> > >> > > > >> > >> > >> > > > -- >> > >> > >> > > > You received this message because you are subscribed to >> the >> > >> Google >> > >> > >> > > Groups "Prometheus Developers" group. >> > >> > >> > > > To unsubscribe from this group and stop receiving emails >> from >> > >> it, >> > >> > >> send >> > >> > >> > > an email to [email protected]. >> > >> > >> > > > To view this discussion on the web visit >> > >> > >> > > >> > >> > >> >> > >> >> https://groups.google.com/d/msgid/prometheus-developers/89922987-42e4-4807-9956-5590cb2de558n%40googlegroups.com >> >> > >> > >> > > . >> > >> > >> > > >> > >> > >> > > >> > >> > >> > > -- >> > >> > >> > > Julien Pivotto >> > >> > >> > > @roidelapluie >> > >> > >> > > >> > >> > >> > >> > >> > >> > -- >> > >> > >> > You received this message because you are subscribed to the >> Google >> > >> > >> Groups "Prometheus Developers" group. >> > >> > >> > To unsubscribe from this group and stop receiving emails from >> it, >> > >> send >> > >> > >> an email to [email protected]. >> > >> > >> > To view this discussion on the web visit >> > >> > >> >> > >> >> https://groups.google.com/d/msgid/prometheus-developers/CALGL%2BUBExJmaD9rzXWnTJf5h6my3cbYrrh-uQjCD9U9PGoW-Aw%40mail.gmail.com >> >> > >> > >> . >> > >> > >> >> > >> > >> -- >> > >> > >> Julien Pivotto >> > >> > >> @roidelapluie >> > >> > >> >> > >> > > >> > >> > >> > >> > -- >> > >> > You received this message because you are subscribed to the Google >> > >> Groups "Prometheus Developers" group. >> > >> > To unsubscribe from this group and stop receiving emails from it, >> send >> > >> an email to [email protected]. >> > >> > To view this discussion on the web visit >> > >> >> https://groups.google.com/d/msgid/prometheus-developers/7db91c8f-33e2-4253-8cc6-daf596a1d1ban%40googlegroups.com. >> >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Julien Pivotto >> > >> @roidelapluie >> > >> >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/47fdf607-fac5-41bd-94ea-683fe67e38bfn%40googlegroups.com. >> >> >> >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/16e4997f-3d3c-493c-b183-f503789cbf66n%40googlegroups.com.
