Another option if your security will allow opening one port to Prometheus data, there is the "exporter_exporter"
https://github.com/QubitProducts/exporter_exporter On the subject of external labels, you should have those anyway on your Prometheus instances otherwise you can't identify them properly. On Thu, Mar 25, 2021 at 2:59 PM robbe vaes <[email protected]> wrote: > Alright, did not know all of that. We are now setting up the Prometheus > Proxy and are hoping that it does what we want it to do. VictoriaMetrics > would be a last case scenario anyway > > Op donderdag 25 maart 2021 om 14:54:22 UTC+1 schreef [email protected] > : > >> What should be noted when bringing up VictoriaMetrics in the Prometheus >> context is that it is deliberately incompatible with Prometheus in multiple >> ways: >> >> - VM's MetricsQL behaves differently from PromQL in a multitude of ways >> and is *not* backwards-compatible with PromQL (see also >> https://promlabs.com/promql-compliance-tests). >> - The VMAgent does remote_write without the necessary pre-processing of >> series staleness handling, and it throws away datapoints like special float >> values (like NaN). At least last time I checked. >> - The TSDB has lossy compression, so it throws away part of your data >> even further. Prometheus and Thanos store and retrieve float sample values >> exactly as sent, which depending on the use case, may matter. >> >> All that is to say that you can use VictoriaMetrics, but you are "on your >> own" then in terms of proper Prometheus ecosystem support and compatibility. >> >> On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <[email protected]> wrote: >> >>> It is correct that Thanos' deduplication works when using just 2 or more >>> Prometheus servers. The thing is, Thanos cannot use existing labels to do >>> deduplication. I want it to deduplicate using the label called >>> exported_instance, but i cannot predefine this label in the external labels >>> for thanos because it is not from prometheus but from the collectd >>> exporter. We are now looking into Prometheus Proxy, since we already tried >>> PushProx and it did not do what we wanted it to do. VictoriaMetrics might >>> be interesting to look into as well! >>> >>> Regards, >>> Robbe >>> >>> Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef [email protected]: >>> >>>> Probably VictoriaMetrics could help with this setup. It supports data >>>> push via various protocols >>>> <https://victoriametrics.github.io/#how-to-import-time-series-data>, >>>> it supports data deduplication >>>> <https://victoriametrics.github.io/#deduplication> and it is compatible >>>> with Prometheus datasource in Grafana >>>> <https://victoriametrics.github.io/#grafana-setup>. >>>> >>>> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <[email protected]> wrote: >>>> >>>>> The thing is we can implement multiple prometheus instances as well, >>>>> thats no issue and will probably happen anyway. The thing is, we tried >>>>> using Thanos to manage multiple prometheus servers and do deduplication, >>>>> but the deduplication does not work for the collectd exporters. The >>>>> problem >>>>> its having i think is that we need to filter out duplicate data by >>>>> exported >>>>> instance, but this is not possible for Thanos since it needs predefined >>>>> external labels within prometheus. >>>>> >>>>> Regards, >>>>> >>>>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>>>> >>>>>> The way this is usually solved is by duplicating prometheus - it >>>>>> seems >>>>>> that now you have moved the SPOF from the exporter to prometheus. >>>>>> >>>>>> Regards, >>>>>> >>>>>> On 25 Mar 06:01, robbe vaes wrote: >>>>>> > Okay so, we want to have an environment using Prometheus, where we >>>>>> can >>>>>> > monitor our servers etc with a push method rather then pull due to >>>>>> network >>>>>> > security aspects. As of now, we managed to set up collectd together >>>>>> with >>>>>> > collectd exporter for prometheus. This way we can have the clients >>>>>> or >>>>>> > servers push their data to the exporter and have prometheus scrape >>>>>> the data >>>>>> > from the exporter to minimize the amount of pulling that has to be >>>>>> done. >>>>>> > The issue now is, we want to have multiple collectd exporters to >>>>>> improve >>>>>> > HA, but the problem now is that in prometheus, when scraping both >>>>>> > exporters, it takes the data from both. It shows for example for >>>>>> server A >>>>>> > the cpu load but it shows it twice instead of only once because its >>>>>> being >>>>>> > pulled from both exporters. We want to have something like this but >>>>>> it >>>>>> > needs to have a form of deduplication so we dont have the same data >>>>>> twice. >>>>>> > >>>>>> > >>>>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef >>>>>> [email protected]: >>>>>> > >>>>>> > > Can you describe more about what your network topology is >>>>>> exactly? There >>>>>> > > are a number of solutions for dealing with distributed >>>>>> monitoring. >>>>>> > > >>>>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >>>>>> wrote: >>>>>> > > >>>>>> > >> Hi, >>>>>> > >> >>>>>> > >> I am trying to setup a monitoring environment with Prometheus, >>>>>> but it has >>>>>> > >> to be using a push mechanism instead of the standard pull >>>>>> mechanism >>>>>> > >> Prometheus uses. I was wondering what options there are to >>>>>> create an >>>>>> > >> environment like this. It would also have to perfom data >>>>>> deduplication. The >>>>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>>>> itself, >>>>>> > >> but rather that it scrapes a certain location for all the >>>>>> metrics and that >>>>>> > >> the clients push their metrics to that location automatically. >>>>>> > >> >>>>>> > >> Suggestions are very welcome! >>>>>> > >> >>>>>> > >> -- >>>>>> > >> You received this message because you are subscribed to the >>>>>> Google Groups >>>>>> > >> "Prometheus Users" group. >>>>>> > >> To unsubscribe from this group and stop receiving emails from >>>>>> it, send an >>>>>> > >> email to [email protected]. >>>>>> > >> To view this discussion on the web visit >>>>>> > >> >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>>>> > >> < >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> >>>>>> > >> . >>>>>> > >> >>>>>> > > >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> > To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Julien Pivotto >>>>>> @roidelapluie >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> >>>> Aliaksandr Valialkin, CTO VictoriaMetrics >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoXowBqQaKz0%3DBD5pBxrcs710xKBD0OFpZJ_%2BLMd8y3Xw%40mail.gmail.com.
