Alright, did not know all of that. We are now setting up the Prometheus Proxy and are hoping that it does what we want it to do. VictoriaMetrics would be a last case scenario anyway
Op donderdag 25 maart 2021 om 14:54:22 UTC+1 schreef [email protected]: > What should be noted when bringing up VictoriaMetrics in the Prometheus > context is that it is deliberately incompatible with Prometheus in multiple > ways: > > - VM's MetricsQL behaves differently from PromQL in a multitude of ways > and is *not* backwards-compatible with PromQL (see also > https://promlabs.com/promql-compliance-tests). > - The VMAgent does remote_write without the necessary pre-processing of > series staleness handling, and it throws away datapoints like special float > values (like NaN). At least last time I checked. > - The TSDB has lossy compression, so it throws away part of your data even > further. Prometheus and Thanos store and retrieve float sample values > exactly as sent, which depending on the use case, may matter. > > All that is to say that you can use VictoriaMetrics, but you are "on your > own" then in terms of proper Prometheus ecosystem support and compatibility. > > On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <[email protected]> wrote: > >> It is correct that Thanos' deduplication works when using just 2 or more >> Prometheus servers. The thing is, Thanos cannot use existing labels to do >> deduplication. I want it to deduplicate using the label called >> exported_instance, but i cannot predefine this label in the external labels >> for thanos because it is not from prometheus but from the collectd >> exporter. We are now looking into Prometheus Proxy, since we already tried >> PushProx and it did not do what we wanted it to do. VictoriaMetrics might >> be interesting to look into as well! >> >> Regards, >> Robbe >> >> Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef [email protected]: >> >>> Probably VictoriaMetrics could help with this setup. It supports data >>> push via various protocols >>> <https://victoriametrics.github.io/#how-to-import-time-series-data>, it >>> supports data deduplication >>> <https://victoriametrics.github.io/#deduplication> and it is compatible >>> with Prometheus datasource in Grafana >>> <https://victoriametrics.github.io/#grafana-setup>. >>> >>> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <[email protected]> wrote: >>> >>>> The thing is we can implement multiple prometheus instances as well, >>>> thats no issue and will probably happen anyway. The thing is, we tried >>>> using Thanos to manage multiple prometheus servers and do deduplication, >>>> but the deduplication does not work for the collectd exporters. The >>>> problem >>>> its having i think is that we need to filter out duplicate data by >>>> exported >>>> instance, but this is not possible for Thanos since it needs predefined >>>> external labels within prometheus. >>>> >>>> Regards, >>>> >>>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>>> >>>>> The way this is usually solved is by duplicating prometheus - it seems >>>>> that now you have moved the SPOF from the exporter to prometheus. >>>>> >>>>> Regards, >>>>> >>>>> On 25 Mar 06:01, robbe vaes wrote: >>>>> > Okay so, we want to have an environment using Prometheus, where we >>>>> can >>>>> > monitor our servers etc with a push method rather then pull due to >>>>> network >>>>> > security aspects. As of now, we managed to set up collectd together >>>>> with >>>>> > collectd exporter for prometheus. This way we can have the clients >>>>> or >>>>> > servers push their data to the exporter and have prometheus scrape >>>>> the data >>>>> > from the exporter to minimize the amount of pulling that has to be >>>>> done. >>>>> > The issue now is, we want to have multiple collectd exporters to >>>>> improve >>>>> > HA, but the problem now is that in prometheus, when scraping both >>>>> > exporters, it takes the data from both. It shows for example for >>>>> server A >>>>> > the cpu load but it shows it twice instead of only once because its >>>>> being >>>>> > pulled from both exporters. We want to have something like this but >>>>> it >>>>> > needs to have a form of deduplication so we dont have the same data >>>>> twice. >>>>> > >>>>> > >>>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef >>>>> [email protected]: >>>>> > >>>>> > > Can you describe more about what your network topology is exactly? >>>>> There >>>>> > > are a number of solutions for dealing with distributed monitoring. >>>>> > > >>>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >>>>> wrote: >>>>> > > >>>>> > >> Hi, >>>>> > >> >>>>> > >> I am trying to setup a monitoring environment with Prometheus, >>>>> but it has >>>>> > >> to be using a push mechanism instead of the standard pull >>>>> mechanism >>>>> > >> Prometheus uses. I was wondering what options there are to create >>>>> an >>>>> > >> environment like this. It would also have to perfom data >>>>> deduplication. The >>>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>>> itself, >>>>> > >> but rather that it scrapes a certain location for all the metrics >>>>> and that >>>>> > >> the clients push their metrics to that location automatically. >>>>> > >> >>>>> > >> Suggestions are very welcome! >>>>> > >> >>>>> > >> -- >>>>> > >> You received this message because you are subscribed to the >>>>> Google Groups >>>>> > >> "Prometheus Users" group. >>>>> > >> To unsubscribe from this group and stop receiving emails from it, >>>>> send an >>>>> > >> email to [email protected]. >>>>> > >> To view this discussion on the web visit >>>>> > >> >>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>>> >>>>> > >> < >>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> >>>>> >>>>> > >> . >>>>> > >> >>>>> > > >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to [email protected]. >>>>> > To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Julien Pivotto >>>>> @roidelapluie >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Best Regards, >>> >>> Aliaksandr Valialkin, CTO VictoriaMetrics >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com.
