What should be noted when bringing up VictoriaMetrics in the Prometheus context is that it is deliberately incompatible with Prometheus in multiple ways:
- VM's MetricsQL behaves differently from PromQL in a multitude of ways and is *not* backwards-compatible with PromQL (see also https://promlabs.com/promql-compliance-tests). - The VMAgent does remote_write without the necessary pre-processing of series staleness handling, and it throws away datapoints like special float values (like NaN). At least last time I checked. - The TSDB has lossy compression, so it throws away part of your data even further. Prometheus and Thanos store and retrieve float sample values exactly as sent, which depending on the use case, may matter. All that is to say that you can use VictoriaMetrics, but you are "on your own" then in terms of proper Prometheus ecosystem support and compatibility. On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <[email protected]> wrote: > It is correct that Thanos' deduplication works when using just 2 or more > Prometheus servers. The thing is, Thanos cannot use existing labels to do > deduplication. I want it to deduplicate using the label called > exported_instance, but i cannot predefine this label in the external labels > for thanos because it is not from prometheus but from the collectd > exporter. We are now looking into Prometheus Proxy, since we already tried > PushProx and it did not do what we wanted it to do. VictoriaMetrics might > be interesting to look into as well! > > Regards, > Robbe > > Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef [email protected]: > >> Probably VictoriaMetrics could help with this setup. It supports data >> push via various protocols >> <https://victoriametrics.github.io/#how-to-import-time-series-data>, it >> supports data deduplication >> <https://victoriametrics.github.io/#deduplication> and it is compatible >> with Prometheus datasource in Grafana >> <https://victoriametrics.github.io/#grafana-setup>. >> >> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <[email protected]> wrote: >> >>> The thing is we can implement multiple prometheus instances as well, >>> thats no issue and will probably happen anyway. The thing is, we tried >>> using Thanos to manage multiple prometheus servers and do deduplication, >>> but the deduplication does not work for the collectd exporters. The problem >>> its having i think is that we need to filter out duplicate data by exported >>> instance, but this is not possible for Thanos since it needs predefined >>> external labels within prometheus. >>> >>> Regards, >>> >>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>> >>>> The way this is usually solved is by duplicating prometheus - it seems >>>> that now you have moved the SPOF from the exporter to prometheus. >>>> >>>> Regards, >>>> >>>> On 25 Mar 06:01, robbe vaes wrote: >>>> > Okay so, we want to have an environment using Prometheus, where we >>>> can >>>> > monitor our servers etc with a push method rather then pull due to >>>> network >>>> > security aspects. As of now, we managed to set up collectd together >>>> with >>>> > collectd exporter for prometheus. This way we can have the clients or >>>> > servers push their data to the exporter and have prometheus scrape >>>> the data >>>> > from the exporter to minimize the amount of pulling that has to be >>>> done. >>>> > The issue now is, we want to have multiple collectd exporters to >>>> improve >>>> > HA, but the problem now is that in prometheus, when scraping both >>>> > exporters, it takes the data from both. It shows for example for >>>> server A >>>> > the cpu load but it shows it twice instead of only once because its >>>> being >>>> > pulled from both exporters. We want to have something like this but >>>> it >>>> > needs to have a form of deduplication so we dont have the same data >>>> twice. >>>> > >>>> > >>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef [email protected]: >>>> >>>> > >>>> > > Can you describe more about what your network topology is exactly? >>>> There >>>> > > are a number of solutions for dealing with distributed monitoring. >>>> > > >>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >>>> wrote: >>>> > > >>>> > >> Hi, >>>> > >> >>>> > >> I am trying to setup a monitoring environment with Prometheus, but >>>> it has >>>> > >> to be using a push mechanism instead of the standard pull >>>> mechanism >>>> > >> Prometheus uses. I was wondering what options there are to create >>>> an >>>> > >> environment like this. It would also have to perfom data >>>> deduplication. The >>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>> itself, >>>> > >> but rather that it scrapes a certain location for all the metrics >>>> and that >>>> > >> the clients push their metrics to that location automatically. >>>> > >> >>>> > >> Suggestions are very welcome! >>>> > >> >>>> > >> -- >>>> > >> You received this message because you are subscribed to the Google >>>> Groups >>>> > >> "Prometheus Users" group. >>>> > >> To unsubscribe from this group and stop receiving emails from it, >>>> send an >>>> > >> email to [email protected]. >>>> > >> To view this discussion on the web visit >>>> > >> >>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>> > >> < >>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> >>>> > >> . >>>> > >> >>>> > > >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> > To unsubscribe from this group and stop receiving emails from it, >>>> send an email to [email protected]. >>>> > To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>> >>>> >>>> >>>> -- >>>> Julien Pivotto >>>> @roidelapluie >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Best Regards, >> >> Aliaksandr Valialkin, CTO VictoriaMetrics >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com.
