Regarding the other points: however you judge the differences, they are still significant departures from the rest of the Prometheus ecosystem (including compatible third-party providers). It's ok to build something incompatible, it's just IMO not ok to superficially make it sound everywhere like it's compatible with Prometheus (like in your email earlier in this thread saying "compatible with Prometheus datasource in Grafana"), while only the fine print points out that it is not. This also leaves surprises up to the users. I think it's doubly problematic to use Prometheus community forums to push for deliberately incompatible solutions in this way.
On Thu, Mar 25, 2021 at 3:15 PM Aliaksandr Valialkin <[email protected]> wrote: > > > On Thu, Mar 25, 2021 at 3:54 PM Julius Volz <[email protected]> > wrote: > >> What should be noted when bringing up VictoriaMetrics in the Prometheus >> context is that it is deliberately incompatible with Prometheus in multiple >> ways: >> >> - VM's MetricsQL behaves differently from PromQL in a multitude of ways >> and is *not* backwards-compatible with PromQL (see also >> https://promlabs.com/promql-compliance-tests). >> > > MetricsQL implements some functions differently compared to PromQL, > because users struggle with the existing behavior in PromQL. This is > explained in the beginning of > https://victoriametrics.github.io/MetricsQL.html . > > >> - The VMAgent does remote_write without the necessary pre-processing of >> series staleness handling, and it throws away datapoints like special float >> values (like NaN). At least last time I checked. >> > > vmagent <https://victoriametrics.github.io/vmagent.html> proxies all the > incoming floating-point values, including infinity values and NaN values. > > >> - The TSDB has lossy compression, so it throws away part of your data >> even further. Prometheus and Thanos store and retrieve float sample values >> exactly as sent, which depending on the use case, may matter. >> > > These concerns are addressed at > https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87 > . > > >> >> All that is to say that you can use VictoriaMetrics, but you are "on your >> own" then in terms of proper Prometheus ecosystem support and compatibility. >> >> On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <[email protected]> wrote: >> >>> It is correct that Thanos' deduplication works when using just 2 or more >>> Prometheus servers. The thing is, Thanos cannot use existing labels to do >>> deduplication. I want it to deduplicate using the label called >>> exported_instance, but i cannot predefine this label in the external labels >>> for thanos because it is not from prometheus but from the collectd >>> exporter. We are now looking into Prometheus Proxy, since we already tried >>> PushProx and it did not do what we wanted it to do. VictoriaMetrics might >>> be interesting to look into as well! >>> >>> Regards, >>> Robbe >>> >>> Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef [email protected]: >>> >>>> Probably VictoriaMetrics could help with this setup. It supports data >>>> push via various protocols >>>> <https://victoriametrics.github.io/#how-to-import-time-series-data>, >>>> it supports data deduplication >>>> <https://victoriametrics.github.io/#deduplication> and it is compatible >>>> with Prometheus datasource in Grafana >>>> <https://victoriametrics.github.io/#grafana-setup>. >>>> >>>> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <[email protected]> wrote: >>>> >>>>> The thing is we can implement multiple prometheus instances as well, >>>>> thats no issue and will probably happen anyway. The thing is, we tried >>>>> using Thanos to manage multiple prometheus servers and do deduplication, >>>>> but the deduplication does not work for the collectd exporters. The >>>>> problem >>>>> its having i think is that we need to filter out duplicate data by >>>>> exported >>>>> instance, but this is not possible for Thanos since it needs predefined >>>>> external labels within prometheus. >>>>> >>>>> Regards, >>>>> >>>>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>>>> >>>>>> The way this is usually solved is by duplicating prometheus - it >>>>>> seems >>>>>> that now you have moved the SPOF from the exporter to prometheus. >>>>>> >>>>>> Regards, >>>>>> >>>>>> On 25 Mar 06:01, robbe vaes wrote: >>>>>> > Okay so, we want to have an environment using Prometheus, where we >>>>>> can >>>>>> > monitor our servers etc with a push method rather then pull due to >>>>>> network >>>>>> > security aspects. As of now, we managed to set up collectd together >>>>>> with >>>>>> > collectd exporter for prometheus. This way we can have the clients >>>>>> or >>>>>> > servers push their data to the exporter and have prometheus scrape >>>>>> the data >>>>>> > from the exporter to minimize the amount of pulling that has to be >>>>>> done. >>>>>> > The issue now is, we want to have multiple collectd exporters to >>>>>> improve >>>>>> > HA, but the problem now is that in prometheus, when scraping both >>>>>> > exporters, it takes the data from both. It shows for example for >>>>>> server A >>>>>> > the cpu load but it shows it twice instead of only once because its >>>>>> being >>>>>> > pulled from both exporters. We want to have something like this but >>>>>> it >>>>>> > needs to have a form of deduplication so we dont have the same data >>>>>> twice. >>>>>> > >>>>>> > >>>>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef >>>>>> [email protected]: >>>>>> > >>>>>> > > Can you describe more about what your network topology is >>>>>> exactly? There >>>>>> > > are a number of solutions for dealing with distributed >>>>>> monitoring. >>>>>> > > >>>>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >>>>>> wrote: >>>>>> > > >>>>>> > >> Hi, >>>>>> > >> >>>>>> > >> I am trying to setup a monitoring environment with Prometheus, >>>>>> but it has >>>>>> > >> to be using a push mechanism instead of the standard pull >>>>>> mechanism >>>>>> > >> Prometheus uses. I was wondering what options there are to >>>>>> create an >>>>>> > >> environment like this. It would also have to perfom data >>>>>> deduplication. The >>>>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>>>> itself, >>>>>> > >> but rather that it scrapes a certain location for all the >>>>>> metrics and that >>>>>> > >> the clients push their metrics to that location automatically. >>>>>> > >> >>>>>> > >> Suggestions are very welcome! >>>>>> > >> >>>>>> > >> -- >>>>>> > >> You received this message because you are subscribed to the >>>>>> Google Groups >>>>>> > >> "Prometheus Users" group. >>>>>> > >> To unsubscribe from this group and stop receiving emails from >>>>>> it, send an >>>>>> > >> email to [email protected]. >>>>>> > >> To view this discussion on the web visit >>>>>> > >> >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>>>> > >> < >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> >>>>>> > >> . >>>>>> > >> >>>>>> > > >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> > To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Julien Pivotto >>>>>> @roidelapluie >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> >>>> Aliaksandr Valialkin, CTO VictoriaMetrics >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Best Regards, > > Aliaksandr Valialkin, CTO VictoriaMetrics > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5zhxJnakq_b%3D0E_ehXPcGddx2P%2BS_GaS2Zx55yoFME_xQ%40mail.gmail.com.
