On Thu, Mar 25, 2021 at 3:15 PM Aliaksandr Valialkin <[email protected]> wrote:
> > > On Thu, Mar 25, 2021 at 3:54 PM Julius Volz <[email protected]> > wrote: > >> What should be noted when bringing up VictoriaMetrics in the Prometheus >> context is that it is deliberately incompatible with Prometheus in multiple >> ways: >> >> - VM's MetricsQL behaves differently from PromQL in a multitude of ways >> and is *not* backwards-compatible with PromQL (see also >> https://promlabs.com/promql-compliance-tests). >> > > MetricsQL implements some functions differently compared to PromQL, > because users struggle with the existing behavior in PromQL. This is > explained in the beginning of > https://victoriametrics.github.io/MetricsQL.html . > > >> - The VMAgent does remote_write without the necessary pre-processing of >> series staleness handling, and it throws away datapoints like special float >> values (like NaN). At least last time I checked. >> > > vmagent <https://victoriametrics.github.io/vmagent.html> proxies all the > incoming floating-point values, including infinity values and NaN values. > Right, thanks for the correction. I remembered this incorrectly / confused it with the inability to do staleness handling, which is also done via a special NaN sample value (so staleness-related series gaps are missing in VM). > - The TSDB has lossy compression, so it throws away part of your data even >> further. Prometheus and Thanos store and retrieve float sample values >> exactly as sent, which depending on the use case, may matter. >> > > These concerns are addressed at > https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87 > . > > >> >> All that is to say that you can use VictoriaMetrics, but you are "on your >> own" then in terms of proper Prometheus ecosystem support and compatibility. >> >> On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <[email protected]> wrote: >> >>> It is correct that Thanos' deduplication works when using just 2 or more >>> Prometheus servers. The thing is, Thanos cannot use existing labels to do >>> deduplication. I want it to deduplicate using the label called >>> exported_instance, but i cannot predefine this label in the external labels >>> for thanos because it is not from prometheus but from the collectd >>> exporter. We are now looking into Prometheus Proxy, since we already tried >>> PushProx and it did not do what we wanted it to do. VictoriaMetrics might >>> be interesting to look into as well! >>> >>> Regards, >>> Robbe >>> >>> Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef [email protected]: >>> >>>> Probably VictoriaMetrics could help with this setup. It supports data >>>> push via various protocols >>>> <https://victoriametrics.github.io/#how-to-import-time-series-data>, >>>> it supports data deduplication >>>> <https://victoriametrics.github.io/#deduplication> and it is compatible >>>> with Prometheus datasource in Grafana >>>> <https://victoriametrics.github.io/#grafana-setup>. >>>> >>>> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <[email protected]> wrote: >>>> >>>>> The thing is we can implement multiple prometheus instances as well, >>>>> thats no issue and will probably happen anyway. The thing is, we tried >>>>> using Thanos to manage multiple prometheus servers and do deduplication, >>>>> but the deduplication does not work for the collectd exporters. The >>>>> problem >>>>> its having i think is that we need to filter out duplicate data by >>>>> exported >>>>> instance, but this is not possible for Thanos since it needs predefined >>>>> external labels within prometheus. >>>>> >>>>> Regards, >>>>> >>>>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>>>> >>>>>> The way this is usually solved is by duplicating prometheus - it >>>>>> seems >>>>>> that now you have moved the SPOF from the exporter to prometheus. >>>>>> >>>>>> Regards, >>>>>> >>>>>> On 25 Mar 06:01, robbe vaes wrote: >>>>>> > Okay so, we want to have an environment using Prometheus, where we >>>>>> can >>>>>> > monitor our servers etc with a push method rather then pull due to >>>>>> network >>>>>> > security aspects. As of now, we managed to set up collectd together >>>>>> with >>>>>> > collectd exporter for prometheus. This way we can have the clients >>>>>> or >>>>>> > servers push their data to the exporter and have prometheus scrape >>>>>> the data >>>>>> > from the exporter to minimize the amount of pulling that has to be >>>>>> done. >>>>>> > The issue now is, we want to have multiple collectd exporters to >>>>>> improve >>>>>> > HA, but the problem now is that in prometheus, when scraping both >>>>>> > exporters, it takes the data from both. It shows for example for >>>>>> server A >>>>>> > the cpu load but it shows it twice instead of only once because its >>>>>> being >>>>>> > pulled from both exporters. We want to have something like this but >>>>>> it >>>>>> > needs to have a form of deduplication so we dont have the same data >>>>>> twice. >>>>>> > >>>>>> > >>>>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef >>>>>> [email protected]: >>>>>> > >>>>>> > > Can you describe more about what your network topology is >>>>>> exactly? There >>>>>> > > are a number of solutions for dealing with distributed >>>>>> monitoring. >>>>>> > > >>>>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >>>>>> wrote: >>>>>> > > >>>>>> > >> Hi, >>>>>> > >> >>>>>> > >> I am trying to setup a monitoring environment with Prometheus, >>>>>> but it has >>>>>> > >> to be using a push mechanism instead of the standard pull >>>>>> mechanism >>>>>> > >> Prometheus uses. I was wondering what options there are to >>>>>> create an >>>>>> > >> environment like this. It would also have to perfom data >>>>>> deduplication. The >>>>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>>>> itself, >>>>>> > >> but rather that it scrapes a certain location for all the >>>>>> metrics and that >>>>>> > >> the clients push their metrics to that location automatically. >>>>>> > >> >>>>>> > >> Suggestions are very welcome! >>>>>> > >> >>>>>> > >> -- >>>>>> > >> You received this message because you are subscribed to the >>>>>> Google Groups >>>>>> > >> "Prometheus Users" group. >>>>>> > >> To unsubscribe from this group and stop receiving emails from >>>>>> it, send an >>>>>> > >> email to [email protected]. >>>>>> > >> To view this discussion on the web visit >>>>>> > >> >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>>>> > >> < >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> >>>>>> > >> . >>>>>> > >> >>>>>> > > >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> > To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Julien Pivotto >>>>>> @roidelapluie >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> >>>> Aliaksandr Valialkin, CTO VictoriaMetrics >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Best Regards, > > Aliaksandr Valialkin, CTO VictoriaMetrics > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5y9a4WCFrf6ViqBWa0DFQuGOcakDFW4%3DGAA__PRj_VGwA%40mail.gmail.com.
