Deduplication works fine with Thanos, you need to make sure you setup replica labels in your Prometheus external labels.
On Thu, Mar 25, 2021 at 2:20 PM robbe vaes <[email protected]> wrote: > The thing is we can implement multiple prometheus instances as well, thats > no issue and will probably happen anyway. The thing is, we tried using > Thanos to manage multiple prometheus servers and do deduplication, but the > deduplication does not work for the collectd exporters. The problem its > having i think is that we need to filter out duplicate data by exported > instance, but this is not possible for Thanos since it needs predefined > external labels within prometheus. > > Regards, > > Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: > >> The way this is usually solved is by duplicating prometheus - it seems >> that now you have moved the SPOF from the exporter to prometheus. >> >> Regards, >> >> On 25 Mar 06:01, robbe vaes wrote: >> > Okay so, we want to have an environment using Prometheus, where we can >> > monitor our servers etc with a push method rather then pull due to >> network >> > security aspects. As of now, we managed to set up collectd together >> with >> > collectd exporter for prometheus. This way we can have the clients or >> > servers push their data to the exporter and have prometheus scrape the >> data >> > from the exporter to minimize the amount of pulling that has to be >> done. >> > The issue now is, we want to have multiple collectd exporters to >> improve >> > HA, but the problem now is that in prometheus, when scraping both >> > exporters, it takes the data from both. It shows for example for server >> A >> > the cpu load but it shows it twice instead of only once because its >> being >> > pulled from both exporters. We want to have something like this but it >> > needs to have a form of deduplication so we dont have the same data >> twice. >> > >> > >> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef [email protected]: >> > >> > > Can you describe more about what your network topology is exactly? >> There >> > > are a number of solutions for dealing with distributed monitoring. >> > > >> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <[email protected]> >> wrote: >> > > >> > >> Hi, >> > >> >> > >> I am trying to setup a monitoring environment with Prometheus, but >> it has >> > >> to be using a push mechanism instead of the standard pull mechanism >> > >> Prometheus uses. I was wondering what options there are to create an >> > >> environment like this. It would also have to perfom data >> deduplication. The >> > >> main issue is that I don't want Prometheus to scrape the clients >> itself, >> > >> but rather that it scrapes a certain location for all the metrics >> and that >> > >> the clients push their metrics to that location automatically. >> > >> >> > >> Suggestions are very welcome! >> > >> >> > >> -- >> > >> You received this message because you are subscribed to the Google >> Groups >> > >> "Prometheus Users" group. >> > >> To unsubscribe from this group and stop receiving emails from it, >> send an >> > >> email to [email protected]. >> > >> To view this discussion on the web visit >> > >> >> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >> > >> < >> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >> >> > >> . >> > >> >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >> >> >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoizB2ndpa-r82iKq_uOohOz20GJp0nThotrVhG-Hvu8Q%40mail.gmail.com.
