On Tue, 19 May 2020 at 16:02, Andy Kruta <[email protected]> wrote: > My apologies if this has been answered already, but I've looked through > the configs for a setting that would allow me to define how many targets > can be scraped at once and came up empty. Essentially, what I've got going > on here is my prometheus is being blocked by my checkpoint firewalls (for > between 10-20 minutes) due to the number of targets that it's scraping at > once ( because of the Suspicious Activity Monitoring module.) > > My configuration: > > > - Central Prometheus server > - Multiple Data Centers > - SNMP monitored by local SNMP Exporters local to each datacenter > - Windows / Linux boxes monitored via Telegraf scraping > - Various other exporters (generally on the Prometheus server > itself unless large number of targets in remote datacenter) > > > Unfortunately, I've already talked to Checkpoint and made all of the > changes they recommend without any improvement. I've also already > increased the scrape interval (currently sitting at 4m) but the scrapes > appear to all be happening within say a minute of each other. This results > in the checkpoints blocking the activity and the targets appearing to be > down. > > My only other idea to resolve this is to increase the time in the alert > configuration to give additional time so that while the firewall is still > blocking the traffic, we don't get the alerts. This feels moronic though, > and I'm holding it back as a "just keep my mailbox empty" route. > > Has anyone come up with a clever way to work around this? >
Prometheus already spreads the scrapes across time, this is fundamentally an issue with your firewall blocking scrapes. The generally recommended architecture would be to have a Prometheus inside each datacenter, rather than trying to scrape everything across datacenters. -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLr%2BZ3VLX7dnQa09tOn9qc8wY2MhCQzRyeDKY%3D1i0jzMiQ%40mail.gmail.com.
