Although I wish it was, unfortunately, it's not an option. The good news is that I don't have to deal with the checkpoints much longer. The bad news is that until I get rid of them, I have to silence the noise.
On Tuesday, May 19, 2020 at 10:53:38 AM UTC-5, Brian Brazil wrote: > > On Tue, 19 May 2020 at 16:02, Andy Kruta <[email protected] <javascript:>> > wrote: > >> My apologies if this has been answered already, but I've looked through >> the configs for a setting that would allow me to define how many targets >> can be scraped at once and came up empty. Essentially, what I've got going >> on here is my prometheus is being blocked by my checkpoint firewalls (for >> between 10-20 minutes) due to the number of targets that it's scraping at >> once ( because of the Suspicious Activity Monitoring module.) >> >> My configuration: >> >> >> - Central Prometheus server >> - Multiple Data Centers >> - SNMP monitored by local SNMP Exporters local to each datacenter >> - Windows / Linux boxes monitored via Telegraf scraping >> - Various other exporters (generally on the Prometheus server >> itself unless large number of targets in remote datacenter) >> >> >> Unfortunately, I've already talked to Checkpoint and made all of the >> changes they recommend without any improvement. I've also already >> increased the scrape interval (currently sitting at 4m) but the scrapes >> appear to all be happening within say a minute of each other. This results >> in the checkpoints blocking the activity and the targets appearing to be >> down. >> >> My only other idea to resolve this is to increase the time in the alert >> configuration to give additional time so that while the firewall is still >> blocking the traffic, we don't get the alerts. This feels moronic though, >> and I'm holding it back as a "just keep my mailbox empty" route. >> >> Has anyone come up with a clever way to work around this? >> > > Prometheus already spreads the scrapes across time, this is fundamentally > an issue with your firewall blocking scrapes. The generally recommended > architecture would be to have a Prometheus inside each datacenter, rather > than trying to scrape everything across datacenters. > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fa25acb2-43ec-4deb-8b53-b2c841c1b83d%40googlegroups.com.
