Please remove me from this email list On Sat, Apr 6, 2024 at 12:15 PM Baburao Ganpatrao < baburaoganpatrao...@gmail.com> wrote:
> Actually, I initially thought of this as support, then now I know its > discussion forum. And I am seeking for help to resolve this strange > behavior of prometheus alertmanager. > > On Saturday 6 April 2024 at 22:39:25 UTC+5:30 Ben Kochie wrote: > >> This is not a support team, this is a users discussion list. >> >> If you require professional support, please see our support page: >> https://prometheus.io/support-training/ >> >> On Sat, Apr 6, 2024 at 7:07 PM Baburao Ganpatrao <baburaoga...@gmail.com> >> wrote: >> >>> Dear Support Team, >>> >>> I am currently experiencing some discrepancies and unexpected behavior >>> with the configuration of Prometheus AlertManager. Despite configuring >>> alerts to fire every 4 hours, I am encountering issues with the actual >>> number of alerts sent and received by AlertManager. There seems to be a >>> significant disparity between the expected and observed counts. >>> >>> Furthermore, even when no conditions are satisfied for triggering >>> alerts, Prometheus continues to send alerts every 4 to 5 hours, and >>> AlertManager subsequently forwards the payload to the configured webhook. I >>> have noticed this behavior persisting even after a prolonged period of no >>> data changes – in one instance, alerts were still being received 20 hours >>> after data had ceased to change. >>> >>> Below are the relevant configuration files: prometheus.yml: Scrape >>> Interval: 2m >>> >>> *rules.yml:* groups: >>> - name: recording-rule >>> interval: 1h >>> rules: >>> - record: myRecord >>> expr: [expression calculating ratio by dividing two metrics, >>> greater than a specified value] >>> - name: alerting-rule >>> interval: 4h >>> rules: >>> - alert: myAlert >>> expr: max_over_time(myRecord[4h]) >>> labels: >>> severity: warning >>> annotations: >>> summary: "Summary" >>> * alertmanager.yml:* group_by: ['alertname'] >>> group_wait: 30s >>> group_interval: 1h >>> repeat_interval: 4h >>> I would appreciate your assistance in resolving the following issues: >>> >>> 1. Identifying and rectifying the discrepancies in the number of >>> alerts between Prometheus and AlertManager. >>> 2. Investigating and addressing the persistence of alerts being >>> fired by Prometheus and forwarded by AlertManager, even in the absence of >>> new data or changes. >>> 3. Resolving the issue of receiving alerts at regular intervals, >>> despite a prolonged period of no new data. >>> >>> *It's worth noting that a substantial amount of data is being processed >>> for alerts, more than 25,000 unique metrics.* >>> >>> Your prompt attention to these matters would be greatly appreciated. >>> >>> Thank you for your assistance. >>> >> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMazdJ8DdA5iGGBXDux-_C1Ww%2BSRJ8mpt_o_d8yJCaW3z74akQ%40mail.gmail.com.
