Hi Danny, Did you try using a reflex for the target_match? And also adding another inhibitor rule with target_match as critical?
On Wed, Apr 15, 2020, 7:17 AM Danny de Waard <[email protected]> wrote: > I have a alrtmanger setup and prometheus setup that are working good. > I cab inhibit severity 'warning' when maintenance is on. > > But how do i also inhibit critical (and/or) info messages when maintenance > is on. > > So if my maintenance_mode == 1 all warning alerts are inhibited. But i > also want critical to be inhibited. > > How do i do that? > > prometheus rules > groups: > - name: targets > rules: > - alert: MaintenanceMode > expr: maintenance_mode == 1 > for: 1m > labels: > severity: warning > annotations: > summary: "This is a maintenance alert" > description: "Fires during maintenance mode and is routed to a > blackhole by Alertmanager" > - alert: monitor_service_down > expr: up == 0 > for: 40s > labels: > severity: critical > annotations: > summary: "A exporter service is non-operational" > description: "One of the exporters on {{ $labels.instance }} is or > was down. Check the up/down dashboard in grafana. > http://lsrv2289.linux.rabobank.nl:3000/d/RSNFpMXZz/up-down-monitor?refresh=1m > " > - alert: server_down > expr: probe_success == 0 > for: 30s > labels: > severity: critical > annotations: > summary: "Server is down (no probes are up)" > description: "Server {{ $labels.instance }} is down." > - alert: loadbalancer_down > expr: loadbalancer_stats < 1 > for: 30s > labels: > severity: critical > annotations: > summary: "A loadbalancer is down" > description: "Loadbalancer for {{ $labels.instance }} is down." > - alert: high_cpu_load15 > expr: node_load15 > 4.5 > for: 900s > labels: > severity: critical > annotations: > summary: "Server under high load (load 15m) for 15 minutes." > description: "Host is under high load, the avg load 15m is at {{ > $value}}. Reported by instance {{ $labels.instance }} of job {{ $labels.job > }}." > > > > > > > alertmanager.yml > > global: > route: > group_by: [instance,severity,job] > receiver: 'default' > routes: > - match: > alertname: 'MaintenanceMode' > receiver: 'blackhole' > - match: > severity: warning > job: PAT > receiver: 'pat' > - match: > severity: warning > job: PROD > receiver: 'prod' > - match: > severity: critical > job: PAT > receiver: 'pat-crit' > - match: > severity: critical > job: PROD > receiver: 'prod-crit' > continue: true > - match: > severity: critical > job: PROD > receiver: 'sms-waard' > - match: > severity: info > receiver: 'info' > - match: > severity: atombomb > receiver: 'webhook' > receivers: > - name: 'default' > email_configs: > - to: 'mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'pat' > email_configs: > - to: 'mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'prod' > email_configs: > - to: 'mailaddress, mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'pat-crit' > email_configs: > - to: 'mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'prod-crit' > email_configs: > - to: 'mailaddress, mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'info' > email_configs: > - to: 'mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'sms-waard' > email_configs: > - to: 'mailaddress' ##fill in your email > from: '[email protected]' > smarthost: 'localhost:25' > require_tls: false > - name: 'webhook' > webhook_configs: > - url: 'http://127.0.0.1:9000' > - name: 'blackhole' > > inhibit_rules: > - source_match: > alertname: MaintenanceMode > target_match: > severity: warning > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f403f910-5db8-4bd1-b632-0f3e0cb376b4%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/f403f910-5db8-4bd1-b632-0f3e0cb376b4%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKimyZszpu1JbAd%2BdB4Mie656D51HUJPGFn5BNRDDs9bOaZayw%40mail.gmail.com.
