HI Brian & Team,
Alertmanager federation issue.
MGMT zone which ever the alerts we trigger it triggering the alerts as
expected i.e repeat interval every 24 hours but the alerts which are coming
from appzone it is not working as expected.
Could you please help any way i can add configuratoin from app zone to
trigger the alerts same like mgmt zone.
Appreciate your reply on this.
Below is my configuration files:
Alertmanager configuration MGMT zone:
global:
resolve_timeout: 30s
route:
receiver: 'email_receiver_dummy'
group_wait: 360h
group_interval: 480h
repeat_interval: 480h
group_by: ['alertname']
routes:
- receiver: 'email_receiver'
match_re:
alertname:
mssql-cluster-down|NodedownAlert|ServiceDown|Service-Down|mssql-secondary-node-down|mssql-primary-node-down|Mongoservicedegraded|Mongoservicedown|gfsdown|gfs-master1-down|gfs-master2-down|gfs-master3-down|Servicedegrade|Service-Degrade
group_wait: 30s
group_interval: 5m
repeat_interval: 24h
group_by: ['alertname']
continue: true
- receiver: 'webhook_receiver'
match_re:
alertname:
mssql-cluster-down|NodedownAlert|ServiceDown|Service-Down|mssql-secondary-node-down|mssql-primary-node-down|Mongoservicedegraded|Mongoservicedown|gfsdown|gfs-master1-down|gfs-master2-down|gfs-master3-down|Servicedegrade|Service-Degrade
group_wait: 30s
group_interval: 5m
repeat_interval: 24h
group_by: ['alertname']
receivers:
- name: email_receiver_dummy
email_configs:
- smarthost: xxxxxx:25
auth_username: xxxxx
auth_password: yyyyyy
require_tls: true
to: [email protected]
from: [email protected]
html: '{{ template "__email_text_list" . }}'
headers:
subject: '{{ template "_email_subject" . }}'
- name: email_receiver
email_configs:
- smarthost: xxxxxxxxxxxxxx:25
auth_username: xxxxxxxxxxxxx
auth_password: xxxxxxxxxxxxxxxxxx
require_tls: true
to: xxxxxxxxxxxxxxxxxxxx@org
# to: xxxxxxxxxxxxxxxxxxxx@org
from: xxxxxxxxxxxxxxxxxxxx@org
html: '{{ template "__email_text_list" . }}'
headers:
subject: '{{ template "_email_subject" . }}'
- name: webhook_receiver
webhook_configs:
- send_resolved: false
url:
http://prometheus-webhook.ns-prometheus-mgmt:8080/v1/webhook?group=operation
templates:
- '/etc/alertmanager/config/notification.tmpl'
App zone prometheus configuration where we have configured federation which
sends the alerts app to mgmt tier in our environment.
prometheus:
service:
nodePort: 31540
type: NodePort
sessionAffinity: ClientIP
prometheusSpec:
image:
repository: quay.io/prometheus/prometheus
tag: v2.15.2
externalLabels:
cluster: 'qa-app'
externalUrl:
"http://prometheus-operator-prometheus.ns-prometheus-qa:9090/";
secrets:
- etcd-client-cert
replicas: 3
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: nfs-client
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 150Gi
additionalAlertManagerConfigs:
- static_configs:
- targets: [ 'mgmtzoneip:32541' ]
scheme: https
tls_config:
insecure_skip_verify: true
MGMT zone which ever the alerts we trigger it triggering the alerts as
expected i.e repeat interval every 24 hours but the alerts which are coming
from appzone it is not working as expected.
Could you please help any way i can add configuratoin from app zone to
trigger the alerts same like mgmt zone.
Appreciate your reply on this.
On Tuesday, September 15, 2020 at 2:06:16 PM UTC+8 Sandeep Rao Kokkirala
wrote:
> Thanks Brian . it's working
>
> On Monday, September 14, 2020 at 7:24:31 PM UTC+8 [email protected] wrote:
>
>> On Monday, 14 September 2020 11:38:43 UTC+1, Sandeep Rao Kokkirala wrote:
>>>
>>> consider 1st service is down ..our alertmanager is triggers the alert
>>> ...when 2nd service is down after 10 minutes ..that time it triggering both
>>> 1st service and 2nd service alerts ...1st service alert already triggered
>>> so we don't want 1st service alert to repeat.
>>>
>>
>> It's sending a new *grouped* alert which contains both the original alert
>> and the 2nd alert, grouped together. You asked for this with group_by:
>> ['alertname'] which means that two alerts with the same 'alertname'
>> label should be considered part of the same group.
>>
>> You can disable grouping entirely with: group_by: ['...'] (yes,
>> that's literally three dots in there)
>>
>> https://prometheus.io/docs/alerting/latest/configuration/#route
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/eef6a279-ddb2-4294-b7f0-71b05adde05cn%40googlegroups.com.
