Thanks for the template snip -
Here's what I finally ended up going with -
*alertmanager.yml*
slack_configs:
- api_url:
channel: '#xxxxxxxxxxxxxxxxxxxx'
color: '{{ template "SLACK_MSG_COLOR" . }}'
send_resolved: true
title: '{{ template "SLACK_MSG_TITLE" . }}'
text: '{{ template "SLACK_MSG_TEXT" . }}'
pagerduty_configs:
- routing_key: '{{ template "Global_PD_Service_Key" . }}'
description: '{{ template "PAGERDUTY_DESCRIPTION" . }}'
severity: '
{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toLower }}{{ else
}}critical{{ end }}
'
links:
- text: 'Prometheus'
href: '{{ (index .Alerts 0).GeneratorURL }}'
- text: 'Search Runbooks'
href: '{{ template "RUNBOOK_SEARCH" . }}'
*and the .tmpl*
# Pagerduty description
{{ define "PAGERDUTY_DESCRIPTION" }}
{{if .CommonAnnotations.summary }}{{ .CommonAnnotations.summary }}
{{ else }}{{ .GroupLabels.alertname }}
{{ if .CommonLabels.instance }}{{ .CommonLabels.instance }}
{{ end }}
{{ end }}
{{ end }}
################
# Runbook
################
# Runbook Search
{{ define "RUNBOOK_SEARCH" }}https://dsmith73.github.io/101-docs/search/?q={{
.CommonLabels.alertname }}{{ end }}
################
# Slack
################
# Slack Color
{{ define "SLACK_MSG_COLOR" }}{{ if eq .Status "firing" }}{{ if eq .
CommonLabels.severity "critical" }}danger{{ else if eq .CommonLabels.severity
"error" }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{
else }}#439FE0{{ end }}{{ else}}good{{ end }}{{ end }}
# Slack Text
{{define "SLACK_MSG_TEXT" }}
<!here> - {{ .CommonAnnotations.description }}
`View:` :chart_with_upwards_trend:*<{{ (index .Alerts 0).GeneratorURL
}}|Prometheus>* or :notebook:*<{{ template "RUNBOOK_SEARCH" . }}|Runbook>*
*Details:*
{{ range .CommonLabels.SortedPairs }}• *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}
#Slack Summary
{{ define "SLACK_TITLE_SUMMARY" -}}
{{- if .CommonAnnotations.summary -}}
{{- .CommonAnnotations.summary -}}
{{- else -}}
{{- with index .Alerts 0 -}}
{{- .Annotations.summary -}}
{{- end -}}
{{- end -}}
{{- end -}}
# Slack Title
{{ define "SLACK_MSG_TITLE" }}
{{ if eq .Status "resolved" }}
{{- .Status | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
{{ else if eq .Status "firing" }}
{{ .CommonLabels.severity | toUpper }} : {{ template
"SLACK_TITLE_SUMMARY" . }}
{{ end }}
{{ end }}
It's all a wire-frame, but I like using the product label and alert-name to
execute a query style search for runbooks. I've been tossing around the
idea of moving our runbooks to git, and the docksy-jeckyll website brings
it all together IMO.
cheers!
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d6fa8b17-5335-4103-8136-5328de663e8f%40googlegroups.com.
