Hello everyone,
I have come across hosted metrics providers (including Grafana Cloud that I
am personally looking at) facing issues in being compliant with alert
delivery. Here are the main problems that I identify:
*Problem 1:* They have some kind of embedded alert routing mechanism
("alertmanager") which you cannot bypass to send directly to external
alertmanagers. Allowing external alertmanagers is not trivial and also has
security implications. *So the alert payload that it sends might not
exactly match what the test suite is expecting.*
*Problem 2:* Related to the above, the alert routing mechanism *can add
unwanted delays in sending alerts or reflecting the changes in the alert
annotations.*
Here is what I am proposing for the above problems:
*Solution to Problem 1:* Allow cloud providers to have custom unmarshal
logic for the alert payload embedded in the test suite. For example,
currently we require the alert payload to be directly unmarshalled into
[]notifier.Alert here
<https://github.com/prometheus/compliance/blob/c7c726de89973d77cb491faa1b32cfddf7dcde8a/alert_generator/server.go#L91>.
But now we can allow embedding of custom unmarshalling logic such that at
the end of the custom logic, it provides *[]notifier.Alert* as the result
to the test suite to verify.
*Solution to Problem 2:* It is in the hands of cloud providers to take care
of it, to allow forwarding of the alerts to the end receiver with minimal
delays and to forward all the alerts. For those who use upstream Prometheus
Alertmanager, we need support from upstream since Prometheus Alertmanager
does not allow forwarding of all alerts that it gets. For that I have
requested a feature here
<https://github.com/prometheus/alertmanager/issues/2868>.
Please let me know if you have any better suggestions and/or any objections
to the proposed solutions and/or +1 to the proposed solutions.
Thanks,
Ganesh
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/CAERru7Oft05OYPYDW%2Bm5ozMFvmHTjVGcnK0q%3Dh5wxS4Az7HFog%40mail.gmail.com.
