Hello,
I am getting started on "*Solution to Problem 1:*". I will share the PR
here once it's ready for review.
Thanks,
Ganesh
On Wednesday, 30 March 2022 at 13:38:01 UTC+5:30 Ganesh Vernekar wrote:
> Hello everyone,
>
> I have come across hosted metrics providers (including Grafana Cloud that
> I am personally looking at) facing issues in being compliant with alert
> delivery. Here are the main problems that I identify:
>
> *Problem 1:* They have some kind of embedded alert routing mechanism
> ("alertmanager") which you cannot bypass to send directly to external
> alertmanagers. Allowing external alertmanagers is not trivial and also has
> security implications. *So the alert payload that it sends might not
> exactly match what the test suite is expecting.*
>
> *Problem 2:* Related to the above, the alert routing mechanism *can add
> unwanted delays in sending alerts or reflecting the changes in the alert
> annotations.*
>
> Here is what I am proposing for the above problems:
>
> *Solution to Problem 1:* Allow cloud providers to have custom unmarshal
> logic for the alert payload embedded in the test suite. For example,
> currently we require the alert payload to be directly unmarshalled into
> []notifier.Alert here
> <https://github.com/prometheus/compliance/blob/c7c726de89973d77cb491faa1b32cfddf7dcde8a/alert_generator/server.go#L91>.
>
> But now we can allow embedding of custom unmarshalling logic such that at
> the end of the custom logic, it provides *[]notifier.Alert* as the result
> to the test suite to verify.
>
> *Solution to Problem 2:* It is in the hands of cloud providers to take
> care of it, to allow forwarding of the alerts to the end receiver with
> minimal delays and to forward all the alerts. For those who use upstream
> Prometheus Alertmanager, we need support from upstream since Prometheus
> Alertmanager does not allow forwarding of all alerts that it gets. For that
> I have requested a feature here
> <https://github.com/prometheus/alertmanager/issues/2868>.
>
> Please let me know if you have any better suggestions and/or any
> objections to the proposed solutions and/or +1 to the proposed solutions.
>
> Thanks,
> Ganesh
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/4d7eef8c-76be-406b-862a-49e3ed61873dn%40googlegroups.com.
