Getting rate-limited errors on slack integration with an alertmanager.
There was similar issue posted a while
ago: https://github.com/prometheus/alertmanager/issues/2111
but looks like it should be fixed in 0.24.0 version.
Should I be ignoring these errors or would need to investigate further?
Environment
System information:
Docker container
Alertmanager version:
v0.24.0
Prometheus version:
N/A
Alertmanager configuration file:
N/A - contains confidential information
Prometheus configuration:
- name: slack-notifications-abc-default
slack_configs:
- api_url: https://hooks.slack.com/services/{}
channel: alerts-abc-services
send_resolved: true
Logs:
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=2
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=2
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=3
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=2
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=3
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.455Z caller=dispatch.go:354 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=2
err="slack-notifications-abc-default/slack[0]: notify retry canceled due to
unrecoverable error after 1 attempts: channel \"alerts-abc-services\":
unexpected status code 429:
{\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8d585d50-cd39-4ccb-bd04-ea9d44bf9032n%40googlegroups.com.
