Hi, I am getting rate_limited errors on slack integration with the alertmanager.
There was a similar issue https://github.com/prometheus/alertmanager/issues/2111 reported in the past which should be fixed on 0.24.0 version I am using. Should I be ignoring these errors as alerts are still pouring in, or do I need to investigate this further? What is the best way to troubleshoot to see if alerts are missed sending to the slack channel? Environment System information: Docker container Alertmanager version: v0.24.0 Prometheus version: N/A Alertmanager configuration file: N/A - contains confidential information Prometheus configuration: - name: slack-notifications-abc-default slack_configs: - api_url: https://hooks.slack.com/services/ channel: alerts-abc-services send_resolved: true Logs: ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" ts=2022-09-07T18:42:28.455Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" Thanks, Seema -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ef4d3eb5-cab1-4ed7-bc53-4e6da589f469n%40googlegroups.com.
