Le dimanche 8 mars 2020 21:35:58 UTC+1, Christian Hoffmann a écrit :
>
> Hi,
>
> On 3/7/20 6:01 PM, BDT wrote:
> > I have a problem to send alerts to slack via webhook. I have a traefik
> > proxy and alertmanager which run in docker swarm.
> > So the communication between prometheus and alert is done by docker
> > network service (alermanager:9093).
> >
> > Traefik generates certficates with acme let's encrypt and working well
> > but when alertmanager push an alert to slack, i get this error: "Post
> > <redacted>: x509: certificate signed by unknown authority"
> > I don't know if something has changed with let s encrypt certificate or
> > slack webhook or alertmanager version (v0.19) but it worked before.
>
> Not sure if I understand your setup completely. Some ideas nevertheless:
>
> Could it be that you are affected by the recent Let's encrypt cert
> revocations?
>
> https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864
>
>
> If you have confirmed that this is not the case, it may help to get some
> more debugging insights:
>
> - Increase --log.level to debug
> - Capture the traffic using tcpdump and analyze it (wireshark is
> probably helpful) -- what is the actual certificate? does it look alright?
>
> Kind regards,
> Christian
>
Hi Christian Hoffmann,
Ok I did a mistake when I explain my setup. I have a traefik reverse proxy
but alertmanager sends directly the alert to slack. There is not traefik
between.
(alertmanager -> loadbalancer ovh -> internet -> slack)
I have enabled debug level:
=dispatch.go:104 component=dispatcher msg="Received alert"
alert=InstanceDown[921d528][active]
level=debug ts=2020-03-09T08:46:12.118Z caller=dispatch.go:104
component=dispatcher msg="Received alert"
alert=InstanceDown[0d9a507][active]
level=debug ts=2020-03-09T08:46:12.118Z caller=dispatch.go:104
component=dispatcher msg="Received alert"
alert=InstanceDown[ef6c116][active]
level=debug ts=2020-03-09T08:46:12.119Z caller=dispatch.go:432
component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\",
job=\"dockerd-exporter\"}" msg=flushing
alerts="[InstanceDown[921d528][active] InstanceDown[0d9a507][active]
InstanceDown[ef6c116][active]]"
level=debug ts=2020-03-09T08:46:12.143Z caller=notify.go:667
component=dispatcher msg="Notify attempt failed" attempt=1
integration=slack receiver=slack_general err="Post <redacted>: x509:
certificate signed by unknown authority"
My certificate is valid and I have checked for revocation, it's ok.
You can go to alertmanager.patrowl.cloud.patrowl.io and check. Just cancel
the http auth
Logs for tcpdump port https - Alertmanager container:
09:01:36.128928 IP (tos 0x0, ttl 64, id 43540, offset 0, flags [DF], proto
TCP (6), length 60)
****.36078 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[S], cksum 0x9b9c (incorrect -> 0x0dac), seq 1115653932, win 29200, options
[mss 1460,sackOK,TS val 1409908855 ecr 0,nop,wscale 7], length 0
09:01:36.135810 IP (tos 0x0, ttl 240, id 0, offset 0, flags [DF], proto TCP
(6), length 60)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36078: Flags
[S.], cksum 0xcf86 (correct), seq 3649616458, ack 1115653933, win 28960,
options [mss 1460,sackOK,TS val 105751778 ecr 1409908855,nop,wscale 8],
length 0
09:01:36.135860 IP (tos 0x0, ttl 64, id 43541, offset 0, flags [DF], proto
TCP (6), length 52)
****.36078 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[.], cksum 0x9b94 (incorrect -> 0x6e88), ack 1, win 229, options
[nop,nop,TS val 1409908862 ecr 105751778], length 0
09:01:36.136204 IP (tos 0x0, ttl 64, id 43542, offset 0, flags [DF], proto
TCP (6), length 267)
****.36078 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[P.], cksum 0x9c6b (incorrect -> 0x8d07), seq 1:216, ack 1, win 229,
options [nop,nop,TS val 1409908863 ecr 105751778], length 215
09:01:36.143236 IP (tos 0x0, ttl 240, id 7848, offset 0, flags [DF], proto
TCP (6), length 52)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36078: Flags
[.], cksum 0x6e1e (correct), ack 216, win 118, options [nop,nop,TS val
105751779 ecr 1409908863], length 0
09:01:36.143255 IP (tos 0x0, ttl 240, id 7849, offset 0, flags [DF], proto
TCP (6), length 2948)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36078: Flags
[.], cksum 0xa6e4 (incorrect -> 0x9386), seq 1:2897, ack 216, win 118,
options [nop,nop,TS val 105751779 ecr 1409908863], length 2896
09:01:36.143302 IP (tos 0x0, ttl 64, id 43543, offset 0, flags [DF], proto
TCP (6), length 52)
****.36078 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[.], cksum 0x9b94 (incorrect -> 0x622b), ack 2897, win 274, options
[nop,nop,TS val 1409908870 ecr 105751779], length 0
09:01:36.145411 IP (tos 0x0, ttl 240, id 7851, offset 0, flags [DF], proto
TCP (6), length 1102)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36078: Flags
[P.], cksum 0x46af (correct), seq 2897:3947, ack 216, win 118, options
[nop,nop,TS val 105751779 ecr 1409908863], length 1050
09:01:48.657107 IP (tos 0x0, ttl 64, id 64508, offset 0, flags [DF], proto
TCP (6), length 60)
****.36086 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[S], cksum 0x9b9c (incorrect -> 0x8c7c), seq 1465565832, win 29200, options
[mss 1460,sackOK,TS val 1409921383 ecr 0,nop,wscale 7], length 0
09:01:48.664267 IP (tos 0x0, ttl 240, id 0, offset 0, flags [DF], proto TCP
(6), length 60)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36086: Flags
[S.], cksum 0x1678 (correct), seq 284513267, ack 1465565833, win 28960,
options [mss 1460,sackOK,TS val 101911527 ecr 1409921383,nop,wscale 8],
length 0
09:01:48.664321 IP (tos 0x0, ttl 64, id 64509, offset 0, flags [DF], proto
TCP (6), length 52)
****.36086 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[.], cksum 0x9b94 (incorrect -> 0xb578), ack 1, win 229, options
[nop,nop,TS val 1409921391 ecr 101911527], length 0
09:01:48.664630 IP (tos 0x0, ttl 64, id 64510, offset 0, flags [DF], proto
TCP (6), length 267)
****.36086 > server-54-240-168-90.ams54.r.cloudfront.net.443: Flags
[P.], cksum 0x9c6b (incorrect -> 0x4b95), seq 1:216, ack 1, win 229,
options [nop,nop,TS val 1409921391 ecr 101911527], length 215
09:01:48.671642 IP (tos 0x0, ttl 240, id 22556, offset 0, flags [DF], proto
TCP (6), length 52)
server-54-240-168-90.ams54.r.cloudfront.net.443 > ****.36086: Flags
[.], cksum 0xb50f (correct), ack 216, win 118, options [nop,nop,TS val
101911528 ecr 1409921391], length 0
09:01:48.671874 IP (tos 0x0, ttl 240, id 22557, offset 0, flags [DF], proto
TCP
feel free to ask questions if you don't understand something
Thanks for your help
Best regards.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/c72cb152-f71e-474b-922a-a6dbf0507813%40googlegroups.com.
