Ok, i changed the log level to debug and I'm seeing that its "sending" the
notifications:
level=debug ts=2020-06-15T18:17:54.150Z caller=dispatch.go:465
component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing
alerts=[InstanceDown[48afd0d][active]]
level=debug ts=2020-06-15T18:18:09.135Z caller=dispatch.go:135
component=dispatcher msg="Received alert" alert=InstanceDown[48afd0d][active]
level=debug ts=2020-06-15T18:18:09.152Z caller=dispatch.go:465
component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing
alerts=[InstanceDown[48afd0d][active]]
level=debug ts=2020-06-15T18:18:24.154Z caller=dispatch.go:465
component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing
alerts=[InstanceDown[48afd0d][active]]
level=debug ts=2020-06-15T18:18:39.155Z caller=dispatch.go:465
component=dispatcher aggrGroup="{}:{alertname=\"InstanceDown\"}" msg=flushing
alerts=[InstanceDown[48afd0d][active]]
Also I tried checking the metrics in prometheus, and the notificiations are
beeing recorded as sent without errors
prometheus_notifications_sent_total{alertmanager=
"http://prometheus-alert-manager:9093/api/v1/alerts";, instance=
"localhost:9090", job="prometheus"} 8
prometheus_notifications_errors_total{alertmanager=
"http://prometheus-alert-manager:9093/api/v1/alerts";, instance=
"localhost:9090", job="prometheus"} 0
But in SendGrid there is still no trace of those messages... I tested the
API-KEY again using the php library and it sent the mails like normal...
El lunes, 15 de junio de 2020, 11:54:11 (UTC+2), Cristhian Rivera escribió:
>
> How can I enable the debug log? I tried looking in the configuration
> section and found no option for that:
> prometheus.io/docs/alerting/latest/configuration/
>
> Regarding SendGrid behaviour with incorrect credentials i expect for it to
> fail, but currently have no got the result. I tried the interaction with
> SendGrid using a PHP client, but that uses the API provided by SendGrid.
>
>
>> From this doc
>> <https://sendgrid.com/docs/API_Reference/SMTP_API/getting_started_smtp.html>,
>>
>> SendGrid seems to use LOGIN not PLAIN. Try leaving out `smtp_auth_identity`
>> and only specifying user and password?
>>
>
> I tried that but same result,
>
> Thanks,
>
> El lunes, 15 de junio de 2020, 11:14:47 (UTC+2), Matthias Rampke escribió:
>>
>> Can you enable debug logging for Alertmanager? That should show whether
>> it is even trying to send an email.
>>
>> How does SendGrid behave when you submit something with invalid
>> credentials? Do you expect that to explicitly fail?
>>
>> From this doc
>> <https://sendgrid.com/docs/API_Reference/SMTP_API/getting_started_smtp.html>,
>>
>> SendGrid seems to use LOGIN not PLAIN. Try leaving out `smtp_auth_identity`
>> and only specifying user and password?
>>
>> /MR
>>
>>
>> On Sat, Jun 13, 2020 at 8:14 PM Cristhian Rivera <[email protected]>
>> wrote:
>>
>>> Hi I'm trying to send the alerts generated by Prometheus trough the
>>> SendGrid SMTP.
>>>
>>> Following the SendGrid docs:
>>> https://sendgrid.com/docs/API_Reference/SMTP_API/integrating_with_the_smtp_api.html
>>>
>>> I included the corresponding settings:
>>>
>>> global:
>>> smtp_from: "AlertManager <[email protected]>"
>>> smtp_smarthost: smtp.sendgrid.net:587
>>> smtp_require_tls: true
>>> smtp_auth_username: apikey
>>> smtp_auth_identity: apikey
>>> smtp_auth_password: "<MY API KEY>"
>>>
>>> route:
>>> repeat_interval: 1m
>>> receiver: default
>>>
>>> receivers:
>>> - name: default
>>> email_configs:
>>> - to: [email protected]
>>>
>>> Then with a current alert active in prometheus I'm not receiving any
>>> emails. And in the SendGrid dashboard there is no activity either.
>>>
>>> Also in the logs of the container there is no indication that there is
>>> an issue:
>>>
>>> level=info ts=2020-06-13T20:03:47.844Z caller=main.go:231 msg="Starting
>>> Alertmanager" version="(version=0.20.0, branch=HEAD,
>>> revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)"
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.844Z caller=main.go:232
>>> build_context="(go=go1.13.5, user=root@00c3106655f8,
>>> date=20191211-14:13:14)"
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.845Z caller=cluster.go:161
>>> component=cluster msg="setting advertise address explicitly"
>>> addr=172.19.0.4 port=9094
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.846Z caller=cluster.go:623
>>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.872Z caller=coordinator.go:119
>>> component=configuration msg="Loading configuration file"
>>> file=/etc/alertmanager/alert-manager.yaml
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.872Z caller=coordinator.go:131
>>> component=configuration msg="Completed loading of configuration file"
>>> file=/etc/alertmanager/alert-manager.yaml
>>>
>>>
>>> level=info ts=2020-06-13T20:03:47.874Z caller=main.go:497 msg=Listening
>>> address=:9093
>>>
>>>
>>> level=info ts=2020-06-13T20:03:49.848Z caller=cluster.go:648
>>> component=cluster msg="gossip not settled" polls=0 before=0 now=1
>>> elapsed=2.001599783s
>>>
>>>
>>> level=info ts=2020-06-13T20:03:57.857Z caller=cluster.go:640
>>> component=cluster msg="gossip settled; proceeding" elapsed=10.010722253s
>>>
>>>
>>> I've tried with the other ports that SendGrid supports but the same
>>> result
>>>
>>> Only when the port is 25 and the smtp_require_tls: is set to false I
>>> receive a warning of
>>>
>>> level=error ts=2020-06-13T20:13:33.862Z caller=dispatch.go:301
>>> component=dispatcher msg="Notify for alerts failed" num_alerts=1
>>> err="*smtp.plainAuth auth: unencrypted connection"
>>>
>>>
>>> Any idea of what I'm missing here?
>>>
>>> Thanks!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/2d159d2b-a1fa-43fa-ade2-9e9db3be0952o%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/prometheus-users/2d159d2b-a1fa-43fa-ade2-9e9db3be0952o%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d2a4d5cc-d289-43cf-a4c4-08b5915b465ao%40googlegroups.com.
