Hi Brian, The issue was resolved. It was sth. wrong in the Alertmanager configuration. There were 2 routes. One goes to ServiceNow and the other goes to MSTeam. The alert was only fed to ServiceNow successful. My troubleshooting direction was wrong yesterday. I presumed that the Alertmanager.yaml was working on MSTeams as I was able to see alerts on ServiceNow. So I was focusing on manually triggering alerts json file that were fired on SNOW to test the MSTeam functionality. But actually ServiceNow does change the type of the data structure from kv to string even though it says it’s raw json. That’s why I ran into the parsing issue I shared in my first email. I also updated the helm chart to the latest version. After all that. the alert started working.
Thank you for your advice. Appreciate it. Zhang > On Jun 4, 2020, at 11:20 PM, Brian Candler <[email protected]> wrote: > > What about logs from alertmanager itself? Add "--log.level=debug" to > alertmanager's command line args to get maximum debugging. > > You could also try using tcpdump to see if a message is sent from > alertmanager to prometheus-msteams: > > tcpdump -i lo -nn -s0 -X tcp port 2000 > > Also you mentioned helm charts in passing, implying kubernetes is being uesd. > The destination "localhost" is only going to work if prometheus-msteams is > running in the same pod as alertmanager. If that's not the case, then you'll > have to give it the correct endpoint instead of "localhost". > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ba42d502-6e36-4ed8-8eac-b05afc9cb964o%40googlegroups.com > > <https://groups.google.com/d/msgid/prometheus-users/ba42d502-6e36-4ed8-8eac-b05afc9cb964o%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/D235AB6E-BA5A-4E14-8B9F-C2FB02A03540%40gmail.com.
