Hi,
I have a working setup of Prometheus-Alert manager with slack and Teams and
alerts are working fine and reaching both Slack and Teams.
Today I added a new alert with the following query. The alert does not have
any wait time configured. All alert rules are getting evaluated every 15
seconds. Based on the query the alert became active in Prometheus UI but it
did not reach slack or Teams. It resolved before doing so. What could be
the reason? Please note, during the same time some existing alerts worked
alright.
The query
count_over_time(K_Event_Count{EvId="2417164311",EvMessage="fan alarm"}[2m])
> 0
During this time the only logs I was able to find in alert manager were
these but I am not sure the error logs are related:
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:12:35.567Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:13:29.493Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:13:31.031Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:13:32.568Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:13:34.297Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:14:29.419Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:14:31.007Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:14:33.991Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:14:36.078Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:15:29.448Z","version":"v1"}
{"caller":"api.go:781","component":"api","err":"bad_data: json: cannot
unmarshal bool into Go struct field Alert.labels of type
model.LabelValue","level":"error","msg":"API
error","ts":"2020-10-29T15:15:30.979Z","version":"v1"}
Could you please tell me what could have gone wrong?
Thanks,
Arnav
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/6327677e-a8d0-478b-a97b-30a3b5ea3211n%40googlegroups.com.
