I have few java applications that I'll deploy in my cluster. I need to know how can I detect if a instance is up or down with Prometheus.
*Alerting with AlertManager* I have a alert that check for "instanceDown" and send a alert to AlertManager-webhook. So when one instance is down, i'm receiving alerts in my application. But how can I extract the labels that are in that instance ? ex : I have a special labels in all my application that link the pod to the information that I have in the database releaseUUIDGroup=bf79b8ab-a7c1-4d27-8f3c-6e0f0a089c70 there is a way to add that information in the message that AlertManager send ? right now I configure AlertManager to send the alert to : https://webhook.site/#!/815a0b0b-f40c-4fc2-984d-e29cb9606840/b0dd701d-e972-48d4-9083-385e6a788d55/1 for an example, I kill the pod : prometheus-pushgateway and I received this message : { "receiver": "default-receiver", "status": "resolved", "alerts": [ { "status": "resolved", "labels": { "alertname": "InstanceDown", "instance": "prometheus-pushgateway.default.svc:9091", "job": "prometheus-pushgateway", "severity": "page" }, "annotations": { "description": "prometheus-pushgateway.default.svc:9091 of job prometheus-pushgateway has been down for more than 1 minute.", "summary": "Instance prometheus-pushgateway.default.svc:9091 down" }, "startsAt": "2020-06-19T17:09:53.862877577Z", "endsAt": "2020-06-22T11:23:53.862877577Z", "generatorURL": "http://prometheus-server-57d8dcc67f-qnmkj:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1";, "fingerprint": "1ed4a1dca68d64fb" } ], "groupLabels": {}, "commonLabels": { "alertname": "InstanceDown", "instance": "prometheus-pushgateway.default.svc:9091", "job": "prometheus-pushgateway", "severity": "page" }, "commonAnnotations": { "description": "prometheus-pushgateway.default.svc:9091 of job prometheus-pushgateway has been down for more than 1 minute.", "summary": "Instance prometheus-pushgateway.default.svc:9091 down" }, "externalURL": "http://localhost:9093";, "version": "4", "groupKey": "{}:{}" } -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20ec33e0-e9bf-4f2a-b366-092743dad957o%40googlegroups.com.
