Hi,

On 6/25/20 8:55 PM, Sébastien Dionne wrote:
> I have few java applications that I'll deploy in my cluster.  I need to
> know how can I detect if a instance is up or down with Prometheus. 
> 
> *Alerting with AlertManager*
> *
> *
> I have a alert that check for "instanceDown" and send a alert to
> AlertManager-webhook. So when one instance is down, i'm receiving alerts
> in my application.  
> 
> But how can I extract the labels that are in that instance ?
What do you mean by "in that instance"?

If the label is part of your service discovery, then it should be
attached to all series from that target. This would also imply that it
would be part of any alert by default unless you aggregate it away (e.g.
by using sum, avg or something).

If the label is only part of some info-style metric, you will have to
mix this metric into your alert.

Can you share one of the relevant alert rules if you need more specific
guidance?

Note: I don't know how many releaseUUIDGroups you have, but having UUIDs
as label values might ring some alarm bells due to the potential for
high cardinality issues. :)

Kind regards,
Christian


> 
> ex : I have a special labels in all my application that link the pod to
> the information that I have in the database
> 
> releaseUUIDGroup=bf79b8ab-a7c1-4d27-8f3c-6e0f0a089c70
> 
> 
> there is a way to add that information in the message that AlertManager
> send ?
> 
> right now I configure AlertManager to send the alert to
> : 
> https://webhook.site/#!/815a0b0b-f40c-4fc2-984d-e29cb9606840/b0dd701d-e972-48d4-9083-385e6a788d55/1
> 
> for an example, I kill the pod : prometheus-pushgateway
> 
> and I received this message : 
> 
> {
>   "receiver": "default-receiver",
>   "status": "resolved",
>   "alerts": [
>     {
>       "status": "resolved",
>       "labels": {
>         "alertname": "InstanceDown",
>         "instance": "prometheus-pushgateway.default.svc:9091",
>         "job": "prometheus-pushgateway",
>         "severity": "page"
>       },
>       "annotations": {
>         "description": "prometheus-pushgateway.default.svc:9091 of job 
> prometheus-pushgateway has been down for more than 1 minute.",
>         "summary": "Instance prometheus-pushgateway.default.svc:9091 down"
>       },
>       "startsAt": "2020-06-19T17:09:53.862877577Z",
>       "endsAt": "2020-06-22T11:23:53.862877577Z",
>       "generatorURL": 
> "http://prometheus-server-57d8dcc67f-qnmkj:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1";,
>       "fingerprint": "1ed4a1dca68d64fb"
>     }
>   ],
>   "groupLabels": {},
>   "commonLabels": {
>     "alertname": "InstanceDown",
>     "instance": "prometheus-pushgateway.default.svc:9091",
>     "job": "prometheus-pushgateway",
>     "severity": "page"
>   },
>   "commonAnnotations": {
>     "description": "prometheus-pushgateway.default.svc:9091 of job 
> prometheus-pushgateway has been down for more than 1 minute.",
>     "summary": "Instance prometheus-pushgateway.default.svc:9091 down"
>   },
>   "externalURL": "http://localhost:9093";,
>   "version": "4",
>   "groupKey": "{}:{}"
> }
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-users+unsubscr...@googlegroups.com
> <mailto:prometheus-users+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/20ec33e0-e9bf-4f2a-b366-092743dad957o%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/20ec33e0-e9bf-4f2a-b366-092743dad957o%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f46ecd4e-0986-939f-a98e-def56d0f1fe9%40hoffmann-christian.info.

Reply via email to