Hi, On 3/15/20 10:07 AM, Yagyansh S. Kumar wrote: > Thanks for the quick response. I appreciate your advice and I know that > instance label shouldn't contain the port number and that should be the > ideal way forward, but now my setup is huge. I'll have to change things > over all my dashboards and I am monitoring close to 3000 servers. > Eventually, I am going to change the instance label altogether but > currently I need a short term solution to remove the port from alert > annotations. Is it at all possible?
Some ideas come to mind. I would call them all workarounds, as they can become rather ugly maintenance-wise. 1) Have another meta metric which provides an instance="a:9100" to "instance_ip" mapping. This could be generated via a recording rule in combination with label_replace(), e.g.: - record: instance_to_ip_mapping expr: count by(instance, instance_ip) (label_replace(up, "instance_ip", "$1", "instance", "(.*):\d+")) https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace You could then mix in this metric into the relevant alerts. 2) You could attach such an instance_ip label to some or all of your metrics using relabeling. This will likely cause a larger overhead though. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs 3) You could run an additional query in your alerts, e.g. dashboard: "DashbordLink?orgId=1&var-node={{ printf "label_replace(up{instance='%s'}, 'instance_ip', '$1', 'instance', '(.*):\d+'" .Labels.instance | query | first | label 'instance_ip' }}&var-name={{ $labels.nodename }}&fullscreen&panelId=174" 4) You could try to offload this mapping to Grafana, i.e. make your dashboard work with just a name selected and let Grafana do the mapping. Not sure, if/how this is possible, might also depend on your dashboard and the variables. All in all, I would still follow Brian's suggestion of getting the instance= label right in the first place. It is possible -- we went through the same process at some time. ;) As always, all untested examples -- it's likely I've got some syntax/quoting/escaping wrong there. If you end up using one of those examples, please report back with the final solution, so that others can also benefit from this. :) Kind regards, Christian -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5daa9314-2efa-559f-aeb6-bc40f06bf4b2%40hoffmann-christian.info.
