Thanks a lot, Christian. Will try them out and report back. Also, according to you will the Step 3 add any significant overhead? I mean will it cause any kind of slowness?
On Sunday, March 15, 2020 at 5:26:47 PM UTC+5:30, Christian Hoffmann wrote: > > Hi, > > On 3/15/20 10:07 AM, Yagyansh S. Kumar wrote: > > Thanks for the quick response. I appreciate your advice and I know that > > instance label shouldn't contain the port number and that should be the > > ideal way forward, but now my setup is huge. I'll have to change things > > over all my dashboards and I am monitoring close to 3000 servers. > > Eventually, I am going to change the instance label altogether but > > currently I need a short term solution to remove the port from alert > > annotations. Is it at all possible? > > Some ideas come to mind. I would call them all workarounds, as they can > become rather ugly maintenance-wise. > > 1) Have another meta metric which provides an instance="a:9100" to > "instance_ip" mapping. This could be generated via a recording rule in > combination with label_replace(), e.g.: > > - record: instance_to_ip_mapping > expr: count by(instance, instance_ip) (label_replace(up, > "instance_ip", "$1", "instance", "(.*):\d+")) > > > https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace > > > You could then mix in this metric into the relevant alerts. > > > 2) You could attach such an instance_ip label to some or all of your > metrics using relabeling. This will likely cause a larger overhead though. > > > https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs > > > > 3) You could run an additional query in your alerts, e.g. > > dashboard: "DashbordLink?orgId=1&var-node={{ printf > "label_replace(up{instance='%s'}, 'instance_ip', '$1', 'instance', > '(.*):\d+'" .Labels.instance | query | first | label 'instance_ip' > }}&var-name={{ $labels.nodename }}&fullscreen&panelId=174" > > > 4) You could try to offload this mapping to Grafana, i.e. make your > dashboard work with just a name selected and let Grafana do the mapping. > Not sure, if/how this is possible, might also depend on your dashboard > and the variables. > > > All in all, I would still follow Brian's suggestion of getting the > instance= label right in the first place. It is possible -- we went > through the same process at some time. ;) > > As always, all untested examples -- it's likely I've got some > syntax/quoting/escaping wrong there. If you end up using one of those > examples, please report back with the final solution, so that others can > also benefit from this. :) > > > Kind regards, > Christian > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5aa5eb4a-00c7-49e6-a094-eb639a581b18%40googlegroups.com.
