Hello Christopher, Thank you for your quick answer.
On Mon, Nov 18, 2019 at 04:28:37PM +0100, Christopher Faulet wrote: > Thanks. Having a way to filter metrics in the Prometheus exporter was on my > todo-list :) Filtering on scopes is pretty simple and it is a good start to > solve performance issues for huge configs. It is a good idea. > However, IMHO, it is easier to use the query-string to select exported > metrics. I don't know if it is compatible with the way Prometheus is used. > For instance, based on your idea, to get only metrics about servers, the URI > could be "/metric?scope=server". And to get metrics about frontends and > backends, it could be "/metric?scope=frontend&scope=backend". Of course, it > is extensible. We can imagine to add filters on frontend/backend/server > names. > Here is a quick patch based on this. What do you think about it ? If you > said me your way to select metrics is better from the Prometheus point of > view, I'm ok with that. In fact I even did not thought about it because my state of mind was to filter at startup like we were doing before. I like you propostion as it is way more extensible than mine. I was not even thinking it would be possible to access the URL, I now have a lot of ideas of the future ;) example config would look like: - job_name: 'lb-builtin-metrics' metrics_path: '/metrics?scope=backend' consul_sd_configs: - server: localhost:8500 services: - lb-builtin-exporter relabel_configs: - source_labels: ['__meta_consul_dc'] target_label: 'dc' - source_labels: ['__address__', '__meta_consul_service_id'] target_label: 'instance' If you have other things on your todo list regarding that feel free to share so we might share the work around it. Also have you seen the message from Pierre? They are a few things we would like to discuss whether it would be possible to aggregate a few things on backend side. https://firstname.lastname@example.org/msg35369.html (we somehow forgot to put you in copy though) > From 413eedf7814660218d2207e7be5e203433c399a7 Mon Sep 17 00:00:00 2001 > From: Christopher Faulet <cfau...@haproxy.com> > Date: Mon, 18 Nov 2019 14:47:08 +0100 > Subject: [PATCH] MINOR: contrib/prometheus-exporter: filter exported metrics > by scope > > Now, the prometheus exporter parses the HTTP query-string to filter or to > adapt > the exported metrics. In this first version, it is only possible select the > scopes of metrics to export. To do so, one or more parameters with "scope" as > name must be passed in the query-string, with one of those values: global, > frontend, backend, server or '*' (means all). A scope parameter with no value > means to filter out all scopes (nothing is returned). The scope parameters are > parsed in their appearance order in the query-string. So an empty scope will > reset all scopes already parsed. But it can be overridden by following scope > parameters in the query-string. By default everything is exported. > > The filtering can also be done on prometheus scraping configuration, but > general > aim is to optimise the source of data to improve load and scraping time. This > is > particularly true for huge configuration with thousands of backends and > servers. > Also note that this configuration was possible on the previous official > haproxy > exporter but with even more parameters to select the needed metrics. Here we > thought it was sufficient to simply avoid a given type of metric. However, > more > filters are still possible. > > Thanks to William Dauchy. This patch is based on his work. > --- > contrib/prometheus-exporter/README | 16 ++ > .../prometheus-exporter/service-prometheus.c | 157 +++++++++++++++--- > 2 files changed, 148 insertions(+), 25 deletions(-) Thank you for this patch. Do you think it could be acceptable to mark it as a possible backport for v2.0.x? It's quite critical on our side as we are dealing with ~130MB on some cases. If not we will do the backport on our wide while waiting for the v2.1. I will push a test based on your patch tomorrow on our side. -- William