Le 16/11/2019 à 21:02, William Dauchy a écrit :
this patch is an attempt to permit parameters on the prometheus exporter
configuration: global, frontend, backend, listener, server.
This allows to avoid exporting metrics you don't necessary use. The
filtering can also be done on prometheus scraping configuration, but
general aim is to optimise the source of data to improve load and
scraping time. This is particularly true for huge configuration with
thousands of backends and servers.
Also note that this configuration was possible on the previous official
haproxy exporter but with even more parameters to select the needed
metrics. Here we thought it was sufficient to simply avoid a given type
of metric.
By default, everything is exported as before.
Hi William,
Thanks. Having a way to filter metrics in the Prometheus exporter was on my
todo-list :) Filtering on scopes is pretty simple and it is a good start to
solve performance issues for huge configs. It is a good idea.
However, IMHO, it is easier to use the query-string to select exported metrics.
I don't know if it is compatible with the way Prometheus is used. For instance,
based on your idea, to get only metrics about servers, the URI could be
"/metric?scope=server". And to get metrics about frontends and backends, it
could be "/metric?scope=frontend&scope=backend". Of course, it is extensible. We
can imagine to add filters on frontend/backend/server names.
Here is a quick patch based on this. What do you think about it ? If you said me
your way to select metrics is better from the Prometheus point of view, I'm ok
with that.
--
Christopher Faulet
>From 413eedf7814660218d2207e7be5e203433c399a7 Mon Sep 17 00:00:00 2001
From: Christopher Faulet <cfau...@haproxy.com>
Date: Mon, 18 Nov 2019 14:47:08 +0100
Subject: [PATCH] MINOR: contrib/prometheus-exporter: filter exported metrics
by scope
Now, the prometheus exporter parses the HTTP query-string to filter or to adapt
the exported metrics. In this first version, it is only possible select the
scopes of metrics to export. To do so, one or more parameters with "scope" as
name must be passed in the query-string, with one of those values: global,
frontend, backend, server or '*' (means all). A scope parameter with no value
means to filter out all scopes (nothing is returned). The scope parameters are
parsed in their appearance order in the query-string. So an empty scope will
reset all scopes already parsed. But it can be overridden by following scope
parameters in the query-string. By default everything is exported.
The filtering can also be done on prometheus scraping configuration, but general
aim is to optimise the source of data to improve load and scraping time. This is
particularly true for huge configuration with thousands of backends and servers.
Also note that this configuration was possible on the previous official haproxy
exporter but with even more parameters to select the needed metrics. Here we
thought it was sufficient to simply avoid a given type of metric. However, more
filters are still possible.
Thanks to William Dauchy. This patch is based on his work.
---
contrib/prometheus-exporter/README | 16 ++
.../prometheus-exporter/service-prometheus.c | 157 +++++++++++++++---
2 files changed, 148 insertions(+), 25 deletions(-)
diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README
index 915fc7f54..f540d496f 100644
--- a/contrib/prometheus-exporter/README
+++ b/contrib/prometheus-exporter/README
@@ -50,6 +50,22 @@ spend much more ressources to produce the Prometheus metrics than the CSV export
through the stats page. To give a comparison order, quick benchmarks shown that
a PROMEX dump is 5x slower and 20x more verbose than a CSV export.
+It is possible to dynamically select the metrics to export if you don't use all
+of them passing parameters in the query-string. The metrics may be filtered by
+scope. To do so, one or more parameters with "scope" as name must be passed in
+the query-string, with one of those values: global, frontend, backend, server or
+'*' (means all). A scope parameter with no value means to filter out all scopes
+(nothing is returned). The scope parameters are parsed in their appearance order
+in the query-string. So an empty scope will reset all scopes already parsed. But
+it can be overridden by following scope parameters in the query-string. By
+default everything is exported. Here are examples:
+
+ /metrics?scope=server # ==> server metrics will be exported
+ /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
+ /metrics?scope=*&scope= # ==> no metrics will be exported
+ /metrics?scope=&scope=global # ==> global metrics will be exported
+
+
Exported metrics
------------------
diff --git a/contrib/prometheus-exporter/service-prometheus.c b/contrib/prometheus-exporter/service-prometheus.c
index 508e6b1fc..09e8ad403 100644
--- a/contrib/prometheus-exporter/service-prometheus.c
+++ b/contrib/prometheus-exporter/service-prometheus.c
@@ -64,6 +64,12 @@ enum {
#define PROMEX_FL_METRIC_HDR 0x00000001
#define PROMEX_FL_INFO_METRIC 0x00000002
#define PROMEX_FL_STATS_METRIC 0x00000004
+#define PROMEX_FL_SCOPE_GLOBAL 0x00000008
+#define PROMEX_FL_SCOPE_FRONT 0x00000010
+#define PROMEX_FL_SCOPE_BACK 0x00000020
+#define PROMEX_FL_SCOPE_SERVER 0x00000040
+
+#define PROMEX_FL_SCOPE_ALL (PROMEX_FL_SCOPE_GLOBAL|PROMEX_FL_SCOPE_FRONT|PROMEX_FL_SCOPE_BACK|PROMEX_FL_SCOPE_SERVER)
/* The max length for metrics name. It is a hard limit but it should be
* enougth.
@@ -2102,72 +2108,81 @@ static int promex_dump_srv_metrics(struct appctx *appctx, struct htx *htx)
static int promex_dump_metrics(struct appctx *appctx, struct stream_interface *si, struct htx *htx)
{
int ret;
+ int scope_flags = (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_ALL);
switch (appctx->st1) {
case PROMEX_DUMPER_INIT:
appctx->ctx.stats.px = NULL;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_INFO_METRIC);
+ appctx->ctx.stats.flags = (scope_flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_INFO_METRIC);
appctx->st2 = promex_global_metrics[INF_NAME];
appctx->st1 = PROMEX_DUMPER_GLOBAL;
/* fall through */
case PROMEX_DUMPER_GLOBAL:
- ret = promex_dump_global_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_GLOBAL) {
+ ret = promex_dump_global_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (scope_flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_front_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_FRONT;
/* fall through */
case PROMEX_DUMPER_FRONT:
- ret = promex_dump_front_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_FRONT) {
+ ret = promex_dump_front_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (scope_flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_back_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_BACK;
/* fall through */
case PROMEX_DUMPER_BACK:
- ret = promex_dump_back_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_BACK) {
+ ret = promex_dump_back_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = (appctx->ctx.stats.px ? appctx->ctx.stats.px->srv : NULL);
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (scope_flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_srv_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_SRV;
/* fall through */
case PROMEX_DUMPER_SRV:
- ret = promex_dump_srv_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_SERVER) {
+ ret = promex_dump_srv_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = NULL;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = 0;
+ appctx->ctx.stats.flags = scope_flags;
appctx->st2 = 0;
appctx->st1 = PROMEX_DUMPER_DONE;
/* fall through */
@@ -2192,6 +2207,92 @@ static int promex_dump_metrics(struct appctx *appctx, struct stream_interface *s
return -1;
}
+/* Parse the query stirng of request URI to filter the metrics. It returns 1 on
+ * success and -1 on error. */
+static int promex_parse_uri(struct appctx *appctx, struct stream_interface *si)
+{
+ struct channel *req = si_oc(si);
+ struct channel *res = si_ic(si);
+ struct htx *req_htx, *res_htx;
+ struct htx_sl *sl;
+ const char *p, *end;
+ struct buffer *err;
+ int default_scopes = PROMEX_FL_SCOPE_ALL;
+ int len;
+
+ /* Get the query-string */
+ req_htx = htxbuf(&req->buf);
+ sl = http_get_stline(req_htx);
+ if (!sl)
+ goto error;
+ p = http_find_param_list(HTX_SL_REQ_UPTR(sl), HTX_SL_REQ_ULEN(sl), '?');
+ if (!p)
+ goto end;
+ end = HTX_SL_REQ_UPTR(sl) + HTX_SL_REQ_ULEN(sl);
+ len = end-p;
+
+ /* Decode the query-string */
+ chunk_reset(&trash);
+ memcpy(trash.area, p, len);
+ trash.area[len] = 0;
+ len = url_decode(trash.area);
+ if (len == -1)
+ goto error;
+ p = trash.area;
+ end = p + len;
+
+ /* Parse the query-string */
+ while (p < end) {
+ if (*p == '&')
+ ++p;
+ else if (*p == 's' && (end-p) >= 6 && !memcmp(p, "scope=", 6)) {
+ default_scopes = 0; /* at least a scope defined, unset default scopes */
+ p += 6; /* now p point on the parameter value */
+ len = 0; /* len is the value length */
+ while ((p+len) < end && *(p+len) != '&')
+ ++len;
+
+ if (len == 0)
+ appctx->ctx.stats.flags &= ~PROMEX_FL_SCOPE_ALL;
+ else if (len == 1 && *p == '*')
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_ALL;
+ else if (len == 6) {
+ if (!memcmp(p, "global", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_GLOBAL;
+ else if (!memcmp(p, "server", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_SERVER;
+ }
+ else if (len == 7 && !memcmp(p, "backend", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_BACK;
+ else if (len == 8 && !memcmp(p, "frontend", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_FRONT;
+ else
+ goto error;
+
+ p += len;
+ }
+ else {
+ /* ignore all other params for now */
+ while (p < end && *p != '&')
+ p++;
+ }
+ }
+
+ end:
+ appctx->ctx.stats.flags |= default_scopes;
+ return 1;
+
+ error:
+ err = &http_err_chunks[HTTP_ERR_400];
+ channel_erase(res);
+ res->buf.data = b_data(err);
+ memcpy(res->buf.area, b_head(err), b_data(err));
+ res_htx = htx_from_buf(&res->buf);
+ channel_add_input(res, res_htx->data);
+ appctx->st0 = PROMEX_ST_END;
+ return -1;
+}
+
/* Send HTTP headers of the response. It returns 1 on success and 0 if <htx> is
* full. */
static int promex_send_headers(struct appctx *appctx, struct stream_interface *si, struct htx *htx)
@@ -2252,6 +2353,12 @@ static void promex_appctx_handle_io(struct appctx *appctx)
switch (appctx->st0) {
case PROMEX_ST_INIT:
+ ret = promex_parse_uri(appctx, si);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto out;
+ }
appctx->st0 = PROMEX_ST_HEAD;
appctx->st1 = PROMEX_DUMPER_INIT;
/* fall through */
--
2.21.0
