Le 19/11/2019 à 14:51, Christopher Faulet a écrit :
Regarding the problem of servers in maintenance, since we parse the
query-string, it is possible to add more filters. I may add a parameter to
filter out servers in maintenance. For instance, by passing "no-maint" in the
query-string, all servers in maintenance could be ignored. This way, it would
still be possible to have servers metrics.
William,
Here is updated patches with the support for "scope" and "no-maint" parameters.
If this solution is good enough for you (and if it works :), I will push it.
Thanks,
--
Christopher Faulet
>From 205c4775ccec914a2a501fe9499a77b96285315b Mon Sep 17 00:00:00 2001
From: Christopher Faulet <cfau...@haproxy.com>
Date: Mon, 18 Nov 2019 14:47:08 +0100
Subject: [PATCH 1/2] MINOR: contrib/prometheus-exporter: filter exported
metrics by scope
Now, the prometheus exporter parses the HTTP query-string to filter or to adapt
the exported metrics. In this first version, it is only possible select the
scopes of metrics to export. To do so, one or more parameters with "scope" as
name must be passed in the query-string, with one of those values: global,
frontend, backend, server or '*' (means all). A scope parameter with no value
means to filter out all scopes (nothing is returned). The scope parameters are
parsed in their appearance order in the query-string. So an empty scope will
reset all scopes already parsed. But it can be overridden by following scope
parameters in the query-string. By default everything is exported.
The filtering can also be done on prometheus scraping configuration, but general
aim is to optimise the source of data to improve load and scraping time. This is
particularly true for huge configuration with thousands of backends and servers.
Also note that this configuration was possible on the previous official haproxy
exporter but with even more parameters to select the needed metrics. Here we
thought it was sufficient to simply avoid a given type of metric. However, more
filters are still possible.
Thanks to William Dauchy. This patch is based on his work.
---
contrib/prometheus-exporter/README | 24 +++
.../prometheus-exporter/service-prometheus.c | 157 +++++++++++++++---
2 files changed, 156 insertions(+), 25 deletions(-)
diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README
index 915fc7f54..84ae8e27e 100644
--- a/contrib/prometheus-exporter/README
+++ b/contrib/prometheus-exporter/README
@@ -50,6 +50,30 @@ spend much more ressources to produce the Prometheus metrics than the CSV export
through the stats page. To give a comparison order, quick benchmarks shown that
a PROMEX dump is 5x slower and 20x more verbose than a CSV export.
+
+metrics filtering
+-------------------
+
+It is possible to dynamically select the metrics to export if you don't use all
+of them passing parameters in the query-string.
+
+* Filtering on scopes
+
+The metrics may be filtered by scopes. Multiple parameters with "scope" as name
+may be passed in the query-string to filter exported metrics, with one of those
+values: global, frontend, backend, server or '*' (means all). A scope parameter
+with no value means to filter out all scopes (nothing is returned). The scope
+parameters are parsed in their appearance order in the query-string. So an empty
+scope will reset all scopes already parsed. But it can be overridden by
+following scope parameters in the query-string. By default everything is
+exported. Here are examples:
+
+ /metrics?scope=server # ==> server metrics will be exported
+ /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
+ /metrics?scope=*&scope= # ==> no metrics will be exported
+ /metrics?scope=&scope=global # ==> global metrics will be exported
+
+
Exported metrics
------------------
diff --git a/contrib/prometheus-exporter/service-prometheus.c b/contrib/prometheus-exporter/service-prometheus.c
index 508e6b1fc..c70daa905 100644
--- a/contrib/prometheus-exporter/service-prometheus.c
+++ b/contrib/prometheus-exporter/service-prometheus.c
@@ -64,6 +64,12 @@ enum {
#define PROMEX_FL_METRIC_HDR 0x00000001
#define PROMEX_FL_INFO_METRIC 0x00000002
#define PROMEX_FL_STATS_METRIC 0x00000004
+#define PROMEX_FL_SCOPE_GLOBAL 0x00000008
+#define PROMEX_FL_SCOPE_FRONT 0x00000010
+#define PROMEX_FL_SCOPE_BACK 0x00000020
+#define PROMEX_FL_SCOPE_SERVER 0x00000040
+
+#define PROMEX_FL_SCOPE_ALL (PROMEX_FL_SCOPE_GLOBAL|PROMEX_FL_SCOPE_FRONT|PROMEX_FL_SCOPE_BACK|PROMEX_FL_SCOPE_SERVER)
/* The max length for metrics name. It is a hard limit but it should be
* enougth.
@@ -2102,72 +2108,81 @@ static int promex_dump_srv_metrics(struct appctx *appctx, struct htx *htx)
static int promex_dump_metrics(struct appctx *appctx, struct stream_interface *si, struct htx *htx)
{
int ret;
+ int flags = appctx->ctx.stats.flags;
switch (appctx->st1) {
case PROMEX_DUMPER_INIT:
appctx->ctx.stats.px = NULL;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_INFO_METRIC);
+ appctx->ctx.stats.flags = (flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_INFO_METRIC);
appctx->st2 = promex_global_metrics[INF_NAME];
appctx->st1 = PROMEX_DUMPER_GLOBAL;
/* fall through */
case PROMEX_DUMPER_GLOBAL:
- ret = promex_dump_global_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_GLOBAL) {
+ ret = promex_dump_global_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_front_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_FRONT;
/* fall through */
case PROMEX_DUMPER_FRONT:
- ret = promex_dump_front_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_FRONT) {
+ ret = promex_dump_front_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_back_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_BACK;
/* fall through */
case PROMEX_DUMPER_BACK:
- ret = promex_dump_back_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_BACK) {
+ ret = promex_dump_back_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = proxies_list;
appctx->ctx.stats.sv = (appctx->ctx.stats.px ? appctx->ctx.stats.px->srv : NULL);
- appctx->ctx.stats.flags = (PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
+ appctx->ctx.stats.flags = (flags|PROMEX_FL_METRIC_HDR|PROMEX_FL_STATS_METRIC);
appctx->st2 = promex_srv_metrics[ST_F_PXNAME];
appctx->st1 = PROMEX_DUMPER_SRV;
/* fall through */
case PROMEX_DUMPER_SRV:
- ret = promex_dump_srv_metrics(appctx, htx);
- if (ret <= 0) {
- if (ret == -1)
- goto error;
- goto full;
+ if (appctx->ctx.stats.flags & PROMEX_FL_SCOPE_SERVER) {
+ ret = promex_dump_srv_metrics(appctx, htx);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto full;
+ }
}
appctx->ctx.stats.px = NULL;
appctx->ctx.stats.sv = NULL;
- appctx->ctx.stats.flags = 0;
+ appctx->ctx.stats.flags = flags;
appctx->st2 = 0;
appctx->st1 = PROMEX_DUMPER_DONE;
/* fall through */
@@ -2192,6 +2207,92 @@ static int promex_dump_metrics(struct appctx *appctx, struct stream_interface *s
return -1;
}
+/* Parse the query stirng of request URI to filter the metrics. It returns 1 on
+ * success and -1 on error. */
+static int promex_parse_uri(struct appctx *appctx, struct stream_interface *si)
+{
+ struct channel *req = si_oc(si);
+ struct channel *res = si_ic(si);
+ struct htx *req_htx, *res_htx;
+ struct htx_sl *sl;
+ const char *p, *end;
+ struct buffer *err;
+ int default_scopes = PROMEX_FL_SCOPE_ALL;
+ int len;
+
+ /* Get the query-string */
+ req_htx = htxbuf(&req->buf);
+ sl = http_get_stline(req_htx);
+ if (!sl)
+ goto error;
+ p = http_find_param_list(HTX_SL_REQ_UPTR(sl), HTX_SL_REQ_ULEN(sl), '?');
+ if (!p)
+ goto end;
+ end = HTX_SL_REQ_UPTR(sl) + HTX_SL_REQ_ULEN(sl);
+ len = end-p;
+
+ /* Decode the query-string */
+ chunk_reset(&trash);
+ memcpy(trash.area, p, len);
+ trash.area[len] = 0;
+ len = url_decode(trash.area);
+ if (len == -1)
+ goto error;
+ p = trash.area;
+ end = p + len;
+
+ /* Parse the query-string */
+ while (p < end) {
+ if (*p == '&')
+ ++p;
+ else if (*p == 's' && (end-p) >= 6 && !memcmp(p, "scope=", 6)) {
+ default_scopes = 0; /* at least a scope defined, unset default scopes */
+ p += 6; /* now p point on the parameter value */
+ len = 0; /* len is the value length */
+ while ((p+len) < end && *(p+len) != '&')
+ ++len;
+
+ if (len == 0)
+ appctx->ctx.stats.flags &= ~PROMEX_FL_SCOPE_ALL;
+ else if (len == 1 && *p == '*')
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_ALL;
+ else if (len == 6) {
+ if (!memcmp(p, "global", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_GLOBAL;
+ else if (!memcmp(p, "server", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_SERVER;
+ }
+ else if (len == 7 && !memcmp(p, "backend", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_BACK;
+ else if (len == 8 && !memcmp(p, "frontend", len))
+ appctx->ctx.stats.flags |= PROMEX_FL_SCOPE_FRONT;
+ else
+ goto error;
+
+ p += len;
+ }
+ else {
+ /* ignore all other params for now */
+ while (p < end && *p != '&')
+ p++;
+ }
+ }
+
+ end:
+ appctx->ctx.stats.flags |= default_scopes;
+ return 1;
+
+ error:
+ err = &http_err_chunks[HTTP_ERR_400];
+ channel_erase(res);
+ res->buf.data = b_data(err);
+ memcpy(res->buf.area, b_head(err), b_data(err));
+ res_htx = htx_from_buf(&res->buf);
+ channel_add_input(res, res_htx->data);
+ appctx->st0 = PROMEX_ST_END;
+ return -1;
+}
+
/* Send HTTP headers of the response. It returns 1 on success and 0 if <htx> is
* full. */
static int promex_send_headers(struct appctx *appctx, struct stream_interface *si, struct htx *htx)
@@ -2252,6 +2353,12 @@ static void promex_appctx_handle_io(struct appctx *appctx)
switch (appctx->st0) {
case PROMEX_ST_INIT:
+ ret = promex_parse_uri(appctx, si);
+ if (ret <= 0) {
+ if (ret == -1)
+ goto error;
+ goto out;
+ }
appctx->st0 = PROMEX_ST_HEAD;
appctx->st1 = PROMEX_DUMPER_INIT;
/* fall through */
--
2.21.0
>From 2ddd8a00e052d241619bfde4332d01c09d86dea3 Mon Sep 17 00:00:00 2001
From: Christopher Faulet <cfau...@haproxy.com>
Date: Tue, 19 Nov 2019 14:18:24 +0100
Subject: [PATCH 2/2] MINOR: contrib/prometheus-exporter: Add a param to ignore
servers in maintenance
By passing the parameter "no-maint" in the query-string, it is now possible to
ignore servers in maintenance. It means that the metrics for servers in this
state will not be exported.
---
contrib/prometheus-exporter/README | 8 ++++++++
contrib/prometheus-exporter/service-prometheus.c | 9 +++++++++
2 files changed, 17 insertions(+)
diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README
index 84ae8e27e..b19acc1bd 100644
--- a/contrib/prometheus-exporter/README
+++ b/contrib/prometheus-exporter/README
@@ -73,6 +73,14 @@ exported. Here are examples:
/metrics?scope=*&scope= # ==> no metrics will be exported
/metrics?scope=&scope=global # ==> global metrics will be exported
+* Filtering on servers state
+
+It is possible to exclude from returned metrics all servers in maintenance mode
+passing the parameter "no-maint" in the query-string. This parameter may help to
+solve performance issues of configuration that use the server templates to
+manage dynamic provisionning. Note there is no consistency check on the servers
+state. So, if the state of a server changes while the exporter is running, only
+a part of the metrics for this server will be dumped.
Exported metrics
------------------
diff --git a/contrib/prometheus-exporter/service-prometheus.c b/contrib/prometheus-exporter/service-prometheus.c
index c70daa905..45a4b7aab 100644
--- a/contrib/prometheus-exporter/service-prometheus.c
+++ b/contrib/prometheus-exporter/service-prometheus.c
@@ -68,6 +68,7 @@ enum {
#define PROMEX_FL_SCOPE_FRONT 0x00000010
#define PROMEX_FL_SCOPE_BACK 0x00000020
#define PROMEX_FL_SCOPE_SERVER 0x00000040
+#define PROMEX_FL_NO_MAINT_SRV 0x00000080
#define PROMEX_FL_SCOPE_ALL (PROMEX_FL_SCOPE_GLOBAL|PROMEX_FL_SCOPE_FRONT|PROMEX_FL_SCOPE_BACK|PROMEX_FL_SCOPE_SERVER)
@@ -1905,6 +1906,9 @@ static int promex_dump_srv_metrics(struct appctx *appctx, struct htx *htx)
while (appctx->ctx.stats.sv) {
sv = appctx->ctx.stats.sv;
+ if ((appctx->ctx.stats.flags & PROMEX_FL_NO_MAINT_SRV) && (sv->cur_admin & SRV_ADMF_MAINT))
+ goto next_sv;
+
switch (appctx->st2) {
case ST_F_STATUS:
metric = mkf_u32(FO_STATUS, promex_srv_status(sv));
@@ -2075,6 +2079,7 @@ static int promex_dump_srv_metrics(struct appctx *appctx, struct htx *htx)
if (!promex_dump_metric(appctx, htx, prefix, &metric, &out, max))
goto full;
+ next_sv:
appctx->ctx.stats.sv = sv->next;
}
@@ -2271,6 +2276,10 @@ static int promex_parse_uri(struct appctx *appctx, struct stream_interface *si)
p += len;
}
+ else if (*p == 'n' && (end-p) >= 8 && !memcmp(p, "no-maint", 8)) {
+ appctx->ctx.stats.flags |= PROMEX_FL_NO_MAINT_SRV;
+ p += 8;
+ }
else {
/* ignore all other params for now */
while (p < end && *p != '&')
--
2.21.0
