We have found that the current mechanism of qtime, ctime, rtime, and ttime
based on last 1024 requests is not the most suitable to debug/visualize
latency issues with servers, especially if they happen to last a very short
time. For live dashboards showing server timings, we found an additional
last-'n' seconds metrics useful. The logs could also be parsed to derive
these
values, but suffers from delays at high volume, requiring higher processing
power and enabling logs.
The 'last-n' seconds metrics per server/backend can be configured as follows
in the HAProxy configuration file:
backend backend-1
stats-period 32
...
To retrieve these stats at the CLI (in addition to existing metrics), run:
echo show stat-duration time 3 | socat /var/run/admin.sock stdio
These are also available on the GUI.
The justification for this patch are:
1. Allows to capture spikes for a server during a short period. This helps
having dashboards that show server response times every few seconds (e.g.
every 1 second), so as to be able to chart it across timelines.
2. Be able to get an average across different time intervals, e.g. the
configuration file may specify to save the last 32 seconds, but the cli
interface can request for average across any interval upto 32 seconds.
E.g.
the following command prints the existing metrics appended by the time
based ones for the last 1 second:
echo show stat-duration time 1 | socat /var/run/admin.sock stdio
Running the following existing command appends the time-based metric
values
based on the time period configured in the configuration file per
backend/server:
echo show stat | socat /var/run/admin.sock stdio
3. Option per backend for configuring the server stat's time interval, and.
no API breakage to stats (new metrics are added at end of line).
Please review, any feedback on the code/usability/extensibility is very much
appreciated.
Thanks,
- Krishna Kumar
From 55d8966ee185031814aa97368adc262c78540705 Mon Sep 17 00:00:00 2001
From: Krishna Kumar <krishna.ku@kkumar>
Date: Thu, 22 Dec 2016 09:28:02 +0530
Subject: [PATCH] Implement a last-'n' seconds based counters for
backend/server's queue, connect, response and session times.
---
doc/configuration.txt | 29 ++++++
include/common/cfgparse.h | 3 +
include/proto/stats.h | 3 +-
include/types/applet.h | 1 +
include/types/counters.h | 35 +++++++
include/types/proxy.h | 4 +
include/types/server.h | 1 +
include/types/stats.h | 7 ++
src/cfgparse.c | 51 ++++++++++-
src/haproxy.c | 6 ++
src/hlua_fcn.c | 2 +-
src/proxy.c | 39 ++++++++
src/server.c | 7 ++
src/stats.c | 226 +++++++++++++++++++++++++++++++++++++++++++++-
src/stream.c | 7 ++
15 files changed, 414 insertions(+), 7 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index f654c8e..dc0ddae 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -8078,6 +8078,35 @@ stats uri <prefix>
See also : "stats auth", "stats enable", "stats realm"
+stats-period <nsecs>
+ Enable time based average queue, connect, response and session times for
+ this backend and all servers contained in it.
+ May be used in sections : defaults | frontend | listen | backend
+ no | no | no | yes
+ Arguments :
+ <nsecs> is the maximum seconds for which to maintain the four
+ averages (queue, connect, response and session times). 'nsecs' is a
+ non-negative number, and should be a power of 2, with the maximum value
+ being 128. To retrieve the averages, use one of the following methods:
+ - echo show stat | socat /var/run/admin.sock stdio
+ Shows existing statistics, appended by 4 metrics corresponding to
+ average queue, connect, response and session times. These four
+ times are averaged across the last 'nsecs' seconds.
+ - echo show stat-duration time <n> | socat /var/run/admin.sock stdio
+ Shows existing statistics, appended by 4 metrics corresponding to
+ average queue, connect, response and session times. These four
+ times are averaged across the last 'n' seconds.
+
+ Note : When the backend does not specify the 'stats-period' option, these
+ metrics are zeroed out.
+
+ Example :
+ # Preserve last 32 seconds of average queue, connect, response and session
+ # times for a backend and all servers in that backend:
+ backend backend-1
+ stats-period 32
+ server a.b.c.d a.b.c.d:80
+ server a.b.c.e a.b.c.e:80
stick match <pattern> [table <table>] [{if | unless} <cond>]
Define a request pattern matching condition to stick a user to a server
diff --git a/include/common/cfgparse.h b/include/common/cfgparse.h
index fd04b14..101cad9 100644
--- a/include/common/cfgparse.h
+++ b/include/common/cfgparse.h
@@ -110,6 +110,9 @@ static inline int warnifnotcap(struct proxy *proxy, int cap, const char *file, i
return 0;
}
+/* Maximum seconds of stats to be preserved per server/backend */
+#define MAX_STATS_TIME_PERIOD 128
+
#endif /* _COMMON_CFGPARSE_H */
/*
diff --git a/include/proto/stats.h b/include/proto/stats.h
index ac893b8..f33ceb1 100644
--- a/include/proto/stats.h
+++ b/include/proto/stats.h
@@ -95,7 +95,8 @@ int stats_fill_fe_stats(struct proxy *px, struct field *stats, int len);
int stats_fill_li_stats(struct proxy *px, struct listener *l, int flags,
struct field *stats, int len);
int stats_fill_sv_stats(struct proxy *px, struct server *sv, int flags,
- struct field *stats, int len);
+ struct field *stats, int len,
+ struct stream_interface *si);
int stats_fill_be_stats(struct proxy *px, int flags, struct field *stats, int len);
extern struct applet http_stats_applet;
diff --git a/include/types/applet.h b/include/types/applet.h
index 759b905..146e0cd 100644
--- a/include/types/applet.h
+++ b/include/types/applet.h
@@ -110,6 +110,7 @@ struct appctx {
unsigned int flags; /* STAT_* */
int iid, type, sid; /* proxy id, type and service id if bounding of stats is enabled */
int st_code; /* the status code returned by an action */
+ int duration; /* duration/period for stats */
} stats;
struct {
struct bref bref; /* back-reference from the session being dumped */
diff --git a/include/types/counters.h b/include/types/counters.h
index 06ce617..a33e01f 100644
--- a/include/types/counters.h
+++ b/include/types/counters.h
@@ -105,6 +105,41 @@ struct be_counters {
} p; /* protocol-specific stats */
};
+/*
+ * Struct time-stat-counters:
+ *
+ * Implement a last-n seconds based counters for backend/server's q/c/r/t
+ * times. Total memory consumption on x86_64 to maintain all 4 event time
+ * values per backend/server is: 16 + #seconds * 56 bytes.
+ * for 1 second: 72 bytes
+ * for 16 seconds: 912 bytes
+ * for 32 seconds: 1808 bytes
+ * for 64 second: 3600 bytes
+ */
+struct __tstat_counters {
+ unsigned int timestamp; /* time when this entry was added */
+ long t_queue; /* sum of queue times */
+ long t_connect; /* sum of times to connect to server */
+ long t_data; /* sum of times to get response */
+ long t_close; /* sum of session times */
+ int t_queue_counter; /* total number of 'queue' entries */
+ int t_connect_counter; /* total number of 'connect' entries */
+ int t_data_counter; /* total number of 'response' entries */
+ int t_close_counter; /* total number of 'session' entries */
+};
+
+struct tstat_counters {
+ int max_entries; /* max number of 'times' entries */
+ struct __tstat_counters *times; /* Stats for every second */
+};
+
+extern int tstat_counters_init(struct tstat_counters *tstat,
+ int num_entries);
+extern int enable_be_tstat_counters(void); /* wrapper function */
+extern void set_tstat_counters(struct tstat_counters *tstat, unsigned int now,
+ int t_queue, int t_connect, int t_data,
+ int t_close);
+
#endif /* _TYPES_COUNTERS_H */
/*
diff --git a/include/types/proxy.h b/include/types/proxy.h
index 3bf5a4d..97c5732 100644
--- a/include/types/proxy.h
+++ b/include/types/proxy.h
@@ -365,6 +365,8 @@ struct proxy {
struct be_counters be_counters; /* backend statistics counters */
struct fe_counters fe_counters; /* frontend statistics counters */
+ struct tstat_counters be_tstat_counters; /* last 'n' time counters for backend */
+
struct list listener_queue; /* list of the temporarily limited listeners because of lack of a proxy resource */
struct stktable table; /* table for storing sticking streams */
@@ -435,6 +437,8 @@ struct proxy {
* name is used
*/
struct list filter_configs; /* list of the filters that are declared on this proxy */
+
+ int stats_period; /* time based stats duration */
};
struct switching_rule {
diff --git a/include/types/server.h b/include/types/server.h
index 20c314b..792708e 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -193,6 +193,7 @@ struct server {
int maxqueue; /* maximum number of pending connections allowed */
struct freq_ctr sess_per_sec; /* sessions per second on this server */
struct be_counters counters; /* statistics counters */
+ struct tstat_counters srv_tstat_counters; /* last 'n' time counter per server */
struct list pendconns; /* pending connections */
struct list actconns; /* active connections */
diff --git a/include/types/stats.h b/include/types/stats.h
index 48cf645..213c1cc 100644
--- a/include/types/stats.h
+++ b/include/types/stats.h
@@ -23,6 +23,7 @@
/* Flags for applet.ctx.stats.flags */
#define STAT_FMT_HTML 0x00000001 /* dump the stats in HTML format */
#define STAT_FMT_TYPED 0x00000002 /* use the typed output format */
+#define STAT_DURATION 0x00000004 /* time stats for a specific duration */
#define STAT_HIDE_DOWN 0x00000008 /* hide 'down' servers in the stats page */
#define STAT_NO_REFRESH 0x00000010 /* do not automatically refresh the stats page */
#define STAT_ADMIN 0x00000020 /* indicate a stats admin level */
@@ -377,6 +378,12 @@ enum stat_field {
ST_F_DCON,
ST_F_DSES,
+ /* Offsets for time based stat counter for Q/C/R/T times */
+ ST_F_QTIME_PERIOD,
+ ST_F_CTIME_PERIOD,
+ ST_F_RTIME_PERIOD,
+ ST_F_TTIME_PERIOD,
+
/* must always be the last one */
ST_F_TOTAL_FIELDS
};
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 3ed2c22..b212ee6 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -2551,6 +2551,50 @@ static void free_email_alert(struct proxy *p)
p->email_alert.myhostname = NULL;
}
+/*
+ * Parse "stats-period" option in backend specifying how many
+ * seconds of stats to keep per server/backend.
+ */
+int parse_stats_period(const char *file, int linenum, char **args,
+ struct proxy *curproxy, struct proxy *defproxy)
+{
+ int err_code = 0;
+
+ if (!(curproxy->cap & PR_CAP_BE)) {
+ Alert("parsing [%s:%d] : '%s' should be in the backend section.\n",
+ file, linenum, args[0]);
+ err_code = ERR_ALERT | ERR_FATAL | ERR_ABORT;
+ goto out;
+ }
+
+ if (!*args[1] || *args[2]) {
+ Alert("parsing [%s:%d] : '%s' expects <time-period> as argument.\n",
+ file, linenum, args[0]);
+ err_code = ERR_ALERT | ERR_FATAL | ERR_ABORT;
+ goto out;
+ }
+
+ curproxy->stats_period = atoi(args[1]);
+ if (curproxy->stats_period <= 0 ||
+ curproxy->stats_period & (curproxy->stats_period - 1)) {
+ Alert("parsing [%s:%d] : '%s' expects time period to be a power of 2.\n",
+ file, linenum, args[0]);
+ err_code = ERR_ALERT | ERR_FATAL | ERR_ABORT;
+ goto out;
+ }
+
+ if (curproxy->stats_period > MAX_STATS_TIME_PERIOD) {
+ Alert("parsing [%s:%d] : Time period %d greater than %d is not supported as counters may overflow.\n",
+ file, linenum, curproxy->stats_period,
+ MAX_STATS_TIME_PERIOD);
+ err_code = ERR_ALERT | ERR_FATAL | ERR_ABORT;
+ goto out;
+ }
+
+out:
+ return err_code;
+}
+
int cfg_parse_listen(const char *file, int linenum, char **args, int kwm)
{
static struct proxy *curproxy = NULL;
@@ -2906,7 +2950,12 @@ int cfg_parse_listen(const char *file, int linenum, char **args, int kwm)
curproxy->conf.args.line = linenum;
/* Now let's parse the proxy-specific keywords */
- if (!strcmp(args[0], "server") || !strcmp(args[0], "default-server")) {
+ if (!strcmp(args[0], "stats-period")) {
+ err_code |= parse_stats_period(file, linenum, args, curproxy, &defproxy);
+ if (err_code & ERR_FATAL)
+ goto out;
+ }
+ else if (!strcmp(args[0], "server") || !strcmp(args[0], "default-server")) {
err_code |= parse_server(file, linenum, args, curproxy, &defproxy);
if (err_code & ERR_FATAL)
goto out;
diff --git a/src/haproxy.c b/src/haproxy.c
index 239db80..57536ea 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -860,6 +860,12 @@ static void init(int argc, char **argv)
}
#endif
+ /* Enable time stats counters for those backends that asked for it */
+ if (enable_be_tstat_counters()) {
+ Alert("Unable to allocate backend time-counter stats.\n");
+ exit(1);
+ }
+
/* Apply server states */
apply_server_state();
diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index 5ac533a..f21603f 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -517,7 +517,7 @@ int hlua_server_get_stats(lua_State *L)
return 1;
}
- stats_fill_sv_stats(srv->proxy, srv, ST_SHLGNDS, stats, STATS_LEN);
+ stats_fill_sv_stats(srv->proxy, srv, ST_SHLGNDS, stats, STATS_LEN, NULL);
lua_newtable(L);
for (i=0; i<ST_F_TOTAL_FIELDS; i++) {
diff --git a/src/proxy.c b/src/proxy.c
index 7e5d83b..cfa74c5 100644
--- a/src/proxy.c
+++ b/src/proxy.c
@@ -1546,6 +1546,45 @@ static int cli_parse_enable_frontend(char **args, struct appctx *appctx, void *p
return 1;
}
+/* Initialize data structure for a single server/backend */
+int tstat_counters_init(struct tstat_counters *tstat, int num_entries)
+{
+ tstat->max_entries = num_entries;
+ if (!num_entries) {
+ tstat->times = NULL;
+ return 0;
+ }
+
+ tstat->times = calloc(num_entries, sizeof(*tstat->times));
+ return tstat->times == NULL;
+}
+
+/* Wrapper function for tstat_counters_init() for all backends */
+int enable_be_tstat_counters(void)
+{
+ struct proxy *curproxy;
+ int err_code = 0;
+
+ for (curproxy = proxy; curproxy != NULL; curproxy = curproxy->next) {
+ /*
+ * Allocate BE tstat_counters only for those backends that
+ * have atleast one server configured and stats-period is
+ * > 0. This is safe to call even if stats-period was not set.
+ */
+ if ((curproxy->cap & PR_CAP_BE) &&
+ !curproxy->be_tstat_counters.times) {
+ if (tstat_counters_init(&curproxy->be_tstat_counters,
+ curproxy->stats_period)) {
+ err_code = ERR_ALERT | ERR_FATAL;
+ goto out;
+ }
+ }
+ }
+
+out:
+ return err_code;
+}
+
/* register cli keywords */
static struct cli_kw_list cli_kws = {{ },{
{ { "disable", "frontend", NULL }, "disable frontend : temporarily disable specific frontend", cli_parse_disable_frontend, NULL, NULL },
diff --git a/src/server.c b/src/server.c
index 479631c..e85084f 100644
--- a/src/server.c
+++ b/src/server.c
@@ -1131,6 +1131,13 @@ int parse_server(const char *file, int linenum, char **args, struct proxy *curpr
newsrv->dns_opts.family_prio = AF_INET6;
}
+ if (tstat_counters_init(&newsrv->srv_tstat_counters,
+ curproxy->stats_period)) {
+ Alert("Unable to allocate server time-cntr stats.\n");
+ err_code |= ERR_ALERT | ERR_FATAL;
+ goto out;
+ }
+
while (*args[cur_arg]) {
if (!strcmp(args[cur_arg], "agent-check")) {
global.maxsock++;
diff --git a/src/stats.c b/src/stats.c
index 976496e..571ec11 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -216,6 +216,10 @@ const char *stat_field_names[ST_F_TOTAL_FIELDS] = {
[ST_F_INTERCEPTED] = "intercepted",
[ST_F_DCON] = "dcon",
[ST_F_DSES] = "dses",
+ [ST_F_QTIME_PERIOD] = "qt_period",
+ [ST_F_CTIME_PERIOD] = "ct_period",
+ [ST_F_RTIME_PERIOD] = "rt_period",
+ [ST_F_TTIME_PERIOD] = "tt_period",
};
/* one line of info */
@@ -385,7 +389,7 @@ static int stats_dump_fields_typed(struct chunk *out, const struct field *stats)
* reserved for the checkbox is ST_SHOWADMIN is set in <flags>. Some extra info
* are provided if ST_SHLGNDS is present in <flags>.
*/
-static int stats_dump_fields_html(struct chunk *out, const struct field *stats, unsigned int flags)
+static int stats_dump_fields_html(struct chunk *out, const struct field *stats, unsigned int flags, struct proxy *px)
{
struct chunk src;
@@ -727,6 +731,15 @@ static int stats_dump_fields_html(struct chunk *out, const struct field *stats,
chunk_appendf(out, "<tr><th>- Response time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_RTIME].u.u32));
chunk_appendf(out, "<tr><th>- Total time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_TTIME].u.u32));
+ if (px->be_tstat_counters.times) {
+ chunk_appendf(out, "<tr><th colspan=3>Avg over last %d seconds.</th></tr>", px->stats_period);
+ chunk_appendf(out, "<tr><th>- Queue time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_QTIME_PERIOD].u.s64));
+ chunk_appendf(out, "<tr><th>- Connect time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_CTIME_PERIOD].u.s64));
+ if (strcmp(field_str(stats, ST_F_MODE), "http") == 0)
+ chunk_appendf(out, "<tr><th>- Response time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_RTIME_PERIOD].u.s64));
+ chunk_appendf(out, "<tr><th>- Total time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_TTIME_PERIOD].u.s64));
+ }
+
chunk_appendf(out,
"</table></div></u></td>"
/* sessions: lbtot, last */
@@ -949,6 +962,15 @@ static int stats_dump_fields_html(struct chunk *out, const struct field *stats,
chunk_appendf(out, "<tr><th>- Response time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_RTIME].u.u32));
chunk_appendf(out, "<tr><th>- Total time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_TTIME].u.u32));
+ if (px->be_tstat_counters.times) {
+ chunk_appendf(out, "<tr><th colspan=3>Avg over last %d seconds.</th></tr>", px->stats_period);
+ chunk_appendf(out, "<tr><th>- Queue time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_QTIME_PERIOD].u.s64));
+ chunk_appendf(out, "<tr><th>- Connect time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_CTIME_PERIOD].u.s64));
+ if (strcmp(field_str(stats, ST_F_MODE), "http") == 0)
+ chunk_appendf(out, "<tr><th>- Response time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_RTIME_PERIOD].u.s64));
+ chunk_appendf(out, "<tr><th>- Total time:</th><td>%s</td><td>ms</td></tr>", U2H(stats[ST_F_TTIME_PERIOD].u.s64));
+ }
+
chunk_appendf(out,
"</table></div></u></td>"
/* sessions: lbtot, last */
@@ -1025,7 +1047,7 @@ int stats_dump_one_line(const struct field *stats, unsigned int flags, struct pr
flags |= ST_SHOWADMIN;
if (appctx->ctx.stats.flags & STAT_FMT_HTML)
- return stats_dump_fields_html(&trash, stats, flags);
+ return stats_dump_fields_html(&trash, stats, flags, px);
else if (appctx->ctx.stats.flags & STAT_FMT_TYPED)
return stats_dump_fields_typed(&trash, stats);
else
@@ -1231,6 +1253,176 @@ static const char *srv_hlt_st[SRV_STATS_STATE_COUNT] = {
[SRV_STATS_STATE_NO_CHECK] = "no check"
};
+/*
+ * Return true if time difference is within a 'max' period. This is used
+ * to implement a 'last-n' stats average for queue/connect/response/total
+ * times.
+ */
+int within_period(unsigned int prev, unsigned int curr, int max)
+{
+ return curr - prev <= max;
+}
+
+/*
+ * Time stat entries have one of the following values:
+ * - 0 (never used, or was set with a value of 0. May or may not be valid)
+ * - -1 (time was not relevant at update, this entry is not used to
+ * calculate the average)
+ * - non-zero (used in the past, may be valid or invalid).
+ * To determine whether the value is useful or not, we compare current
+ * time to saved time. We cannot stop once we reach a time outside of the
+ * interval, as intermediate entries may have old values and not be updated
+ * if no updates took place at those times. This means going through the
+ * entire array, but it is better to do extra work at (rare) retrieval times
+ * (control plane), rather than updating unused entries at run time (data
+ * plane). This leads to fast updates and slower retrievals.
+ */
+static void get_tstat_counters(struct tstat_counters *tstat,
+ struct stream_interface *si,
+ unsigned int now, struct field *qtimep,
+ struct field *ctimep, struct field *rtimep,
+ struct field *ttimep)
+{
+ int i;
+ int64_t qtime = 0, ctime = 0, rtime = 0, ttime = 0;
+ int64_t qcounter = 0, ccounter = 0, rcounter = 0, tcounter = 0;
+ int duration = tstat->max_entries;
+
+ /*
+ * Return zero for those backends that did not ask for time based
+ * stats.
+ */
+ if (!tstat->times) {
+ *qtimep = mkf_u32(FN_AVG, 0);
+ *ctimep = mkf_u32(FN_AVG, 0);
+ *rtimep = mkf_u32(FN_AVG, 0);
+ *ttimep = mkf_u32(FN_AVG, 0);
+ return;
+ }
+
+ if (si && si->end) {
+ /*
+ * If user via CLI specified a specific time period over which
+ * to report stats, then retrieve that value instead of the
+ * default value specified in the haproxy configuration file.
+ */
+ struct appctx *appctx = objt_appctx(si->end);
+
+ if (appctx && (appctx->ctx.stats.flags & STAT_DURATION))
+ duration = MAX(MIN(duration,
+ appctx->ctx.stats.duration), 0);
+ }
+
+ /* Sum all entries to calculate time window based average */
+ for (i = 0; i < tstat->max_entries; i++) {
+ if (!within_period(tstat->times[i].timestamp, now, duration)) {
+ /* Outside the requested period, but cannot break out */
+ continue;
+ }
+
+ /* Valid time period - increment counts/average */
+ qtime += tstat->times[i].t_queue;
+ ctime += tstat->times[i].t_connect;
+ rtime += tstat->times[i].t_data;
+ ttime += tstat->times[i].t_close;
+
+ qcounter += tstat->times[i].t_queue_counter;
+ ccounter += tstat->times[i].t_connect_counter;
+ rcounter += tstat->times[i].t_data_counter;
+ tcounter += tstat->times[i].t_close_counter;
+ }
+
+ /* Avoid divide-by-zero errors */
+ if (!qcounter)
+ qcounter = 1;
+ if (!ccounter)
+ ccounter = 1;
+ if (!rcounter)
+ rcounter = 1;
+ if (!tcounter)
+ tcounter = 1;
+
+ /* Convert average times to microsecond */
+ *qtimep = mkf_u32(FN_AVG, qtime / qcounter);
+ *ctimep = mkf_u32(FN_AVG, ctime / ccounter);
+ *rtimep = mkf_u32(FN_AVG, rtime / rcounter);
+ *ttimep = mkf_u32(FN_AVG, ttime / tcounter);
+}
+
+/* Update time-stat counter values if enabled for this backend */
+void set_tstat_counters(struct tstat_counters *tstat,
+ unsigned int now, int t_queue,
+ int t_connect, int t_data, int t_close)
+{
+ if (tstat->times) {
+ unsigned int slot = now % tstat->max_entries;
+
+ /*
+ * Optimize for heavy load when many requests are made in
+ * the same second. During very low load, the following
+ * 'likely' will become false more often.
+ */
+ if (likely(now == tstat->times[slot].timestamp)) {
+ /*
+ * Times for the same second - increment times.
+ * Do this only if time is not -1.
+ */
+ if (likely(t_queue >= 0)) {
+ tstat->times[slot].t_queue += t_queue;
+ tstat->times[slot].t_queue_counter++;
+ }
+ if (likely(t_connect >= 0)) {
+ tstat->times[slot].t_connect += t_connect;
+ tstat->times[slot].t_connect_counter++;
+ }
+ if (likely(t_data >= 0)) {
+ tstat->times[slot].t_data += t_data;
+ tstat->times[slot].t_data_counter++;
+ }
+ if (likely(t_close >= 0)) {
+ tstat->times[slot].t_close += t_close;
+ tstat->times[slot].t_close_counter++;
+ }
+ } else {
+ /*
+ * Times for a new interval, overwrite
+ * times/counters. Note: earlier entries are not
+ * reset, and may be stale.
+ */
+ tstat->times[slot].timestamp = now;
+
+ if (likely(t_queue >= 0)) {
+ tstat->times[slot].t_queue = t_queue;
+ tstat->times[slot].t_queue_counter = 1;
+ } else {
+ tstat->times[slot].t_queue = 0;
+ tstat->times[slot].t_queue_counter = 0;
+ }
+ if (likely(t_connect >= 0)) {
+ tstat->times[slot].t_connect = t_connect;
+ tstat->times[slot].t_connect_counter = 1;
+ } else {
+ tstat->times[slot].t_connect = 0;
+ tstat->times[slot].t_connect_counter = 0;
+ }
+ if (likely(t_data >= 0)) {
+ tstat->times[slot].t_data = t_data;
+ tstat->times[slot].t_data_counter = 1;
+ } else {
+ tstat->times[slot].t_data = 0;
+ tstat->times[slot].t_data_counter = 0;
+ }
+ if (likely(t_close >= 0)) {
+ tstat->times[slot].t_close = t_close;
+ tstat->times[slot].t_close_counter = 1;
+ } else {
+ tstat->times[slot].t_close = 0;
+ tstat->times[slot].t_close_counter = 0;
+ }
+ }
+ }
+}
+
/* Fill <stats> with the server statistics. <stats> is
* preallocated array of length <len>. The length of the array
* must be at least ST_F_TOTAL_FIELDS. If this length is less
@@ -1238,7 +1430,8 @@ static const char *srv_hlt_st[SRV_STATS_STATE_COUNT] = {
* returns 1. <flags> can take the value ST_SHLGNDS.
*/
int stats_fill_sv_stats(struct proxy *px, struct server *sv, int flags,
- struct field *stats, int len)
+ struct field *stats, int len,
+ struct stream_interface *si)
{
struct server *via, *ref;
char str[INET6_ADDRSTRLEN];
@@ -1440,6 +1633,12 @@ int stats_fill_sv_stats(struct proxy *px, struct server *sv, int flags,
stats[ST_F_RTIME] = mkf_u32(FN_AVG, swrate_avg(sv->counters.d_time, TIME_STATS_SAMPLES));
stats[ST_F_TTIME] = mkf_u32(FN_AVG, swrate_avg(sv->counters.t_time, TIME_STATS_SAMPLES));
+ get_tstat_counters(&sv->srv_tstat_counters, si, now.tv_sec,
+ &stats[ST_F_QTIME_PERIOD],
+ &stats[ST_F_CTIME_PERIOD],
+ &stats[ST_F_RTIME_PERIOD],
+ &stats[ST_F_TTIME_PERIOD]);
+
if (flags & ST_SHLGNDS) {
switch (addr_to_str(&sv->addr, str, sizeof(str))) {
case AF_INET:
@@ -1477,7 +1676,7 @@ static int stats_dump_sv_stats(struct stream_interface *si, struct proxy *px, in
{
struct appctx *appctx = __objt_appctx(si->end);
- if (!stats_fill_sv_stats(px, sv, flags, stats, ST_F_TOTAL_FIELDS))
+ if (!stats_fill_sv_stats(px, sv, flags, stats, ST_F_TOTAL_FIELDS, si))
return 0;
return stats_dump_one_line(stats, flags, px, appctx);
@@ -1562,6 +1761,12 @@ int stats_fill_be_stats(struct proxy *px, int flags, struct field *stats, int le
stats[ST_F_RTIME] = mkf_u32(FN_AVG, swrate_avg(px->be_counters.d_time, TIME_STATS_SAMPLES));
stats[ST_F_TTIME] = mkf_u32(FN_AVG, swrate_avg(px->be_counters.t_time, TIME_STATS_SAMPLES));
+ get_tstat_counters(&px->be_tstat_counters, NULL, now.tv_sec,
+ &stats[ST_F_QTIME_PERIOD],
+ &stats[ST_F_CTIME_PERIOD],
+ &stats[ST_F_RTIME_PERIOD],
+ &stats[ST_F_TTIME_PERIOD]);
+
return 1;
}
@@ -3139,6 +3344,18 @@ static int cli_parse_show_stat(char **args, struct appctx *appctx, void *private
return 0;
}
+static int cli_parse_show_stat_duration(char **args, struct appctx *appctx,
+ void *private)
+{
+ if (!strcmp(args[2], "time")) {
+ appctx->ctx.stats.duration = atoi(args[3]);
+ appctx->ctx.stats.flags |= STAT_DURATION;
+ return 0;
+ }
+ else
+ return cli_parse_show_stat(args, appctx, private);
+}
+
static int cli_io_handler_dump_info(struct appctx *appctx)
{
return stats_dump_info_to_buffer(appctx->owner);
@@ -3157,6 +3374,7 @@ static struct cli_kw_list cli_kws = {{ },{
{ { "clear", "counters", NULL }, "clear counters : clear max statistics counters (add 'all' for all counters)", cli_parse_clear_counters, NULL, NULL },
{ { "show", "info", NULL }, "show info : report information about the running process", cli_parse_show_info, cli_io_handler_dump_info, NULL },
{ { "show", "stat", NULL }, "show stat : report counters for each proxy and server", cli_parse_show_stat, cli_io_handler_dump_stat, NULL },
+ { { "show", "stat-duration", NULL }, "show stat-duration time <seconds> : report averages for each server within last 'seconds' duration", cli_parse_show_stat_duration, cli_io_handler_dump_stat, NULL },
{{},}
}};
diff --git a/src/stream.c b/src/stream.c
index 522cd34..cc36d78 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -2472,11 +2472,18 @@ void stream_update_time_stats(struct stream *s)
swrate_add(&srv->counters.c_time, TIME_STATS_SAMPLES, t_connect);
swrate_add(&srv->counters.d_time, TIME_STATS_SAMPLES, t_data);
swrate_add(&srv->counters.t_time, TIME_STATS_SAMPLES, t_close);
+
+ set_tstat_counters(&srv->srv_tstat_counters,
+ now.tv_sec, t_queue, t_connect,
+ t_data, t_close);
}
swrate_add(&s->be->be_counters.q_time, TIME_STATS_SAMPLES, t_queue);
swrate_add(&s->be->be_counters.c_time, TIME_STATS_SAMPLES, t_connect);
swrate_add(&s->be->be_counters.d_time, TIME_STATS_SAMPLES, t_data);
swrate_add(&s->be->be_counters.t_time, TIME_STATS_SAMPLES, t_close);
+
+ set_tstat_counters(&s->be->be_tstat_counters, now.tv_sec,
+ t_queue, t_connect, t_data, t_close);
}
/*
--
2.1.4
