Re: different log Info between 1.3 and 1.4
On Thu, 23 Sep 2010, Guido Krause wrote:
Hello,
we use haproxy 1.3.22 an when a server goes down I see
in the log:
Aug 19 09:02:54 lbA haproxy[14478]: Server
webA/webA is DOWN. 1 active and 0 backup servers left.
0 sessions active, 0 requeued, 0 remaining in queue.
now we test the 1.4.8 and if a server goes down I see in the log:
Sep 21 17:41:19 lbB haproxy[27434]: Server webA/webA is DOWN, reason: Layer7 wrong
status, code: 503, info: Service Unavailable, check duration: 106ms
why i cone not see - 1 active and 0 backup servers left?
Please test the attached patch. It should restore the old behavior.
Best regards,
Krzysztof OlędzkiFrom 5b88610452727542b9cab6315570feb13e54f644 Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Mon, 27 Sep 2010 13:10:50 +0200
Subject: [BUG] Restore info about available active/backup servers
Bug introduced by 5f5b7d2c1db1fabe3d372b0080a8773147762ee7
---
src/checks.c |2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/src/checks.c b/src/checks.c
index d326625..ef7839b 100644
--- a/src/checks.c
+++ b/src/checks.c
@@ -170,7 +170,7 @@ static void server_status_printf(struct chunk *msg, struct
server *s, unsigned o
chunk_printf(msg, , check duration: %ldms,
s-check_duration);
}
- if (xferred 0) {
+ if (xferred = 0) {
if (!(s-state SRV_RUNNING))
chunk_printf(msg, . %d active and %d backup servers
left.%s
%d sessions active, %d requeued, %d remaining
in queue,
--
1.6.4.2
Re: MySQL + Haproxy Question
On Sat, 24 Oct 2009, Joseph Hardeman wrote: Hey Guys, Hi, I was wondering if there was a way to have Haproxy handle mysql requests. I know that I can use the TCP option instead of HTTP and it will work, but I was wondering if anyone has a way to make haproxy send all requests for Select statements to a set of servers and all Insert, Updates, and Deletes to a master MySQL server. I was just thinking about it and was wondering if this was possible and if anyone has done it. If you have would you be willing to share how your setup is. Currently, there is no MySQL support in HAProxy. However, you should try MySQL_Proxy: http://forge.mysql.com/wiki/MySQL_Proxy Best regards, Krzysztof Olędzki
Re: [PATCH] [MINOR] Capture display more data from health checks
On Mon, 5 Oct 2009, Krzysztof Oledzki wrote:
On Mon, 5 Oct 2009, Willy Tarreau wrote:
On Mon, Oct 05, 2009 at 12:51:07AM +0200, Krzysztof Oledzki wrote:
Non ascii or HTML control characters are masked.
I don't see the reason for masking HTML characters, since the output
is sent over syslog. I'm OK for masking non-ascii characters however.
Because the output is presented on html page - it is added to a popup
generated by td title. But you are right - we should also add it to
syslog.
I'd prefer that we correctly encode at the destination, which means
to do this in the html stats functions. If necessary I can write an
HTML encoder which would take two chunks for instance.
OK, how about this one? If this version is more or less acceptable I'll
clean it and split into three patches:
- adding cut_crlf,
- adding chunk_htmlencode, chunk_asciiencode,
- adding the main functionality.
Comments?
diff --git a/include/common/defaults.h b/include/common/defaults.h
index c09f5a4..b0aee86 100644
--- a/include/common/defaults.h
+++ b/include/common/defaults.h
@@ -174,4 +174,9 @@
#define MAX_HOSTNAME_LEN 32
#endif
+/* Maximum health check description length */
+#ifndef HCHK_DESC_LEN
+#define HCHK_DESC_LEN 128
+#endif
+
#endif /* _COMMON_DEFAULTS_H */
diff --git a/include/common/standard.h b/include/common/standard.h
index 715ed24..8dd2731 100644
--- a/include/common/standard.h
+++ b/include/common/standard.h
@@ -282,6 +282,18 @@ extern int strl2ic(const char *s, int len);
extern int strl2irc(const char *s, int len, int *ret);
extern int strl2llrc(const char *s, int len, long long *ret);
+static inline char *cut_crlf(char *s) {
+ while (*s != '\r' || *s == '\n') {
+ char *p = s++;
+
+ if (!*p)
+ return p;
+ }
+
+ *s++ = 0;
+ return s;
+}
+
/* This function converts the time_t value now into a broken out struct tm
* which must be allocated by the caller. It is highly recommended to use this
* function intead of localtime() because that one requires a time_t* which
diff --git a/include/types/server.h b/include/types/server.h
index dae1a71..9971da2 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -127,6 +127,7 @@ struct server {
struct timeval check_start; /* last health check start time
*/
unsigned long check_duration; /* time in ms took to finish
last health check */
short check_status, check_code; /* check result, check code */
+ char check_desc[HCHK_DESC_LEN]; /* healt check descritpion */
struct freq_ctr sess_per_sec; /* sessions per second on this
server */
int puid; /* proxy-unique server ID, used
for SNMP */
diff --git a/src/buffers.c b/src/buffers.c
index ee00f1c..2da3b16 100644
--- a/src/buffers.c
+++ b/src/buffers.c
@@ -10,6 +10,7 @@
*
*/
+#include ctype.h
#include stdarg.h
#include stdio.h
#include string.h
@@ -325,6 +326,89 @@ int chunk_printf(struct chunk *chk, const char *fmt, ...)
}
/*
+ * Encode chunk src into chunk dst, respecting the limit of at most
chk-size chars.
+ * If the chk-len is over, nothing is added. Returns the new chunk size.
+ */
+int chunk_htmlencode(struct chunk *dst, struct chunk *src) {
+
+ int i, l;
+ int olen, free;
+ char c, t[sizeof(#255;)];
+
+ olen = dst-len;
+
+ for (i = 0; i src-len; i++) {
+ free = dst-size - dst-len;
+
+ if (!free) {
+ dst-len = olen;
+ return dst-len;
+ }
+
+ c = src-str[i];
+
+ if (!isascii(c) || !isprint(c) || c == '' || c == '' || c == '\''
|| c == '' || c == '') {
+ sprintf(t, #%d;, c);
+ l = strlen(t);
+
+ if (free l) {
+ dst-len = olen;
+ return dst-len;
+ }
+
+ memcpy(dst-str + dst-len, t, l);
+ dst-len += l;
+ } else {
+ dst-str[dst-len] = c;
+ dst-len++;
+ }
+ }
+
+ return dst-len;
+}
+
+/*
+ * Encode chunk src into chunk dst, respecting the limit of at most
chk-size chars.
+ * If the chk-len is over, nothing is added. Returns the new chunk size.
+ */
+int chunk_asciiencode(struct chunk *dst, struct chunk *src, char qc) {
+ int i, l;
+ int olen, free;
+ char c, t[sizeof(FF)];
+
+ olen = dst-len;
+
+ for (i = 0; i src-len; i++) {
+ free = dst-size - dst-len;
+
+ if (!free) {
+ dst-len = olen;
+ return dst-len;
+ }
+
+ c = src-str[i];
+
+ if (!isascii(c) || !isprint(c) || c == '' || c == '' || c ==
qc
Re: [PATCH] [MINOR] Capture display more data from health checks
On Mon, 5 Oct 2009, Willy Tarreau wrote: On Mon, Oct 05, 2009 at 12:51:07AM +0200, Krzysztof Oledzki wrote: Non ascii or HTML control characters are masked. I don't see the reason for masking HTML characters, since the output is sent over syslog. I'm OK for masking non-ascii characters however. Because the output is presented on html page - it is added to a popup generated by td title. But you are right - we should also add it to syslog. I'd prefer that we correctly encode at the destination, which means to do this in the html stats functions. If necessary I can write an HTML encoder which would take two chunks for instance. Sure, it would be great. Feature can be disabled by defining HCHK_DESC_LEN to 0. What could be the reason for disabling this feature ? After all, if people enable log-health-checks, it means they want more verbose info. To save HCHK_DESC_LEN==128 bytes for each server. If you found it unnecessary I'm more than happy to remove this check. I don't care much about a proxy or a server size. I don't like to waste, but overall it does not change much the memory usage. Even large configs with 1000 servers will still only be hit by 128 kB. You can bet that those running 1000 servers are not counting their RAM in kilobytes ;-) What I care about however, is that most of the hot struct members are grouped so that they remain hot in the CPU caches. The other important thing is to avoid increasing the session size. Yes, I noticed you moved the place where the counters structuers are located. Next time I'll try to play with pahole from DWARF2 to find a better location for new variables. ;) Best regards, Krzysztof Olędzki
Re: [PATCH] [DOC] Add information about http://haproxy1wt.eu/contrib.html
On Sun, 4 Oct 2009, Willy Tarreau wrote: Hi Krzysztof, Hi Willy, I have merged your 5 patches. I have also added minor fixes to the doc and added the clear counters line to the stats usage text. Thanks. Concerning the stats socket, I think we'll soon have to add some levels of access control and/or write protection, because there will surely be some environments where we'll want to prevent the clear counters from happening. True. We may think about adding an option to disable write/clear commands and only provide a read functionality. I'm also planning on adding the ability to connect to the stats socket from a backend, possibly conditionned by an ACL. This would allow more monitoring systems to connect via TCP and/or remotely monitor the stats. So, implementing a write control is going to be more important. Also, at first I was not sure about the use of your socket stats. But I must admit that after having used them, I find them nice and useful. For instance, when there is a bind line per physical interface, it will permit per-interface monitoring. Also, it allows monitoring of upstream servers with very little requirement (only a few more IPs or ports). Exactly. I have been using this functionality for nearly a year now so I can count how much of the traffic goes to remote clients by expensive uplinks or by cheap peerings or to local clients. That's all good work, thanks ! Thank you. I'd recommend you to rebase your tree upon latest -git, since I've finished converting the stats sockets to the new applets. OK. Krzysztof Olędzki
Re: [PATCH] [BUG] Fix NULL pointer dereference in stats_check_uri_auth()
On Sun, 4 Oct 2009, Krzysztof Piotr Oledzki wrote:
From fc217df5d282cfbc275f3eff286885aae0a0e117 Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Sun, 4 Oct 2009 23:00:16 +0200
Subject: [BUG] Fix NULL pointer dereference in stats_check_uri_auth()
Recent struct chunk rework exposed a NULL pointer dereference:
txn-auth_hdr is used even if it was not initialized and now
haproxy segfaults if auth is required for stats but not found.
This patch fixes it.
---
src/proto_http.c |5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/src/proto_http.c b/src/proto_http.c
index 6cd0b40..a73a580 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -4563,7 +4563,7 @@ int stats_check_uri_auth(struct session *t, struct proxy
*backend)
struct http_txn *txn = t-txn;
struct uri_auth *uri_auth = backend-uri_auth;
struct user_auth *user;
- int authenticated, cur_idx;
+ int authenticated, cur_idx, found = 0;
char *h;
memset(t-data_ctx.stats, 0, sizeof(t-data_ctx.stats));
@@ -4632,12 +4632,13 @@ int stats_check_uri_auth(struct session *t, struct
proxy *backend)
!strncasecmp(Authorization:, h, 14)) {
txn-auth_hdr.str = h;
txn-auth_hdr.len = len;
+ found++;
break;
}
h += len + txn-hdr_idx.v[cur_idx].cr + 1;
}
- if (txn-auth_hdr.len 21 ||
+ if (!found || txn-auth_hdr.len 21 ||
memcmp(txn-auth_hdr.str + 14, Basic , 7))
user = NULL;
Please hold, txn-auth_hdr.len 21 should handle it - the bug is
probably in a different place.
Best regards,
Krzysztof Olędzki
Re: [PATCH] [MINOR] Capture display more data from health checks
On Mon, 5 Oct 2009, Willy Tarreau wrote:
On Sun, Oct 04, 2009 at 09:35:33PM +0200, Krzysztof Piotr Oledzki wrote:
From 1299a2fe5768c502786ef28cd78dae83a31f0c83 Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Sun, 4 Oct 2009 21:28:53 +0200
Subject: [MINOR] Capture display more data from health checks
Capture display more data from health checks, like
strerror(errno) for L4 failed checks or a first line
from a response for L7 successes/failed checks.
I have a few questions about this one.
Sure :)
Non ascii or HTML control characters are masked.
I don't see the reason for masking HTML characters, since the output
is sent over syslog. I'm OK for masking non-ascii characters however.
Because the output is presented on html page - it is added to a popup
generated by td title. But you are right - we should also add it to
syslog.
Feature can be disabled by defining HCHK_DESC_LEN to 0.
What could be the reason for disabling this feature ? After
all, if people enable log-health-checks, it means they want
more verbose info.
To save HCHK_DESC_LEN==128 bytes for each server. If you found it
unnecessary I'm more than happy to remove this check.
@@ -656,50 +679,78 @@ static int event_srv_chk_r(int fd)
(memcmp(trash, HTTP/1., 7) != 0 ||
(trash[12] != ' ' trash[12] != '\r')) ||
!isdigit(trash[9]) || !isdigit(trash[10]) ||
!isdigit(trash[11])) {
- set_server_check_status(s, HCHK_STATUS_L7RSP);
+
+ desc = trash;
+ p = strchr(desc, '\r');
+ if (p)
+ *p = '\0';
Here we should also check for '\n' alone, let's add this function
to standard.h to find and replace CR/LFs with a zero. It's small
and efficient enough to be reused for generic uses :
static inline char *cut_crlf(char *s)
{
do {
if (*s == '\n' || *s == '\r')
*s = 0;
} while (*s++);
return s;
}
Then :
desc = trash;
cut_crlf(desc);
Sure. How about adding a break after a first match?
I think that if the outputs are small enough, we'll be able to
make them appear as automatic popups in the HTML stats page
when the check column is highlighted.
Yep, this is exactly the place where you can find them currently. ;)
Best regards,
Krzysztof Olędzki
Re: [PATCH] [MEDIUM] add description, node and show-node/show-desc, deprecate node-name
On Mon, 28 Sep 2009, Willy Tarreau wrote: Hi Krzysztof, On Sun, Sep 27, 2009 at 09:36:19PM +0200, Krzysztof Piotr Oledzki wrote: In my configuration there are several stats backends with different proxies displayed and requiring to put the same name in each one is not very nice. Finally, one day we might use node name for some other purposes. Please also note that this option was implemented very recently - the number of affected users should be very limited. I have a problem with this change. Some users offer shared services to their customers, and for them the node or description will be per-proxy since their customers will only see one instance. For instance, they would prefer to report master or node 1/3 instead of psrv432. But the master or node 1/3 role depends on the customer. Right, I didn't think about such situation. That said, I found that the global info was missing when dumping the stats via the unix socket. Ooops. ;) So what we could do would be the following : - keep the global node and description as you did - keep the show-node and show-desc stats argument - add optional values after those arguments, allowing one to force the names for a customer. Excellent idea. So you could have this : global node Paris description This node is in bay 13, DC xxx, Paris, France ... backend cust-0003 # report global values for this customer stats show-node stats show-desc ... backend cust-0045 # report specific values for this customer stats show-node Europe stats show-desc Master node for Europe, Asia, Africa Would you be OK with such a solution ? Sure, I'll make a v2 patch. So, I'm tagging it [MEDIUM] instead of [MINOR] and keeping the node-name option only to display a warning message. You can leave it MINOR as no version was released with the node-name value yet. I think it's just on my sites (as a test), so I will notice it and fix it when I upgrade. Also in general, I prefer that we don't report keywords which do not exist, because I did that once (accidentely) and got questions about where to find info about keyword XXX which did not appear in the doc. OK. Best regards, Krzysztof Olędzki
Re: [PATCH] [MEDIUM] Health check reporting code rework + health logging, v2
On Fri, 25 Sep 2009, Willy Tarreau wrote: On Fri, Sep 25, 2009 at 02:36:18PM -0700, David Birdsong wrote: For me 404 is always failed check, even with disable-on-404 as it disables balancing to that server. Maybe with disable-on-404 it is not fatal - a proxy may decide that a server is up, but still... Well, it is a special case defined as a success in haproxy's specs. If you set up disable-on-404, AND the server returns 404, it means it is still alive but does not want to receive any new session. This is an admin-defined maintenance mode, which of course, excludes the ability to detect a failure on 404. It's almost only used with dynamic servlets tuned to respond like this, but I know there are a few users running with it on purely static sites in order to ease content deployment. aye, we use 404's for marking a server up/down in a static environment. basically local daemons that make load based decisions to add or remove the static health check file attracting or repelling traffic from the host. Don't worry David, I'm not talking about *always* reporting 404 as a success, just for the situations documented as such and explicitly required by the user (option disable-on-404). Exactly. No one is going to change the current functionality, we are only talking about adding a new one and we are trying to find the best way to do it. ;) Best regards, Krzysztof Olędzki
Re: [PATCH] [MEDIUM] Health check reporting code rework + health logging
On Thu, 24 Sep 2009, Willy Tarreau wrote: Hi Krzysztof, On Thu, Sep 24, 2009 at 08:54:36PM +0200, Krzysztof Piotr Oledzki wrote: From e6f4aeafe8a7f547679d80a65892315f8f5cf047 Mon Sep 17 00:00:00 2001 From: Krzysztof Piotr Oledzki o...@ans.pl Date: Thu, 24 Sep 2009 20:48:34 +0200 Subject: [MEDIUM] Health check reporting code rework + health logging This patch adds health logging so it possible to check what was happening before a crash. Failed healt checks are logged if server is UP and succeeded healt checks if server is DOWN, so the ammountof additional information is limited. That's an interesting feature. Thanks. I've quickly tested it and I think one information is missing : the up or down count (1/3, ...). For instance, I have this : Sep 24 23:04:32 localhost haproxy[24369]: Health check for server Thousand_HTTP/127.0.0.001 failed, reason: Layer4 connection problem, check duration: 0ms. Sep 24 23:04:33 localhost haproxy[24369]: Health check for server Thousand_HTTP/127.0.0.001 failed, reason: Layer4 connection problem, check duration: 0ms. Sep 24 23:04:33 localhost haproxy[24369]: Server Thousand_HTTP/127.0.0.001 is DOWN. 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. There is no way to know if the log above contains a dup or indicates two lost checks. This would be especially true for high traffic sites where checks account for about 1/1000 to 1/1 of the traffic, and where log losses are common. Good idea. Sure, I'll add this. I've seen that you have maintained the LOG_NOTICE level, which is fine because many setups use the log level to split between traffic and UP/Downs. OK. One humble query though. You seem to have difficulties typing health in doc and comments without eating letters ;-) I can't blame you, I can't type health-check at once either ;-) My bad, I make to many mistakes, I know. I should have check my patches with ispell or at least read them more carefully. ;) I'll do my best next time. Best regards, Krzysztof Olędzki
Re: [PATCH] [MEDIUM] Collect show information about last health check, v2
On Thu, 17 Sep 2009, Willy Tarreau wrote: Hi Krzysztof, On Thu, Sep 17, 2009 at 04:21:20PM +0200, Krzysztof Oledzki wrote: (...) Strange, I must have overlooked something, probably because I don't use pure tcp mode so my tests were rather limited, even too limited. :( No problem, what matters is that it's improving :-) Yep, silly overlook: --- a/src/checks.c +++ b/src/checks.c @@ -523,6 +523,7 @@ static int event_srv_chk_w(int fd) /* good TCP connection is enough */ s-result |= SRV_CHK_RUNNING; + set_server_check_status(s, HCHK_STATUS_L4OK); goto out_wakeup; } } I'll do some more tests to make sure all cases are now really covered and send the final patch, I hope. ;) Then we may talk about more improvements. :) Thanks, Krzysztof Olędzki
Re: [PATCH] [MEDIUM] Collect show information about last health check, v2
On Sun, 13 Sep 2009, Willy Tarreau wrote:
Hi Krzysztof,
On Tue, Sep 08, 2009 at 11:52:35PM +0200, Krzysztof Piotr Oledzki wrote:
From c0fafebc289c0f4a02eb10c3134e5752d4a37f97 Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Tue, 8 Sep 2009 23:40:30 +0200
Subject: [MEDIUM] Collect show information about last health check, v2
OK it's a lot better now. I'm still observing something strange. If
the check is pure TCP (no option *chk set), the test appears as
always in progress (with the *) and the last check may either appear
as INI or L4CON. I see INI when I send it to a web server, and
L4CON if I send it to this loop : while sleep 1;do nc -lp80;done.
In fact, it even seems like the state changes from INI only if the
server is started after haproxy.
Strange, I must have overlooked something, probably because I don't use
pure tcp mode so my tests were rather limited, even too limited. :(
I suppose there is something such as data being needed to mark the
test as complete, but that does not appear clear to me. As you can
see below, the connect still happens :
epoll_wait(3, {}, 7, 581) = 0
gettimeofday({1252865407, 589160}, NULL) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 7
fcntl64(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
connect(7, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(127.0.0.1)}, 16) = -1 EINPROGRESS (Operation now in
progress)
connect(7, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(127.0.0.1)}, 16) = 0
epoll_wait(3, {}, 8, 0) = 0
gettimeofday({1252865407, 590139}, NULL) = 0
close(7)= 0
epoll_wait(3, {}, 7, 1000) = 0
Thank you for the detailed report.
If you don't have time to debug this, please tell me, so that I
try to find out what's happening (I'm short in time too in fact).
Otherwise it looks OK to me, and I'm willing to merge it early so
that if we get suggestions of improvements we can merge them ASAP.
Thanks, I'll take care of the above bug so we will be able to marge it
eventually. ;)
Best regards,
Krzysztof Olędzki
Re: [PATCH 1/2] [MEDIUM] Collect show information about last health check
On Tue, 8 Sep 2009, Willy Tarreau wrote:
On Mon, Sep 07, 2009 at 10:47:48PM +0200, Krzysztof Oledzki wrote:
However, I found that it was hard to understand the status codes in
the HTML stats page. Some people are already complaining about columns
they don't trivially understand, but here I think that status codes
are close to cryptic. Also, while it is not *that* hard to tell
which one means what when you can compare all of them, they must be
unambiguous when found individually.
I could propose some adjustments, but it's just a proposal, I'm open
to other suggestions :
Currently there are 9 status:
UNK - unknown
From my tests, this means this one is not tested.
Or not tested yet - when haproxy starts.
Ah OK. Maybe we'd need an additional 'INI' state then ?
Good idea.
(...)
L14OK - check passed on layer 1-4, no upper layers testing enabled
L14TMOUT - layer 1-4 timeout
L14UNR- layer 1-4 unreachable, for example
Connection refused (tcp rst) or No route to host (icmp)
I'd suggest L4 instead of L14. And also after all, layer 4 is what
we're aware of, and people are used to associate it to TCP (eventhough
TCP is rather layer 5 if we're purist).
Are you sure? Last time when I checked TCP was L4,
Problem is, TCP/IP doesn't perfectly fit the ISO model. Everyone seems
to agree that IP is layer 3, and so conclude that anything above IP is
layer 4. However, while I have no problem saying that UDP is layer 4
(it contains port information), TCP is higher as it has session states
which are a very important part of the protocol, and session is layer 5.
Some (rare) firewall vendors who correctly follow TCP sequence numbers
even claim to have layer 5 firewalls as opposed to all other ones which
are more layer 4 (basically support port+flags).
Yep, it is indeed very confusing. However, for me L4 provides end-to-end
connections and reliability and there is no reliability in tcp without
sequence numbers. L5 (session) is defined as interhost communication and
covers protocols like SSH for example, which is definitely above TCP.
Maybe the name - session is misleading? Anyway, I have no intention in
startining an academic discussion, so...
and for example ICMP
(No route to host) is L3
it depends for whom. Due to the reason above, some people see it at
3, others at 4 because it's above IP, and others say 3.5 because UDP
and TCP rely on it too.
and can report problems in for example L2, like
ARP resolution issues.
same here, you can hear about layer 2 because of MAC addresses, L3
because of IP addresses and routing tables checking to decide
whether to reply or not, or layer 2.5 because IP relies on it.
So, I would rather stay with 1-4 if that is OK for you.
I find it a lot more confusing. We all know that layer 4 cannot work
if lower layers don't either. So either L4 is OK, and it implies that
all lower layers are OK, or L4 is not OK and we can't say anything
about lower layers. However, I find it interesting to have L4OK,
because if later we support ping, we'll be able to report L3OK for
example.
... OK, let's agree that L4OK means L1-4 is OK and TCP works.
Same principle here, I'd suggest L7 instead of L57 for the same reasons
as above. Let's shorten the TMOUT one too. We could also turn L57INVRSP
into L7INV.
SSL/TLS is for example L6 and one day we may support checks for L5, like
for example SSH/SCP or RTCP.
once again it depends for whom. For some people, a transport layer is
layer4, whether it's clear or ciphered. For others, there's some encoding
notion so they'd rather tell layer 6. It's all a matter of embedding low
layers into higher ones, which is not covered by the OSI model.
It depends if you want to blindly translate 4 layers of TCP/IP into
7 layers of OSI/ISO or not. Most people do it that way, so we have L3.5
for ICMP or L2.5 for ARP, but it does not mean we have to do it the same way.
After all, when you make an openvpn tunnel work on top of an HTTP
session across an HTTP proxy, you provide layer 3 on top of some layer
6/7 itself on top of another layer 7...
Exactly - it is L3 over L6/7 over L7.
Anyway, what I find commonly understood by people in the field is that
L3 is rounting only, L4 is routing+ports and L7 is anything implying a
payload. This is even how products are advertised and it sort of makes
sense, eventhough it's far from being exact.
Product's names and definitions are very often mangled by marketing
departments so we end up with hardware raid provided by windows drivers
or 24bit LCD displays that are rather 18-19 bits + interpolation. I think
we should tend to focus on the primary definition.
But saying that L7 works
means that we could talk. L4 works means we can connect. L3 works means
we can reach.
OK, but how about pure SSL/TLS? I don't think we should call it L7.
However, we may add an additional L6{OK,TOUT,INV} states and teach the
code to use it when necessary, what do you think?
During my
Re: [PATCH 1/2] [MEDIUM] Collect show information about last health check
On Sat, 5 Sep 2009, Willy Tarreau wrote:
Hi Krzysztof,
Hi Willy,
On Mon, Aug 31, 2009 at 09:27:40PM +0200, Krzysztof Piotr Oledzki wrote:
From cbfda2a0808cd5d5cbf17ec7f0d04f7091bec9cb Mon Sep 17 00:00:00 2001
From: Krzysztof Piotr Oledzki o...@ans.pl
Date: Mon, 31 Aug 2009 21:04:02 +0200
Subject: [MEDIUM] Collect show information about last health check
Collect information about last health check result,
including L5-7 code if possible (for example http or smtp
return code) and time took to finish last check.
Health check info is provided on both stats pages (html csv)
and logged when a server is marekd UP or DOWN. Currently active
check are marked with an asterisk, but only in html mode.
I have tested this a bit, and I must say this is awesome. The idea
of marking a currently active check is really smart too !
Thanks.
However, I found that it was hard to understand the status codes in
the HTML stats page. Some people are already complaining about columns
they don't trivially understand, but here I think that status codes
are close to cryptic. Also, while it is not *that* hard to tell
which one means what when you can compare all of them, they must be
unambiguous when found individually.
I could propose some adjustments, but it's just a proposal, I'm open
to other suggestions :
Currently there are 9 status:
UNK - unknown
From my tests, this means this one is not tested.
Or not tested yet - when haproxy starts.
Maybe we should
not indicate anything at all then, or just report NONE?
Right, if checking is disabled we shouldn't put any info here.
Also we would not want the in 0 ms indicator either for this case.
Sure, will fix it.
SOCKERR - socket error
Good idea to detect that one.
L14OK - check passed on layer 1-4, no upper layers testing enabled
L14TMOUT - layer 1-4 timeout
L14UNR- layer 1-4 unreachable, for example
Connection refused (tcp rst) or No route to host (icmp)
I'd suggest L4 instead of L14. And also after all, layer 4 is what
we're aware of, and people are used to associate it to TCP (eventhough
TCP is rather layer 5 if we're purist).
Are you sure? Last time when I checked TCP was L4, and for example ICMP
(No route to host) is L3 and can report problems in for example L2, like
ARP resolution issues. So, I would rather stay with 1-4 if that is OK for
you.
I'd suggest shortening the
timeout label, something like L4TIM, L4TOUT, L4TMO, ... I think the
former is more intuitive.
Maybe TOUT?
Would you agree to rename L14UNR to L4CON, to indicate a failure to
connect ? In fact, unreachable is a bit too precise and not necessarily
right. It's common to speak about unreachability in terms of routing,
and I find it confusing to read UNR when a check fails on my own machine.
Also, if one day we get more precise reports by exploiting errno a bit
better, we will then be able to report a real unreachable error.
Right, CON seems to be much better choice.
L57OK - check passed on layer 5-7
L57TMOUT - layer 5-7 (HTTP/SMTP/SSL) timeout
L57INVRSP - layer 5-7 invalid response - protocol error
L57RSPERR - layer 5-7 response error, for example HTTP 5xx
Same principle here, I'd suggest L7 instead of L57 for the same reasons
as above. Let's shorten the TMOUT one too. We could also turn L57INVRSP
into L7INV.
SSL/TLS is for example L6 and one day we may support checks for L5, like
for example SSH/SCP or RTCP. Ack for INVRSP - INV.
The response error indicates that the server was reached but that its
status code was not accepted. Would you be OK with L7STS then ?
Good idea.
I also suggest that we reduce the column name Last check to
LastChk to prevent the browser from wrapping it over two lines.
We may also use nowrap here, but ideed, LastChk looks better.
During my tests, I got a SOCKERR while the server was stopped. I
think it's an asynchronous connection refused which triggered
this error, probably here (but I may be wrong) :
@@ -370,8 +468,10 @@ static int event_srv_chk_w(int fd)
}
else if (ret == 0 || errno == EAGAIN)
goto out_poll;
- else
+ else {
+ set_server_check_status(s, HCHK_STATUS_SOCKERR);
goto out_error;
+ }
}
How it is possible to reproduce this situation?
I also found a missing comma at the end of the CSV title (you
remember, we always get this wrong) :
@@ -222,6 +224,7 @@ int print_csv_header(struct chunk *msg, int size)
chkfail,chkdown,lastchg,downtime,qlimit,
pid,iid,sid,throttle,lbtot,tracked,type,
rate,rate_lim,rate_max,
+ check_status,check_code,check_duration
\n);
You should add one after 'check_duration', as all lines in
Re: [PATCH 1/2] [MEDIUM] Collect show information about last health check
On Mon, 7 Sep 2009, Dmitry Sivachenko wrote: On Sat, Sep 05, 2009 at 07:00:05PM +0200, Willy Tarreau wrote: However, I found that it was hard to understand the status codes in the HTML stats page. Some people are already complaining about columns they don't trivially understand, but here I think that status codes are close to cryptic. Also, while it is not *that* hard to tell which one means what when you can compare all of them, they must be unambiguous when found individually. Please consider title attribute, which may be used for most HTML elements. Browser displays it as popup hint when you put mouse cursor over that element. It consumes zero space on display, but can provide helpful information when needed. Example: html body table border=1 tr td title=aaa titleaaa tdbbb /table /body /html Put mouse cursor on aaa table cell and you will see aaa title hint. I like this idea very much, I'll implement it. Thanks! Best regsrds, Krzysztof Olędzki
Re: redispatch optimization
On Mon, 31 Aug 2009, Dmitry Sivachenko wrote:
Hello!
Hello,
If we are running with 'option redispatch' and
'retries' parameter set to some positive value,
the behaviur is as follows:
###
In order to avoid immediate reconnections to a server which is restarting,
a turn-around timer of 1 second is applied before a retry occurs.
When option redispatch is set, the last retry may be performed on another
server even if a cookie references a different server.
###
While is makes sence to wait for some time (1 second)
before attempting another connection to the same
server, there is no reason to wait 1 second before
attempting the last connection to another server
(with option redispatch). It is just a waste of one
second.
Please consider the following patch to attempt last try
immediately. (If our main server does not respond, there is
no reason to assume another one cant answer now).
Very clever idea, however it is no certain that we would not end up on the
same sever anyway, so this may not always work.
PS: another important suggestion is to make that delay tunable
parameter (like timeout.connect, etc), rather than hardcode
1000ms in code.
Why would you like to change the value? I found 1s very well chosen.
Quoting Willy:
Initially, I wanted to reuse the contimeout as the turn-around timeout,
but there are people using very large contimeouts because it was the
same value which dictated the tarpit and queue timeouts in previous
versions, so this would result in reconnection attempts being delayed
too long. So finally I arbitrarily selected a one second delay to
attempt a reconnection after a failure.
If this value is considered too low by some users and too high by
others, then we'll introduce timeout reconnect :-)
--- work/haproxy-1.4-dev2/src/session.c 2009-08-10 00:57:09.0 +0400
+++ /tmp/session.c 2009-08-31 14:28:26.0 +0400
@@ -306,7 +306,11 @@ int sess_update_st_cer(struct session *s
si-err_type = SI_ET_CONN_ERR;
si-state = SI_ST_TAR;
+ if (s-srv s-conn_retries == 0 s-be-options PR_O_REDISP) {
+ si-exp = tick_add(now_ms, MS_TO_TICKS(0));
+ } else {
si-exp = tick_add(now_ms, MS_TO_TICKS(1000));
+ }
return 0;
}
return 0;
There is no value in adding 0ms, also SI_ST_TAR should be moved inside the
condition I think, not sure if it is enough.
Best regards,
Krzysztof Olędzki
Re: Backend Server UP/Down Debugging?
On Thu, 27 Aug 2009, Dmitry Sivachenko wrote: On Thu, Aug 27, 2009 at 08:45:23AM +0200, Krzysztof Oledzki wrote: On Wed, Aug 26, 2009 at 02:00:42PM -0700, Jonah Horowitz wrote: I???m watching my servers on the back end and occasionally they flap. I???m wondering if there is a way to see why they are taken out of service. I???d like to see the actual response, or at least a HTTP status code. right now it's not archived. I would like to keep a local copy of the last request sent and response received which caused a state change, but that's not implemented yet. I wanted to clean up the stats socket first, but now I realize that we could keep at least some info (eg: HTTP status, timeout, ...) in the server struct itself and report it in the log. Nothing of that is performed right now, so you may have to tcpdump at best :-( As always, I have a patch for that, solving it nearly exactly like you described it. ;) However for the last half year I have been rather silent, mostly because it is very important time in my private life, so I think I'm partially excused. ;) I know that there are some unfinished tasks (acl for exapmple) so I'll try to push ASAP, maybe starting from the easier patches, likt this ones. The rest will have to wait when I get back from honeymoon. I see flapping servers in my logs too and also have no clue why haproxy disables them. If you have a patch to log the reason why the particular server was disabled, I'd love to test it (I run 1.4-dev2). Please check the attached patch. This code is far from being ready for inclusion, however I cleaned it and ported to 1.4-dev2, so it should work for you. With this patch you should be able to see status of last check in the stats page and to check in you logs, why servers are considered down, for example: [WARNING] 241/141556 (1971) : Backup Server p1/s1 is DOWN, reason: Layer5-7 response error(10), code: 400, check duration: 0ms. 0 active and 4 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 241/141702 (4006) : Backup Server p2/s6 is DOWN, reason: Layer5-7 timeout(6), check duration: 10001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Best regards, Krzysztof Olędzkidiff -ur haproxy-1.4-dev2-orig/include/proto/checks.h haproxy-1.4-dev2/include/proto/checks.h --- haproxy-1.4-dev2-orig/include/proto/checks.h2009-08-09 22:57:09.0 +0200 +++ haproxy-1.4-dev2/include/proto/checks.h 2009-08-28 19:35:41.0 +0200 @@ -25,6 +25,8 @@ #include types/task.h #include common/config.h +const char *get_check_status_description(short check_status); +const char *get_check_status_info(short check_status); struct task *process_chk(struct task *t); int start_checks(); Only in haproxy-1.4-dev2/include/types: checks.h diff -ur haproxy-1.4-dev2-orig/include/types/server.h haproxy-1.4-dev2/include/types/server.h --- haproxy-1.4-dev2-orig/include/types/server.h2009-08-09 22:57:09.0 +0200 +++ haproxy-1.4-dev2/include/types/server.h 2009-08-28 22:46:03.0 +0200 @@ -35,6 +35,7 @@ #include types/proxy.h #include types/queue.h #include types/task.h +#include types/checks.h /* server flags */ @@ -74,7 +75,7 @@ struct server *next; int state; /* server state (SRV_*) */ int prev_state; /* server state before last change (SRV_*) */ - int cklen; /* the len of the cookie, to speed up checks */ + int cklen; /* the len of the cookie, to speed up checks */ int rdr_len;/* the length of the redirection prefix */ char *cookie; /* the id set in the cookie */ char *rdr_pfx; /* the redirection prefix */ @@ -121,9 +122,12 @@ long long failed_checks, down_trans;/* failed checks and up-down transitions */ unsigned down_time; /* total time the server was down */ time_t last_change; /* last time, when the state was changed */ + struct timeval check_start; /* last health check start time */ + unsigned long check_duration; /* */ + short check_status, check_code; /* check result, check code */ long long failed_conns, failed_resp;/* failed connect() and responses */ - long long retries, redispatches;/* retried and redispatched connections */ + long long retries, redispatches;/* retried and redispatched connections */ long long failed_secu; /* blocked responses because of security concerns */ struct freq_ctr sess_per_sec; /* sessions per second on this server */ unsigned int sps_max
Re: Backend Server UP/Down Debugging?
On Sun, 30 Aug 2009, Willy Tarreau wrote:
Hi Krzysztof,
Hi Willy,
On Sun, Aug 30, 2009 at 02:21:55PM +0200, Krzysztof Oledzki wrote:
Please check the attached patch. This code is far from being ready for
inclusion, however I cleaned it and ported to 1.4-dev2, so it should work
for you.
This is a really nice work.
Thanks.
I'm eseing a few FIXMEs with random status
codes which look like debugging tags. I think they will not have any
impact on the tests though.
Exactly, they poits where I have something to deal with, mostly to find
the proper code to return or to add some docs/comments. ;)
My thought about this feature was to store two statuses :
- the one which made the server change state
- the last one
Indeed, it's very common to see a server first timeout, then crash.
Or it may also return 500 server error or things like this, which
will not be available anymore afterwards. And we need to keep the
last one because it's what helps people troubleshoot in production
typically when one error hides another one.
Right, it is a very good idea.
However, this is something which can be implemented incrementally.
So we can start with your patch and improve it later.
OK. ;)
With this patch you should be able to see status of last check in the
stats page and to check in you logs, why servers are considered down, for
example:
[WARNING] 241/141556 (1971) : Backup Server p1/s1 is DOWN, reason: Layer5-7
response error(10), code: 400, check duration: 0ms. 0 active and 4 backup
servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining
in queue.
This really is excellent. I'm already impatient to merge it ;-)
Thanks, again. :)
I have a few comments/questions below :
@@ -474,40 +572,69 @@
if (s-proxy-options PR_O_HTTP_CHK) {
/* Check if the server speaks HTTP 1.X */
if ((len strlen(HTTP/1.0 000\r)) ||
- (memcmp(trash, HTTP/1., 7) != 0)) {
+ (memcmp(trash, HTTP/1., 7) != 0 ||
+ (trash[12] != ' ' trash[12] != '\r')) ||
+ !isdigit(trash[9]) || !isdigit(trash[10]) ||
!isdigit(trash[11])) {
here you're adding some tests to more accurately check the HTTP
status line. Is it just for cleanness or did you encounter cases
where the check looked valid but was not ?
Mostly for cleanness, however as I need to convert string to int below it
is nice to make sure it is really a three-digit-string. Alternatively I
could use strtol, however I found it easier to do everything in one place.
+ trash[12] = '\0';
+ s-check_code = atoi(trash[9]);
you can use one of the str2* functions (see standard.h and standard.c)
which automatically stops at the first non-digit character.
OK, will do.
/* check the reply : HTTP/1.X 2xx and 3xx are OK */
- if (trash[9] == '2' || trash[9] == '3')
+ if (trash[9] == '2' || trash[9] == '3') {
s-result |= SRV_CHK_RUNNING;
- else if ((s-proxy-options PR_O_DISABLE404)
+ set_server_check_status(s, HCHK_STATUS_L57OKD);
I think you wanted to put HCHK_STATUS_L57OK here, not OKD since we're
in the 2xx/3xx state and not 404 disable. Or maybe I misunderstood the
OKD status ?
OKD means we have Layer5-7 data avalible, like for example http code.
Several times I found that some of my servers were misconfigured and were
returning a 3xx code redirecting to a page-not-found webpage instead of
doing a proper healt-check, so I think it is good to know what was the
response, even if it was OK (2xx/3xx).
+ } else if ((s-proxy-options PR_O_DISABLE404)
(s-state SRV_RUNNING)
-(memcmp(trash[9], 404, 3) == 0)) {
+(s-check_code == 404)) {
/* 404 may be accepted as stopping only if the server
was up */
s-result |= SRV_CHK_RUNNING | SRV_CHK_DISABLE;
+ set_server_check_status(s, HCHK_STATUS_L57OKD);
}
- else
+ else {
s-result |= SRV_CHK_ERROR;
+ set_server_check_status(s, HCHK_STATUS_L57RSPERR);
+ }
(...)
else if (s-proxy-options PR_O_SMTP_CHK) {
+ /* Check if the server speaks SMTP */
+ if ((len strlen(000\r)) ||
+ (trash[3] != ' ' trash[3] != '\r') ||
+ !isdigit(trash[0]) || !isdigit(trash[1]) ||
!isdigit(trash[2])) {
+ s-result |= SRV_CHK_ERROR;
+ set_server_check_status(s, HCHK_STATUS_L57INVRSP);
+ goto out_wakeup;
+ }
+
+ trash[3] = '\0';
+ s-check_code = atoi(trash[0]);
same here for the status code conversion.
Ack.
+
/* Check for SMTP code 2xx
Re: Backend Server UP/Down Debugging?
On Thu, 27 Aug 2009, Willy Tarreau wrote: Hi, Hi, On Wed, Aug 26, 2009 at 02:00:42PM -0700, Jonah Horowitz wrote: I???m watching my servers on the back end and occasionally they flap. I???m wondering if there is a way to see why they are taken out of service. I???d like to see the actual response, or at least a HTTP status code. right now it's not archived. I would like to keep a local copy of the last request sent and response received which caused a state change, but that's not implemented yet. I wanted to clean up the stats socket first, but now I realize that we could keep at least some info (eg: HTTP status, timeout, ...) in the server struct itself and report it in the log. Nothing of that is performed right now, so you may have to tcpdump at best :-( As always, I have a patch for that, solving it nearly exactly like you described it. ;) However for the last half year I have been rather silent, mostly because it is very important time in my private life, so I think I'm partially excused. ;) I know that there are some unfinished tasks (acl for exapmple) so I'll try to push ASAP, maybe starting from the easier patches, likt this ones. The rest will have to wait when I get back from honeymoon. One trick you can do to make this experience better is to set a port or addr option on your server line to run the checks on a different ip/port combination. That way, you can filter that on tcpdump and only get the checks. Or to use something like: echo -e GET query HTTP/1.0\r\nhost: host\r\n|nc host port|less -S Best regards, Krzysztof Olędzki
Re: stats are cut off
On Mon, 22 Jun 2009, Angelo Höngens wrote: Hey guys and girls, Hello, I'm a happy user of HAProxy, and for one of my new projects I'm running into a small problem. I have a new configuration with a 120 different instances (one instance for every site on a couple of servers), and it looks like everything works fine. However, I have a problem with the stats page, it will only show the first 38,5 sites :) Please see this screenshot: http://files.hongens.nl/2009/06/22/haproxystats.png Somewhere it's cut off after a /tr tag. This is kind of annoying, because I use the stats page to see the status of the web server nodes behind HAProxy. Don't see anything interesting in the logs.. Any ideas anyone? Which version? Does it always break at the same position? Could you share your config with us? ;) Best regards, Krzysztof Olędzki
Re: Setting up SNMP
On Tue, 3 Feb 2009, Ruy Diaz wrote: Hi, Hi, I've been trying to get SNMP monitoring to work for the last few days and have had quite a hard time getting it up. I am quite new to Linux, load balancing and SNMP so please take it easy on me. I am running Ubuntu 8.10, Haproxy 1.3.15.7 and I have just compiled net-snmp 5.4.2.1 with perl enabled (tried configuring with both v3 and v2c without success). I have copied haproxy.pl to /etc/snmp/haproxy and I modified /etc/snmp/snmpd.conf to include the lines indicated in the netsnmp-perl README. However, when I run: $ sudo snmpbulkwalk -c public -v2c 127.0.0.1 1.3.6.1.4.1.29385.106 SNMPv2-SMI::enterprises.29385.106 = No more variables left in this MIB View (It is past the end of the MIB tree) Digging through forums I thought what I needed was to add a 'stats socket /var/run/haproxy.stat mod 777' line to my haproxy config, but when I add this, I get the following error: [ALERT] 032/163027 (24099) : parsing [/etc/haproxy/haproxy.cfg:29] : unknown stats parameter 'stats' (expects 'hide-version', 'uri', 'realm', 'auth' or 'enable'). [ALERT] 032/163027 (24099) : Error reading configuration file : /etc/haproxy/haproxy.cfg Are you sure you have the 1.3.15.7 version? Your net-snmp daemon has to be able to communicate with you haproxy instance and it is not possible to do this without the 'stats socket' support. Plase show the output from haproxy -vv. Best regards, Krzysztof Olędzki
