Suggestion: Customisable style sheet
Hi. I am sorry if this subject has been discussed before... I have dedicated one monitor to HAproxy's status page. However as it is now the name given in the config file's listen line is dark red. And red is usually associated with problem situations. So at a quick glance I have many times thought that some server is down. It would be easy for Willy to change the color but to cater for all needs, I believe that a better solution would be a customizable style sheet??? B __ Information from ESET NOD32 Antivirus, version of virus signature database 4953 (20100317) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com
[ANNOUNCE] haproxy-1.4.2
Hi all, While things are calming down with 1.4, quite a bunch of issues were reported and fixed, so it was time to release 1.4.2 without waiting for any more new issue to come by surprise. Those running on 1.4 should really upgrade as there is a risk of response truncation when using the chunked transfer encoding, the anonymous ACLs don't work as expected, the on-error server check speedup may be trigger by clients, the url_param hash may return a dead server, and it's possible to sometimes keep an erroneous connection forever when using keep-alive and pipelining. That's a lot of issues, and there were other ones, please check the changelog. User reports have been invaluable, and I particularly want to thank the ones who sent me network captures and strace outputs. There were some improvements too, especially related to the checks. We can now check servers which return their response in multiple packets. In my opinion, all pending issues on 1.4 are closed, so I think that distro packagers may reasonably start to open the branch (I've seen that some have already started). I don't intend to release 1.4.3 immediately afterwards, because right now I have nothing else to fix. However I hope that we'll find some time to work on the ECV patch and get it merged into 1.4.3. Nick has fixed many issues from the initial one but a few ones still remain. Last, I have not yet opened the knownbugs-1.4 page, but I will have to do so soon. As usual everything's here : site index : http://haproxy.1wt.eu/ sources: http://haproxy.1wt.eu/download/1.4/src/ changelog : http://haproxy.1wt.eu/download/1.4/src/CHANGELOG binaries : http://haproxy.1wt.eu/download/1.4/bin/ I wish you only good experience with this one ! Willy
Re: Truncated health check response from real servers
On Wed, Mar 17, 2010 at 09:48:42PM +0100, Cyril Bonté wrote:
> Le Mercredi 17 Mars 2010 21:12:45, Willy Tarreau a écrit :
> > (...) I don't
> > agree with restting the buffer or even considering we have an error
> > when a session does not close, because it is a regression. For instance,
> > all people using HTTP/1.1 checks will see a problem here.
>
> hey right, I suddenly realize I forgot to recheck that !
and I found that it broke SMTP checks too !
Here's the fix, it works for all cases I could find (SMTP, HTTP,
multi-byte responses, ...). It starts parsing responses as soon
as enough bytes are available or waits for more data if required.
In fact the code to test the length was already present to detect
short responses. I have just added a test to know if we can hope
for more data or not.
I'll also apply your previous patch about the buffer allocation.
Thanks!
Willy
>From 039381855dd2cefa4e0c827514c0124728030c65 Mon Sep 17 00:00:00 2001
From: Willy Tarreau
Date: Wed, 17 Mar 2010 21:52:07 +0100
Subject: [PATCH] [BUG] checks: don't wait for a close to start parsing the
response
MIME-Version: 1.0
Content-Type: text/plain; charset=latin1
Content-Transfer-Encoding: 8bit
Cyril Bonté reported a regression introduced with very last changes
on the checks code, which causes failed checks on if the server does
not close the connection in time. This happens on HTTP/1.1 checks or
on SMTP checks for instance.
This fix consists in restoring the old behaviour of parsing as soon
as something is available in the response buffer, and waiting for
more data if some are missing. This also helps releasing connections
earlier (eg: a GET check will not have to download the whole object).
---
src/checks.c | 45 -
1 files changed, 32 insertions(+), 13 deletions(-)
diff --git a/src/checks.c b/src/checks.c
index 82939dc..e6fec7d 100644
--- a/src/checks.c
+++ b/src/checks.c
@@ -865,6 +865,7 @@ static int event_srv_chk_r(int fd)
struct task *t = fdtab[fd].owner;
struct server *s = t->context;
char *desc;
+ int done;
if (unlikely((s->result & SRV_CHK_ERROR) || (fdtab[fd].state ==
FD_STERROR))) {
/* in case of TCP only, this tells us if the connection failed
*/
@@ -884,24 +885,22 @@ static int event_srv_chk_r(int fd)
* with running the checks without attempting another socket read.
*/
+ done = 0;
for (len = 0; s->check_data_len < BUFSIZE; s->check_data_len += len) {
len = recv(fd, s->check_data + s->check_data_len, BUFSIZE -
s->check_data_len, 0);
if (len <= 0)
break;
}
- if (len < 0) { /* recv returned an error */
- if (errno == EAGAIN) {
- /* not ready, we want to poll first */
- fdtab[fd].ev &= ~FD_POLL_IN;
- return 0;
- }
-
+ if (len == 0)
+ done = 1; /* connection hangup received */
+ else if (len < 0 && errno != EAGAIN) {
/* Report network errors only if we got no other data. Otherwise
* we'll let the upper layers decide whether the response is OK
* or not. It is very common that an RST sent by the server is
* reported as an error just after the last data chunk.
*/
+ done = 1;
if (!s->check_data_len) {
if (!(s->result & SRV_CHK_ERROR))
set_server_check_status(s, HCHK_STATUS_SOCKERR,
NULL);
@@ -909,18 +908,21 @@ static int event_srv_chk_r(int fd)
}
}
- /* Full response received.
-* Terminate string in check_data buffer... */
+ /* Intermediate or complete response received.
+* Terminate string in check_data buffer.
+*/
if (s->check_data_len < BUFSIZE)
s->check_data[s->check_data_len] = '\0';
- else
+ else {
s->check_data[s->check_data_len - 1] = '\0';
-
- /* Close the connection... */
- shutdown(fd, SHUT_RDWR);
+ done = 1; /* buffer full, don't wait for more data */
+ }
/* Run the checks... */
if (s->proxy->options & PR_O_HTTP_CHK) {
+ if (!done && s->check_data_len < strlen("HTTP/1.0 000\r"))
+ goto wait_more_data;
+
/* Check if the server speaks HTTP 1.X */
if ((s->check_data_len < strlen("HTTP/1.0 000\r")) ||
(memcmp(s->check_data, "HTTP/1.", 7) != 0 ||
@@ -955,6 +957,9 @@ static int event_srv_chk_r(int fd)
}
}
else if (s->proxy->options & PR_O_SSL3_CHK) {
+ if (!done && s->check_data_len < 5)
+ goto wait_more_data;
+
/* Check for SSLv3 alert or handshake */
if
Re: Truncated health check response from real servers
Le Mercredi 17 Mars 2010 21:12:45, Willy Tarreau a écrit : > (...) I don't > agree with restting the buffer or even considering we have an error > when a session does not close, because it is a regression. For instance, > all people using HTTP/1.1 checks will see a problem here. hey right, I suddenly realize I forgot to recheck that ! > Rather, I > think we should start the parsing as soon as the response is large > enough to be parsed, and only declare a parse error if we reach the > end of response on a closed response. That way we can still parse > status codes as soon as we get the first packet for most cases, but > we can also wait for more data. It means that for some rare cases, > we can parse the response multiple times waiting for it to be complete, > but that's rather rare and right now those cases don't work at all. > > I'll see what can be done. That looks much better, no need to wait for a complete response if only the status is checked :) And the resource will be released faster in most of the cases. -- Cyril Bonté
Re: Truncated health check response from real servers
Hi Cyril, On Wed, Mar 17, 2010 at 06:56:31PM +0100, Cyril Bonté wrote: > Hi all, > > Le Mardi 16 Mars 2010 21:35:10, Willy Tarreau a écrit : > > I'm now gathering my changes and committing your patch with the small > > fixes above. That way we can concentrate on ECV. > > I've played a little with this commits. > I added some traces to see what happens when a timeout occurs and noticed > that the response is concatenated after the response of the previous timed > out check, until the buffer is full. > This can occur when the server started to send a response but was too long to > finish or if a check uses HTTP/1.1 on a server with a HTTP Keep-Alive longer > than the check timeout (adding "Connection: close" could help prevent this, > maybe it should be added in the "option httpchk" documentation). You make a good point. The initial code was not sensible to such connections that were left open, and it used to parse the response upon the first packet. I don't agree with restting the buffer or even considering we have an error when a session does not close, because it is a regression. For instance, all people using HTTP/1.1 checks will see a problem here. Rather, I think we should start the parsing as soon as the response is large enough to be parsed, and only declare a parse error if we reach the end of response on a closed response. That way we can still parse status codes as soon as we get the first packet for most cases, but we can also wait for more data. It means that for some rare cases, we can parse the response multiple times waiting for it to be complete, but that's rather rare and right now those cases don't work at all. I'll see what can be done. (...) > Also, to save a little memory, the check_data buffer is only allocated for > the servers that are checked. Ah, that's a good point, thanks :-) Regards, Willy
Re: Truncated health check response from real servers
Hi all,
Le Mardi 16 Mars 2010 21:35:10, Willy Tarreau a écrit :
> I'm now gathering my changes and committing your patch with the small
> fixes above. That way we can concentrate on ECV.
I've played a little with this commits.
I added some traces to see what happens when a timeout occurs and noticed that
the response is concatenated after the response of the previous timed out
check, until the buffer is full.
This can occur when the server started to send a response but was too long to
finish or if a check uses HTTP/1.1 on a server with a HTTP Keep-Alive longer
than the check timeout (adding "Connection: close" could help prevent this,
maybe it should be added in the "option httpchk" documentation).
The "expect" option from the ECV patch can be affected by this.
The patch provided in attachment resets the buffer after a timeout bur I wonder
if it can't be done in set_server_check_status().
Also, to save a little memory, the check_data buffer is only allocated for the
servers that are checked.
--
Cyril Bonté
diff -Naur haproxy-ss-20100317/src/cfgparse.c haproxy-ss-20100317-checks/src/cfgparse.c
--- haproxy-ss-20100317/src/cfgparse.c 2010-03-16 22:57:27.0 +0100
+++ haproxy-ss-20100317-checks/src/cfgparse.c 2010-03-17 18:37:21.0 +0100
@@ -3116,12 +3116,6 @@
newsrv->curfd = -1; /* no health-check in progress */
newsrv->health = newsrv->rise; /* up, but will fall down at first failure */
- /* Allocate buffer for partial check results... */
- if ((newsrv->check_data = calloc(BUFSIZE, sizeof(char))) == NULL) {
-Alert("parsing [%s:%d] : out of memory while allocating check buffer.\n", file, linenum);
-err_code |= ERR_ALERT | ERR_ABORT;
-goto out;
- }
cur_arg = 3;
} else {
newsrv = &curproxy->defsrv;
@@ -3563,6 +3557,13 @@
goto out;
}
+ /* Allocate buffer for partial check results... */
+ if ((newsrv->check_data = calloc(BUFSIZE, sizeof(char))) == NULL) {
+Alert("parsing [%s:%d] : out of memory while allocating check buffer.\n", file, linenum);
+err_code |= ERR_ALERT | ERR_ABORT;
+goto out;
+ }
+
newsrv->check_status = HCHK_STATUS_INI;
newsrv->state |= SRV_CHECKED;
}
diff -Naur haproxy-ss-20100317/src/checks.c haproxy-ss-20100317-checks/src/checks.c
--- haproxy-ss-20100317/src/checks.c 2010-03-16 22:57:27.0 +0100
+++ haproxy-ss-20100317-checks/src/checks.c 2010-03-17 18:43:12.0 +0100
@@ -1345,6 +1345,9 @@
else /* HTTP, SMTP */
set_server_check_status(s, HCHK_STATUS_L7TOUT, NULL);
}
+/* Reset the check buffer... */
+*s->check_data = '\0';
+s->check_data_len = 0;
}
//fprintf(stderr, "process_chk: 10\n");
diff -Naur haproxy-ss-20100317/src/haproxy.c haproxy-ss-20100317-checks/src/haproxy.c
--- haproxy-ss-20100317/src/haproxy.c 2010-03-16 22:57:27.0 +0100
+++ haproxy-ss-20100317-checks/src/haproxy.c 2010-03-17 18:38:08.0 +0100
@@ -859,7 +859,9 @@
free(s->id);
free(s->cookie);
- free(s->check_data);
+ if (s->check_data) {
+free(s->check_data);
+ }
free(s);
s = s_next;
}/* end while(s) */
Re: haproxy 1.4.1 port crashes
Hey guys, I've just checked and indeed recent FreeBSD systems now define MSG_NOSIGNAL, which explains why the SIGPIPE is not caught anymore. Maybe it does not always work as expected, but anyway the only systems where the signal was not caught was linux <= 2.4. So now I have changed this to unconditionnaly catch the signal. This should now fix the issue for you. Please use the latest snapshot (or wait to get it in the usual form tomorrow) : http://git.1wt.eu/web?p=haproxy-1.4.git;a=snapshot;sf=tgz Regards, Willy
Re: XML Output interrupt / P-FLAG
On Wed, Mar 17, 2010 at 05:10:27PM +0100, Bernhard Krieger wrote: > Hello Willi, > > > > i applied the patch on our test system and it seems to be ok. Perfect, thanks for confirming this ! > Every request was successfully. No interruptions occurred!! > > Tomorrow i install the patched version on our live system. Then better use the latest snapshot, which also contains other fixes : http://git.1wt.eu/web?p=haproxy-1.4.git;a=snapshot;sf=tgz (it will also appear in the src/snapshot directory tomorrow morning). I intend to release 1.4.2 very soon now. Either I get more info about the FreeBSD crashes to fix them and I wait for that, or I release 1.4.2 without. Regards, Willy
Re: XML Output interrupt / P-FLAG
Hello Willi, i applied the patch on our test system and it seems to be ok. Every request was successfully. No interruptions occurred!! Tomorrow i install the patched version on our live system. Thank you regards Bernhard - Nachricht von w...@1wt.eu - Datum: Wed, 17 Mar 2010 16:07:55 +0100 Von: Willy Tarreau Antwort an: Willy Tarreau Betreff: Re: XML Output interrupt / P-FLAG An: Bernhard Krieger Cc: haproxy@formilux.org Hi Bernhard, thanks very much for your traces. They made it easier for me to reproduce the issue. It happens when the chunked data are split in chunks of a size which divides the buffer size. This causes situations where the chunk parser tries to find a CRLF followed by a chunk size and due to a wrong comparison, it checks for the CRLF in data which are not yet received. It does not see it and finds something else instead, so it concludes the response is invalid. I've fixed the check to consider only unparsed data instead of the whole buffer, and it's now OK for me. Here's the patch that I've merged, in case you want to try now. Thanks again for your kind responsiveness, Willy - Ende der Nachricht von w...@1wt.eu - This message was sent using IMP, the Internet Messaging Program.
Re: haproxy 1.4.1 port crashes
Hi Angelo, On Wed, Mar 17, 2010 at 04:14:28PM +0100, Angelo Höngens wrote: > I upgraded haproxy from 1.3.2x to 1.4.1 on one of my production > balancers this morning from the ports (yes, I was feeling lucky), and > noticed after a while haproxy was stopped. Nothing in any logs (no debug > settings), it just stopped after a few minutes. I'm running FreeBSD > 7.2-RELEASE amd64 with a custom kernel (carp compiled in). Installed > 1.3.22 from haproxy-devel, that works fine.. > > No time to dig deeper now, but if you need any help let me know, perhaps > I can do some tests this week if needed. Those tests would be welcome. If you have the ability to start it in a debugger and see where it stops, it will help a lot. Thanks! Willy
Re: haproxy 1.4.1 port crashes
On 17-3-2010 16:10, Willy Tarreau wrote: > Hello, > > On Wed, Mar 17, 2010 at 09:59:41AM +0100, Pawel Jaskorzynski wrote: >> Hello, >> I would like to report instability of the recent stable branch port of >> haproxy-1.4.1. After some time of processing TCP requests in >> round-robin, haproxy dies with "broken pipe" in debug. Nothing else gets >> reported. >> My systsem is i386 FreeBSD 8.0-STABLE #1: Mon Mar 15, GENERIC kernel. >> The port has been cvsup'ed on the 16th of March. >> Previously operated haproxy-1.2.X for long time with no problems. >> Currently reverted to haproxy-1.2.18, works as expected. > > I already got one report of a crash on FreeBSD. The fact that you see > a broken pipe lets me think that MSG_NOSIGNAL is defined, causing the > sigpipe not to be caught, but it does not have the expected effect. > Could you please remove the #if/#endif around signal(SIGPIPE, SIG_IGN) > in src/haproxy.c and try again ? > > Thanks, > Willy > Me too.. I upgraded haproxy from 1.3.2x to 1.4.1 on one of my production balancers this morning from the ports (yes, I was feeling lucky), and noticed after a while haproxy was stopped. Nothing in any logs (no debug settings), it just stopped after a few minutes. I'm running FreeBSD 7.2-RELEASE amd64 with a custom kernel (carp compiled in). Installed 1.3.22 from haproxy-devel, that works fine.. No time to dig deeper now, but if you need any help let me know, perhaps I can do some tests this week if needed. -- With kind regards, Angelo Höngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist -- NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 a.hong...@netmatch.nl www.netmatch.nl --
Re: haproxy 1.4.1 port crashes
Hello, On Wed, Mar 17, 2010 at 09:59:41AM +0100, Pawel Jaskorzynski wrote: > Hello, > I would like to report instability of the recent stable branch port of > haproxy-1.4.1. After some time of processing TCP requests in > round-robin, haproxy dies with "broken pipe" in debug. Nothing else gets > reported. > My systsem is i386 FreeBSD 8.0-STABLE #1: Mon Mar 15, GENERIC kernel. > The port has been cvsup'ed on the 16th of March. > Previously operated haproxy-1.2.X for long time with no problems. > Currently reverted to haproxy-1.2.18, works as expected. I already got one report of a crash on FreeBSD. The fact that you see a broken pipe lets me think that MSG_NOSIGNAL is defined, causing the sigpipe not to be caught, but it does not have the expected effect. Could you please remove the #if/#endif around signal(SIGPIPE, SIG_IGN) in src/haproxy.c and try again ? Thanks, Willy
Re: XML Output interrupt / P-FLAG
Hi Bernhard, thanks very much for your traces. They made it easier for me to reproduce the issue. It happens when the chunked data are split in chunks of a size which divides the buffer size. This causes situations where the chunk parser tries to find a CRLF followed by a chunk size and due to a wrong comparison, it checks for the CRLF in data which are not yet received. It does not see it and finds something else instead, so it concludes the response is invalid. I've fixed the check to consider only unparsed data instead of the whole buffer, and it's now OK for me. Here's the patch that I've merged, in case you want to try now. Thanks again for your kind responsiveness, Willy >From bf3f1de5b58aa77c2a3da4e143d5a7b2f1056b53 Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Wed, 17 Mar 2010 15:54:24 +0100 Subject: [BUG] http: fix truncated responses on chunk encoding when size divides buffer size Bernhard Krieger reported truncated HTTP responses in presence of some specific chunk-encoded data, and kindly offered complete traces of the issue which made it easy to reproduce it. Those traces showed that the chunks were of exactly 8192 bytes, chunk size and CRLF included, which was exactly half the size of the buffer. In this situation, the function http_chunk_skip_crlf() could erroneously try to parse a CRLF after the chunk believing there were more data pending, because the number of bytes present in the buffer was considered instead of the number of remaining bytes to be parsed. --- src/proto_http.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/proto_http.c b/src/proto_http.c index f1ec7cd..694e98d 100644 --- a/src/proto_http.c +++ b/src/proto_http.c @@ -2245,7 +2245,7 @@ int http_skip_chunk_crlf(struct buffer *buf, struct http_msg *msg) ptr = buf->data; } - if (buf->l < bytes) + if (bytes > buf->l - buf->send_max) return 0; if (*ptr != '\n') -- 1.5.3.3
haproxy 1.4.1 port crashes
Hello, I would like to report instability of the recent stable branch port of haproxy-1.4.1. After some time of processing TCP requests in round-robin, haproxy dies with "broken pipe" in debug. Nothing else gets reported. My systsem is i386 FreeBSD 8.0-STABLE #1: Mon Mar 15, GENERIC kernel. The port has been cvsup'ed on the 16th of March. Previously operated haproxy-1.2.X for long time with no problems. Currently reverted to haproxy-1.2.18, works as expected. Best Regards, Pawel
Re: dynamic referer protecion
Hi Anze, There is also the architecture guide, it is a little outdated but it remains interesting : http://haproxy.1wt.eu/download/1.3/doc/architecture.txt Hervé. On 03/17/2010 08:17 AM, Anze wrote: > Hello Hervé, > > Thanks for the link, it reminded me to read it again and refresh my memory. > :) > > The manual is a really good start to using haproxy, but it doesn't cover the > topics that Willy wrote in previous mail (maxconn, setting timeouts, > tarpit,...). I was thinking more about "what to do if..." kind of guide / > book > (depends on volume and the will to deal with publishers). > > Thanks, > > Anze > > > > > On Tuesday 16 March 2010, Hervé COMMOWICK wrote: > >> Hello Anze, >> >> Willy already wrote this a while ago : >> >> http://1wt.eu/articles/2006_lb/ >> >> Hervé >> >> On 03/16/2010 12:39 PM, Anze wrote: >> >>> Willy: thanks for the explanation on tactics below! Ever thought of >>> writing a book on the subject? I don't mean haproxy configuration but its >>> possible deployments, common setups,... >>> >> > > -- Hervé COMMOWICK, EXOSEC (http://www.exosec.fr/) ZAC des Metz - 3 Rue du petit robinson - 78350 JOUY EN JOSAS Tel: +33 1 30 67 60 65 - Fax: +33 1 75 43 40 70 mailto:hcommow...@exosec.fr
Re: dynamic referer protecion
Hello Hervé, Thanks for the link, it reminded me to read it again and refresh my memory. :) The manual is a really good start to using haproxy, but it doesn't cover the topics that Willy wrote in previous mail (maxconn, setting timeouts, tarpit,...). I was thinking more about "what to do if..." kind of guide / book (depends on volume and the will to deal with publishers). Thanks, Anze On Tuesday 16 March 2010, Hervé COMMOWICK wrote: > Hello Anze, > > Willy already wrote this a while ago : > > http://1wt.eu/articles/2006_lb/ > > Hervé > > On 03/16/2010 12:39 PM, Anze wrote: > > Willy: thanks for the explanation on tactics below! Ever thought of > > writing a book on the subject? I don't mean haproxy configuration but its > > possible deployments, common setups,... >
