Re: is it possible to disable option httpchk per backend?
Le 25/03/2021 à 17:53, Mariusz Gronczewski a écrit : Hi, is it possible to disable "option httpchk" in specific backend when it is enabled in defaults block? I have config where basically every backend sans one is http so I'd like to keep that in defaults and just disable it in tcp backend (which is backend for SPOE/A) but it seems to be one of very few options that do not have "no option httpchk". Hi, Indeed, you can't. But you may override it with the right health-check type. For instance "option tcp-check". Or better "option spop-check" if it is a SPOE backend. -- Christopher Faulet
is it possible to disable option httpchk per backend?
Hi, is it possible to disable "option httpchk" in specific backend when it is enabled in defaults block? I have config where basically every backend sans one is http so I'd like to keep that in defaults and just disable it in tcp backend (which is backend for SPOE/A) but it seems to be one of very few options that do not have "no option httpchk". Cheers Mariusz -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: ad...@efigence.com
Re: HTTP/2 to backend server fails health check when 'option httpchk' set
On Sun, Dec 16, 2018 at 12:01:18PM +0100, Baptiste wrote: > I don't know H2 well, but can't we forge an HTTP/2 query using tcp-check > script? Yes you can but you can't predict the result :-) Willy
Re: HTTP/2 to backend server fails health check when 'option httpchk' set
On Sun, Dec 16, 2018 at 12:20 AM Willy Tarreau wrote: > On Sat, Dec 15, 2018 at 11:11:58PM +, Nick Ramirez wrote: > > Thanks! That points me in the right direction. I found that to enable > Layer > > 7 health checks in this case, I would open another port on the web server > > that does not advertise HTTP/2 support (ALPN HTTP/1.1) or does not use > TLS > > (which also turns off HTTP/2 in the case of the Caddy web server), and > then > > use the "port" parameter on the server line to point to that port. > > > > backend webservers > > balance roundrobin > > option httpchk HEAD / > > server server1 web:443 ssl verify none alpn h2,http/1.1 check port > 80 > > > > Layer 7 health checks back up and running. :-) > > Yes definitely, if you have clear-text there it's the way to do it. > Otherwise you can do it in H1 over TLS since your server is supposed > to serve H1 if no ALPN is negociated, but it really depends how both > sides agree on this. And I would not be surprised if checks run over > H1/TLS force a new handshake to happen for regular traffic since a > single session key can be stored per server. > > Willy > > Hi, I don't know H2 well, but can't we forge an HTTP/2 query using tcp-check script? Baptiste
Re: HTTP/2 to backend server fails health check when 'option httpchk' set
On Sat, Dec 15, 2018 at 11:11:58PM +, Nick Ramirez wrote: > Thanks! That points me in the right direction. I found that to enable Layer > 7 health checks in this case, I would open another port on the web server > that does not advertise HTTP/2 support (ALPN HTTP/1.1) or does not use TLS > (which also turns off HTTP/2 in the case of the Caddy web server), and then > use the "port" parameter on the server line to point to that port. > > backend webservers > balance roundrobin > option httpchk HEAD / > server server1 web:443 ssl verify none alpn h2,http/1.1 check port 80 > > Layer 7 health checks back up and running. :-) Yes definitely, if you have clear-text there it's the way to do it. Otherwise you can do it in H1 over TLS since your server is supposed to serve H1 if no ALPN is negociated, but it really depends how both sides agree on this. And I would not be surprised if checks run over H1/TLS force a new handshake to happen for regular traffic since a single session key can be stored per server. Willy
Re[2]: HTTP/2 to backend server fails health check when 'option httpchk' set
Thanks! That points me in the right direction. I found that to enable Layer 7 health checks in this case, I would open another port on the web server that does not advertise HTTP/2 support (ALPN HTTP/1.1) or does not use TLS (which also turns off HTTP/2 in the case of the Caddy web server), and then use the "port" parameter on the server line to point to that port. backend webservers balance roundrobin option httpchk HEAD / server server1 web:443 ssl verify none alpn h2,http/1.1 check port 80 Layer 7 health checks back up and running. :-) -- Original Message -- From: "Willy Tarreau" To: "Nick Ramirez" Cc: haproxy@formilux.org Sent: 12/15/2018 10:25:42 AM Subject: Re: HTTP/2 to backend server fails health check when 'option httpchk' set Hi Nick, On Fri, Dec 14, 2018 at 10:43:04PM +, Nick Ramirez wrote: This may be something very simple that I am missing. I am using the latest HAProxy Docker image, which is using HAProxy 1.9-dev10 2018/12/08. It is using HTTP/2 to the backend web server (Caddy). It fails its health check if I uncomment the "option httpchk" line: backend webservers balance roundrobin #option httpchk server server1 web:443 check ssl verify none alpn h2 With that line commented out, it works. The project is on Github: https://github.com/NickMRamirez/experiment-haproxy-http2 Am I doing something wrong? It also works if I remove "option http-use-htx" and "alpn h2" and uncomment "option httpchk". You're not really doing anything wrong, it's just the current limitation of health checks that we've wanted to redesing for years and that deserve a year worth of work. Currently health checks are only made of a TCP string sent over the socket and checked in return. Since 1.6 or so, we introduced the ability to send this string over SSL (when "check-ssl" is set) but that's basically the limit. In fact, health checks are completely separate from the traffic. You can see them as being part of the control plane while the traffic is the data plane. You're not even forced to send them to the same IP, ports, nor protocol as your traffic. They only pre-set the same target IP and port for convenience, but that's all. I've thought we could at least implement an H2 preface+settings check but this would provide a very low value for quite some hassle to make it work for the user, so I think it would only steer the efforts away from a real redesign of better HTTP checks. However we should at the very least document this limitation more clearly for 1.9, as chances are that a number of people will want to try this :-/ Thanks, Willy
Re: HTTP/2 to backend server fails health check when 'option httpchk' set
Hi Nick, On Fri, Dec 14, 2018 at 10:43:04PM +, Nick Ramirez wrote: > This may be something very simple that I am missing. I am using the latest > HAProxy Docker image, which is using HAProxy 1.9-dev10 2018/12/08. It is > using HTTP/2 to the backend web server (Caddy). > > It fails its health check if I uncomment the "option httpchk" line: > > backend webservers > balance roundrobin > #option httpchk > server server1 web:443 check ssl verify none alpn h2 > > > With that line commented out, it works. > > The project is on Github: > https://github.com/NickMRamirez/experiment-haproxy-http2 > > Am I doing something wrong? It also works if I remove "option http-use-htx" > and "alpn h2" and uncomment "option httpchk". You're not really doing anything wrong, it's just the current limitation of health checks that we've wanted to redesing for years and that deserve a year worth of work. Currently health checks are only made of a TCP string sent over the socket and checked in return. Since 1.6 or so, we introduced the ability to send this string over SSL (when "check-ssl" is set) but that's basically the limit. In fact, health checks are completely separate from the traffic. You can see them as being part of the control plane while the traffic is the data plane. You're not even forced to send them to the same IP, ports, nor protocol as your traffic. They only pre-set the same target IP and port for convenience, but that's all. I've thought we could at least implement an H2 preface+settings check but this would provide a very low value for quite some hassle to make it work for the user, so I think it would only steer the efforts away from a real redesign of better HTTP checks. However we should at the very least document this limitation more clearly for 1.9, as chances are that a number of people will want to try this :-/ Thanks, Willy
HTTP/2 to backend server fails health check when 'option httpchk' set
This may be something very simple that I am missing. I am using the latest HAProxy Docker image, which is using HAProxy 1.9-dev10 2018/12/08. It is using HTTP/2 to the backend web server (Caddy). It fails its health check if I uncomment the "option httpchk" line: backend webservers balance roundrobin #option httpchk server server1 web:443 check ssl verify none alpn h2 With that line commented out, it works. The project is on Github: https://github.com/NickMRamirez/experiment-haproxy-http2 Am I doing something wrong? It also works if I remove "option http-use-htx" and "alpn h2" and uncomment "option httpchk". Thanks, Nick Ramirez
Re: Mix option httpchk and ssl-hello-chk
On Fri, Sep 26, 2014 at 04:37:17PM +0200, Kevin COUSIN wrote: Here is my conf : backend bk_OWAP-SSL timeout server 30s timeout connect 5s mode http balance roundrobin option forwardfor #option ssl-hello-chk option httplog #option httpchk #hash-type consistent server PP-OWAP01001 172.21.13.79:443 ssl weight 1 check verify none I just want to check the HTTP response (200 OK or 302) of the backend server. So that's simple, just remove option ssl-hello-chk and enable option httpchk instead. It will enable HTTP checks to your server, and since your server is declared with ssl, the check will be done over SSL. Regards, Willy
Re: Mix option httpchk and ssl-hello-chk
Le 22/09/2014 15:44, Baptiste a écrit : On Mon, Sep 22, 2014 at 3:33 PM, Kevin COUSIN ki...@kiven.fr wrote: Hi list, Can I mix the option httpchk and ssl-hello-chk to check the health of an HTTPS website ? Thanks a lot Kevin C. Hi Kevin, No, you can't. It would be easier to answer you with your backend configuration! That said, you can have a look at the check-ssl option, which may help. Baptiste Hi Baptiste, Here is my conf : backend bk_OWAP-SSL timeout server 30s timeout connect 5s mode http balance roundrobin option forwardfor #option ssl-hello-chk option httplog #option httpchk #hash-type consistent server PP-OWAP01001 172.21.13.79:443 ssl weight 1 check verify none I just want to check the HTTP response (200 OK or 302) of the backend server. Regards, Kevin C
Mix option httpchk and ssl-hello-chk
Hi list, Can I mix the option httpchk and ssl-hello-chk to check the health of an HTTPS website ? Thanks a lot Kevin C.
Re: Mix option httpchk and ssl-hello-chk
On Mon, Sep 22, 2014 at 3:33 PM, Kevin COUSIN ki...@kiven.fr wrote: Hi list, Can I mix the option httpchk and ssl-hello-chk to check the health of an HTTPS website ? Thanks a lot Kevin C. Hi Kevin, No, you can't. It would be easier to answer you with your backend configuration! That said, you can have a look at the check-ssl option, which may help. Baptiste
option httpchk
Hello, I have a question to configure health checks in haproxy default I´m not configure option httpchk in my haproxy configuration. webserver1 and webserver2 ist answering. (roundrobin balance) webserver1 is down (service httpd stop) and webserver2 is answering without any problems. GREAT! is it right that haproxy checking without any option only tcp connection? Now I want that haproxy says webserver1 is only healthy if I can read url /index.php and not /index.html for example frontend http bind 192.168.36.59:80 mode http option httpclose default_backend default backend default option httpchk /index.php server web01 192.168.36.57:80 check server web02 192.168.36.58:80 check Now in my opinion http://192.168.36.59/index.html is wrong and no server is showing the page. But I can see the page. How can I configure a health check for really testing the site like only http 200 ok is showing the page or only /index.php is correct /index.html it´s not ok Greetings
AW: option httpchk
thanks for your answers It works -Ursprüngliche Nachricht- Von: Baptiste [mailto:bed...@gmail.com] Gesendet: Donnerstag, 21. März 2013 15:19 An: Wolfgang Routschka Betreff: Re: option httpchk Hi, You must provide a method in your HTTP request: GET or HEAD, usually. Baptiste On Thu, Mar 21, 2013 at 2:53 PM, Wolfgang Routschka wolfgang.routsc...@drumedar.de wrote: Hello, I have a question to configure health checks in haproxy default I´m not configure option httpchk in my haproxy configuration. webserver1 and webserver2 ist answering. (roundrobin balance) webserver1 is down (service httpd stop) and webserver2 is answering without any problems. GREAT! is it right that haproxy checking without any option only tcp connection? Now I want that haproxy says webserver1 is only healthy if I can read url /index.php and not /index.html for example frontend http bind 192.168.36.59:80 mode http option httpclose default_backend default backend default option httpchk /index.php server web01 192.168.36.57:80 check server web02 192.168.36.58:80 check Now in my opinion http://192.168.36.59/index.html is wrong and no server is showing the page. But I can see the page. How can I configure a health check for really testing the site like only http 200 ok is showing the page or only /index.php is correct /index.html it´s not ok Greetings
Re: help with option httpchk - http-check expect
On 22 November 2012 22:14, Owen Marinas omari...@woozworld.com wrote:
option httpchk POST /db/data/ext/feed/graphdb/userFeed
HTTP/1.1\r\nContent-Type: application/json\r\nContent-Length:
35\r\n{userId:8, offset:0, limit:1}\r\n
It might not be related to your original question, but I think you're
missing an extra \r\n that should represent the blank line between
the POST's headers and its body.
Also, I think the length of 35 might be wrong, as it doesn't take into
account the last \r\n (which might be superfluous anyway).
HTH,
Jonathan
--
Jonathan Matthews // Oxford, London, UK
http://www.jpluscplusm.com/contact.html
Re: help with option httpchk - http-check expect
Thx for the advice Jonathan
Willy's advice from an old post was to make it work with printf+nc in
bash first. So I did.
the issue is after I added the lines to the backend(below), the server
still resported UP eben if the expected string is not there.
regards
Owen
---
backend neo4j-stg
option httpchk POST /db/data/ext/feed/graphdb/userFeed
HTTP/1.1\r\nHost: 172.23.10.61:7474\r\nAccept: */*\r\nContent-Type:
application/json\r\nContent-Length: 35\r\n\r\n{userId:8, offset:0,
limit:1}
http-check disable-on-404 expect string userId
server neo4j-stg-01 172.23.10.61:7474
server neo4j-stg-02 172.23.10.62:7474
balance roundrobin
On 12-11-23 05:01 AM, Jonathan Matthews wrote:
On 22 November 2012 22:14, Owen Marinas omari...@woozworld.com wrote:
option httpchk POST /db/data/ext/feed/graphdb/userFeed
HTTP/1.1\r\nContent-Type: application/json\r\nContent-Length:
35\r\n{userId:8, offset:0, limit:1}\r\n
It might not be related to your original question, but I think you're
missing an extra \r\n that should represent the blank line between
the POST's headers and its body.
Also, I think the length of 35 might be wrong, as it doesn't take into
account the last \r\n (which might be superfluous anyway).
HTH,
Jonathan
Re: help with option httpchk - http-check expect
On 23 November 2012 17:10, Owen Marinas omari...@woozworld.com wrote: Thx for the advice Jonathan Willy's advice from an old post was to make it work with printf+nc in bash first. So I did. I think your back-end may be being lenient, then :-) the issue is after I added the lines to the backend(below), the server still resported UP eben if the expected string is not there. That's initially because your server lines lack the requisite check flag to enable health checks. The UP represents layer-3 connectivity to the host working. I can't speak as for any problems you see after you fix that - I've never seen POSTs and request bodies being used in a health check before! HTH, Jonathan -- Jonathan Matthews // Oxford, London, UK http://www.jpluscplusm.com/contact.html
Re: help with option httpchk - http-check expect
I hate myself for this my production LB is running haproxy-1.4.15-1 but the Staging haproxy-1.4.8-1 after upgrade its all working now, the POST, and http-check expect works fine. thx all Owen On 12-11-23 12:50 PM, Jonathan Matthews wrote: On 23 November 2012 17:10, Owen Marinas omari...@woozworld.com wrote: Thx for the advice Jonathan Willy's advice from an old post was to make it work with printf+nc in bash first. So I did. I think your back-end may be being lenient, then :-) the issue is after I added the lines to the backend(below), the server still resported UP eben if the expected string is not there. That's initially because your server lines lack the requisite check flag to enable health checks. The UP represents layer-3 connectivity to the host working. I can't speak as for any problems you see after you fix that - I've never seen POSTs and request bodies being used in a health check before! HTH, Jonathan
Re: help with option httpchk - http-check expect
On Fri, Nov 23, 2012 at 01:28:38PM -0500, Owen Marinas wrote: I hate myself for this my production LB is running haproxy-1.4.15-1 but the Staging haproxy-1.4.8-1 after upgrade its all working now, the POST, and http-check expect works fine. Good reason indeed. BTW, keep in mind that even 1.4.15 is quite outdated, there were something like 75 bugs fixed since! Willy
option httpchk
Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
Hi, What a pity, this could be very useful! Indeed, as Haproxy detects that there is no response, it may perform an another action :-) Christophe Le 31/10/11 13:27, « Baptiste » bed...@gmail.com a écrit : Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
euh, if there is no response, HAProxy can log it Then, you can then detect it and take the decision you want :) Don't ask HAProxy to reload your webservices, it's a bad idea. On Mon, Oct 31, 2011 at 1:48 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, What a pity, this could be very useful! Indeed, as Haproxy detects that there is no response, it may perform an another action :-) Christophe Le 31/10/11 13:27, « Baptiste » bed...@gmail.com a écrit : Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
I'm agree with you but how can I know that HAProxy detects an error ? Christophe Le 31/10/11 15:30, « Baptiste » bed...@gmail.com a écrit : euh, if there is no response, HAProxy can log it Then, you can then detect it and take the decision you want :) Don't ask HAProxy to reload your webservices, it's a bad idea. On Mon, Oct 31, 2011 at 1:48 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, What a pity, this could be very useful! Indeed, as Haproxy detects that there is no response, it may perform an another action :-) Christophe Le 31/10/11 13:27, « Baptiste » bed...@gmail.com a écrit : Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
Hi, On 31.10.2011 13:48, Christophe Rahier wrote: Hi, What a pity, this could be very useful! Indeed, as Haproxy detects that there is no response, it may perform an another action :-) How about a backup setup?! http://haproxy.1wt.eu/download/1.4/doc/configuration.txt 5. Server and default-server options ### ... backup When backup is present on a server line, the server is only used in load balancing when all other non-backup servers are unavailable. Requests coming with a persistence cookie referencing the server will always be served though. By default, only the first operational backup server is used, unless the allbackups option is set in the backend. See also the allbackups option. Supported in default-server: No ... ### Cheers Aleks Christophe Le 31/10/11 13:27, « Baptiste » bed...@gmail.com a écrit : Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: option httpchk
Hi, Oups, sorry, I didn't see it! Too many things to do in the same time and of course not enough time, I'm lost :-) Christophe Le 31/10/11 15:36, « Aleksandar Lazic » al-hapr...@none.at a écrit : Hi, On 31.10.2011 13:48, Christophe Rahier wrote: Hi, What a pity, this could be very useful! Indeed, as Haproxy detects that there is no response, it may perform an another action :-) How about a backup setup?! http://haproxy.1wt.eu/download/1.4/doc/configuration.txt 5. Server and default-server options ### ... backup When backup is present on a server line, the server is only used in load balancing when all other non-backup servers are unavailable. Requests coming with a persistence cookie referencing the server will always be served though. By default, only the first operational backup server is used, unless the allbackups option is set in the backend. See also the allbackups option. Supported in default-server: No ... ### Cheers Aleks Christophe Le 31/10/11 13:27, « Baptiste » bed...@gmail.com a écrit : Hi, no :) cheers On Mon, Oct 31, 2011 at 12:15 PM, Christophe Rahier christo...@qualifio.com wrote: Hi, In my config file, I check my servers with option httpchk HEAD /checkCF.cfm HTTP/1.0 When the response is not 2xx or 3xx, would it possible to test an other url? Thanks for your help. Regards, Christophe
Re: maxconn vs. option httpchk
Hi Bryan, On Wed, Mar 23, 2011 at 09:27:01PM +, Cassidy, Bryan wrote: Hi all, I've noticed an odd (lack of) interaction between maxconn and option httpchk... If a server's maxconn limit has been reached, it appears that HTTP health checks are still dispatched. If I've configured the maxconn limit to match the number of requests the backend server can concurrently dispatch, and all these connections are busy with slow requests, HAProxy will assume the server is down; once the server completes a request, HAProxy waits until rise health checks have succeeded (as expected if the server was really down, but it was only busy). This makes overly busy times even worse. Yes, that's a known situation. Minconn should always leave some room for health checks. When you have two haproxies, you might have to leave at least 2 connections for the health checks. In practice, 1 should be OK because they're supposed to be fast and it generally is not an issue if one waits a little bit to get a connection slot. This issue is sometimes encountered on mongrel servers where only one connection at a time is possible. The usual workaround for this case is to set a check timeout larger than what you consider a long request should be. Even if that can sound frustrating at first, you have to realize that if the server is failing to respond to health checks, there is no way to know whether it's too much busy or if it's dead. So there's nothing wrong with the current approach. If you pointed your browser to the server, you'd observe the same behaviour. If you think that you'd tell the difference because you'd wait longer, then it means you should adjust your check timeout. (...) I know I can work around this by setting maxconn to one less than a server's maximum capacity (perhaps this would be a good idea for other reasons). Yes that's the way to do it, and it will permit you to connect to the server without passing through haproxy. I suspect I could work around this by using TCP status checks instead of HTTP status checks, though I haven't tried this as I like the flexibility HTTP health checks offer (like disable-on-404). You're right, but relying on TCP only will also not tell you when your servers are really dead if they're just frozen. Is this behavior a bug or a feature? Intuitively I would have expected the HTTP health checks to respect maxconn limits, but perhaps there was a conscious decision to not do so (for instance, maybe it was considered unacceptable for a server's health to be unknown when it is fully loaded). We have a task on the TODO list to make health checks pass through the queue and respect the maxconn too. This is especially important for mongrel. But still, doing so does not cover the situation where you have multiple LBs or when you need to check the server by yourself. Regards, Willy
maxconn vs. option httpchk
Hi all, I've noticed an odd (lack of) interaction between maxconn and option httpchk... If a server's maxconn limit has been reached, it appears that HTTP health checks are still dispatched. If I've configured the maxconn limit to match the number of requests the backend server can concurrently dispatch, and all these connections are busy with slow requests, HAProxy will assume the server is down; once the server completes a request, HAProxy waits until rise health checks have succeeded (as expected if the server was really down, but it was only busy). This makes overly busy times even worse. I'm not sure if this explanation is clear; perhaps a concrete configuration might help. listen load_balancer bind :80 mode http balance leastconn option httpchk HEAD /healthchk http-check disable-on-404 default-server port 8080 inter 2s rise 2 fall 1 maxconn 3 server srv1 srv1.example.com:8080 check server srv2 srv2.example.com:8080 check With the above toy example, if each of srv1 and srv2 can only respond to 3 requests concurrently, and 6 slow requests come in (each taking more than 2 seconds), both backend servers will be considered down until up to 4 seconds in the worst case (inter 2s * rise 2) after one of the requests finishes. I know I can work around this by setting maxconn to one less than a server's maximum capacity (perhaps this would be a good idea for other reasons). I suspect I could work around this by using TCP status checks instead of HTTP status checks, though I haven't tried this as I like the flexibility HTTP health checks offer (like disable-on-404). Is this behavior a bug or a feature? Intuitively I would have expected the HTTP health checks to respect maxconn limits, but perhaps there was a conscious decision to not do so (for instance, maybe it was considered unacceptable for a server's health to be unknown when it is fully loaded). Thanks, Bryan
option httpchk and SSL
Hi, I am running haproxy 1.3.2 as a load balancer for some servers running nginx as an ssl terminator which forwards to squid running as a reverse proxy which falls back to apache. I want to use 'option httpchk' to check whether or not a specific resource is available to our users - the idea being that the health-check will actually detect if any particular piece of the webserver is down (nginx, squid or apache). However, I cannot seem to get it working properly. I've tried a variety of things along the lines of: option httpchk GET /index.php HTTP/1.1\r\nHost: foo.bar.org server foo1.private.net 10.0.0.1:443 check inter 2000 rise 5 fall 2 but nothing appears to work - all the requests show up as 400 errors in the nginx logs. My hunch is that this is because the httpchk is not even attempting to negotiate the SSL handshake, but perhaps I am just doing something wrong. Is there any way to get this to work? If not, is there any other httpchk-like thing for https? I know of ssl-hello-chk, but I want to know if I can actually pull a resource from a webserver, not just if my HTTPS service is alive. Thanks very much, Arthur Richards
Re: option httpchk and SSL
On Wed, Nov 24, 2010 at 07:24:10PM -0500, John Marrett wrote: Arthur, I believe that you will need to use stunnel in client mode to initiate the ssl connections to the back end. You'll set up multiple instances, presumably on the machine running haproxy, each confired to point to a specific backend. httpchk will not do https. Confirmed. Also, maybe it would be fine for you to send your check to nginx on port 80 in clear text ? If so, just add port 80 after check on your server lines. It will not validate the SSL part but will at least validate that the whole chain is running. And the SSL part is not much likely to fail without the rest anyway. Regards, Willy
Re: HAProxy option httpchk - Soap?
Cheers Willy, If I get it going I'll post it up for an example! Sam On Fri, Sep 17, 2010 at 5:18 PM, Willy Tarreau w...@1wt.eu wrote: Hi Sam, On Thu, Sep 16, 2010 at 05:37:57PM +1200, Sam Pipe wrote: Hi, This is a long shot but has anyone every been able to test a simple soap service using the httpchk option in haproxy?.. If so any examples around? I've tried a few over using the \r\n but had no luck.. eg.. option httpchk POST /wsdl/NAL/bp1.0 HTTP/1.1\r\nAccept-Encoding: gzip,deflate\r\nContent-Type: text/xml;charset=UTF-8\r\nSOAPAction: \r\nUser-Agent: Jakarta Commons-HttpClient/3.1\r\nHost: 10.119.37.250\r\nContent-Length: 527\r\n\r\nsoapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/; xmlns:ser= http://www.bla.co.nz/xsd/serviceprovisioning soapenv:Header/soapenv:Bodyser:queryser:providerCodebla/ser:providerCodeser:lineser:exchangeID exchangeProvider=TELser:exchangeIDAT2/ser:exchangeID/ser:exchangeIDser:mpfIDL038-07-011/ser:mpfID/ser:lineser:phoneNumber99628030/ser:phoneNumberser:queryPorttrue/ser:queryPortser:queryPhonetrue/ser:queryPhone/ser:query/soapenv:Body/soapenv:Envelope\r\n\r\n Wow, I'm not sure anyone has ever attempted to do that yet. This is a perfect candidate for an extension of the check ! One hint is that you should escape all your spaces. Also, I strongly suggest that you try with printf+netcat first from the command line, you'll save a lot of time ! Regards, Willy
HAProxy option httpchk - Soap?
Hi, This is a long shot but has anyone every been able to test a simple soap service using the httpchk option in haproxy?.. If so any examples around? I've tried a few over using the \r\n but had no luck.. eg.. option httpchk POST /wsdl/NAL/bp1.0 HTTP/1.1\r\nAccept-Encoding: gzip,deflate\r\nContent-Type: text/xml;charset=UTF-8\r\nSOAPAction: \r\nUser-Agent: Jakarta Commons-HttpClient/3.1\r\nHost: 10.119.37.250\r\nContent-Length: 527\r\n\r\nsoapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/; xmlns:ser= http://www.bla.co.nz/xsd/serviceprovisioning;soapenv:Header/soapenv:Bodyser:queryser:providerCodebla/ser:providerCodeser:lineser:exchangeID exchangeProvider=TELser:exchangeIDAT2/ser:exchangeID/ser:exchangeIDser:mpfIDL038-07-011/ser:mpfID/ser:lineser:phoneNumber99628030/ser:phoneNumberser:queryPorttrue/ser:queryPortser:queryPhonetrue/ser:queryPhone/ser:query/soapenv:Body/soapenv:Envelope\r\n\r\n any pointers are most appricated.. Sam
option httpchk timeout
Hello, I'm interested in extending the timeout for my httpchk to be longer than 2s (to, say, 10s). I may be blind, but I wasn't able to find an option in the configuration manual here: http://haproxy.1wt.eu/download/1.4/doc/configuration.txt . Is this possible? Here's the string I'm currently using to configure httpchk: option httpchk GET /ping?profiler HTTP/1.1\r\nHost:\ haproxypinghost\r\nAccept:\ */ Thanks, Jack
Re: option httpchk timeout
Hi, Le jeudi 26 août 2010 01:22:27, Jack Stahl a écrit : Hello, I'm interested in extending the timeout for my httpchk to be longer than 2s This means you're using inter 2s on your backends, right ? (to, say, 10s). I may be blind, but I wasn't able to find an option in the configuration manual here: http://haproxy.1wt.eu/download/1.4/doc/configuration.txt . Is this possible? You can have a look on timeout check : timeout check 10s Also, read the configuration manual to see the effect of timeout connect and inter when timeout check is set. -- Cyril Bonté
Re: option httpchk version 'trick'
Hi Andrew, On Sat, Feb 27, 2010 at 03:15:25PM +1030, Andrew Commons wrote: Hi all, The ability to extend the option httpchk version argument string to dummy up a Host header is described as a 'trick' in the configuration documentation. I have found that the 'trick' can be extended to add User-Agent (HAProxy) and Accept (*/*) headers to keep ModSecurity quiet when checking an Apache server. This leads me to two questions: (1) To what level is this 'trick' supported? Is an haproxy update likely to kill it? It's supported and is unlikely to break on future versions. It's a trick because it happens due to the way the request is built, and it requires that users have a bit of HTTP knowledge to use it. (2) Is there a better way of handling something like ModSecurity that doesn't like the request generated by haproxy because it doesn't look like it has come from a browser? right now I have no other solution ! Note that in respect to question (2) I have messed around a bit with the ModeSecurity configuration and made some progress but the use of the 'trick' was far simpler! Recently the HTTP check was internally changed so that the code now supports adding headers. So we could very well have some http-check statements to add multiple headers, like this : http-check host www.mydomain.com or : http-check header Host www.mydomain.com or maybe : http-check add Host: www.mydomain.com We could even add some POST DATA if required. This needs a bit of thinking before implementing something, but I think we can do useful things now. BTW, if we do that in 1.5 before reorganizing the checks, we'll even be able to backport into 1.4-stable. Regards, Willy
RE: option httpchk version 'trick'
Hi Willy, Thanks for the comprehensive response. My HAProxy experience is measured in days and the mailing list seems a great way to get support and your contributions are always spot on. HAProxy looks like a fantastic, highly professional, piece of software. With respect to the HTTP check internals I think that what you are proposing is good. I'm still working through the implications of adding the cookies to the responses. This is not uncommon in a variety of applications but if we assume that it tells someone that HAProxy is there then does it give them any leverage. I have to read some more of the excellent documentation and play a bit. Thanks again. Cheers, Andrew BTW. Read your blog, sorry to see the VAX retired. I might still find a use for the MicroVAXs I and 2000 gathering dust in my shed :-) Great hardware and software (I'm an old VMS guy), pity about the management. -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Saturday, 27 February 2010 6:56 PM To: Andrew Commons Cc: haproxy@formilux.org Subject: Re: option httpchk version 'trick' Hi Andrew, On Sat, Feb 27, 2010 at 03:15:25PM +1030, Andrew Commons wrote: Hi all, The ability to extend the option httpchk version argument string to dummy up a Host header is described as a 'trick' in the configuration documentation. I have found that the 'trick' can be extended to add User-Agent (HAProxy) and Accept (*/*) headers to keep ModSecurity quiet when checking an Apache server. This leads me to two questions: (1) To what level is this 'trick' supported? Is an haproxy update likely to kill it? It's supported and is unlikely to break on future versions. It's a trick because it happens due to the way the request is built, and it requires that users have a bit of HTTP knowledge to use it. (2) Is there a better way of handling something like ModSecurity that doesn't like the request generated by haproxy because it doesn't look like it has come from a browser? right now I have no other solution ! Note that in respect to question (2) I have messed around a bit with the ModeSecurity configuration and made some progress but the use of the 'trick' was far simpler! Recently the HTTP check was internally changed so that the code now supports adding headers. So we could very well have some http-check statements to add multiple headers, like this : http-check host www.mydomain.com or : http-check header Host www.mydomain.com or maybe : http-check add Host: www.mydomain.com We could even add some POST DATA if required. This needs a bit of thinking before implementing something, but I think we can do useful things now. BTW, if we do that in 1.5 before reorganizing the checks, we'll even be able to backport into 1.4-stable. Regards, Willy
Re: option httpchk version 'trick'
Hi Andrew, On Sat, Feb 27, 2010 at 07:59:27PM +1030, Andrew Commons wrote: Hi Willy, Thanks for the comprehensive response. My HAProxy experience is measured in days and the mailing list seems a great way to get support and your contributions are always spot on. HAProxy looks like a fantastic, highly professional, piece of software. At least we try to make it reach that goal :-) With respect to the HTTP check internals I think that what you are proposing is good. I'm still working through the implications of adding the cookies to the responses. This is not uncommon in a variety of applications but if we assume that it tells someone that HAProxy is there then does it give them any leverage. I have to read some more of the excellent documentation and play a bit. There are many ways to detect intermediate systems. For instance, with apache, you just have to send TRACE requests with Max-Forwards. With haproxy, you can send invalid requests and try to see if the response looks like the default HTTP 400 response, etc... Of course, you could use many tricks to try to hide that, but there are always ways to detect with a great probability what components you're using. I work in environments where the security teams are extremely picky on this, but after the years, they finally make a difference between being able to identify one component and being able to abuse the rest of the infrastructure. After all, if someone can tell you're using haproxy, he also knows that you can rewrite all headers and block the requests you don't like, so he will know that he cannot trust the responses to his identification requests that pass through it. That's the most important point. To make an analogy with doorlocks, sometimes you'd rather have a solid lock with its brand engraved in it that will successfully keep thieves away than have a noname one which breaks at the first lock pick. Regards, Willy
RE: option httpchk version 'trick'
Hi Willy, I was shooting from the lip with respect to the cookies. It's really about what scenarios you need to consider. If the client can choose the server they get directed to then you have to give that some thought and make sure your application behaves in a sensible manner. This would be considered in the case of failovers anyway. If observation of a number of sessions reveals something about your server farm then, again, that needs to be considered. Mitigating these concerns is not something that HAProxy should have to deal with, session management at that level is an application problem. With respect to your lock analogy I would add that all locks can be picked given enough time. Whilst obscurity is not security it can increase the time required to pick the lock. Make your thief work for the information and they may go somewhere else or, more importantly, you increase your chances of seeing them working on the lock. This surveillance is the responsibility of the application. Once there is an understanding of these system failure modes (and it is not an HAProxy failure) then the appropriate security requirements can be formulated. Cheers, Andrew -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Saturday, 27 February 2010 8:39 PM To: Andrew Commons Cc: haproxy@formilux.org Subject: Re: option httpchk version 'trick' Hi Andrew, On Sat, Feb 27, 2010 at 07:59:27PM +1030, Andrew Commons wrote: Hi Willy, Thanks for the comprehensive response. My HAProxy experience is measured in days and the mailing list seems a great way to get support and your contributions are always spot on. HAProxy looks like a fantastic, highly professional, piece of software. At least we try to make it reach that goal :-) With respect to the HTTP check internals I think that what you are proposing is good. I'm still working through the implications of adding the cookies to the responses. This is not uncommon in a variety of applications but if we assume that it tells someone that HAProxy is there then does it give them any leverage. I have to read some more of the excellent documentation and play a bit. There are many ways to detect intermediate systems. For instance, with apache, you just have to send TRACE requests with Max-Forwards. With haproxy, you can send invalid requests and try to see if the response looks like the default HTTP 400 response, etc... Of course, you could use many tricks to try to hide that, but there are always ways to detect with a great probability what components you're using. I work in environments where the security teams are extremely picky on this, but after the years, they finally make a difference between being able to identify one component and being able to abuse the rest of the infrastructure. After all, if someone can tell you're using haproxy, he also knows that you can rewrite all headers and block the requests you don't like, so he will know that he cannot trust the responses to his identification requests that pass through it. That's the most important point. To make an analogy with doorlocks, sometimes you'd rather have a solid lock with its brand engraved in it that will successfully keep thieves away than have a noname one which breaks at the first lock pick. Regards, Willy
option httpchk version 'trick'
Hi all, The ability to extend the option httpchk version argument string to dummy up a Host header is described as a 'trick' in the configuration documentation. I have found that the 'trick' can be extended to add User-Agent (HAProxy) and Accept (*/*) headers to keep ModSecurity quiet when checking an Apache server. This leads me to two questions: (1) To what level is this 'trick' supported? Is an haproxy update likely to kill it? (2) Is there a better way of handling something like ModSecurity that doesn't like the request generated by haproxy because it doesn't look like it has come from a browser? Note that in respect to question (2) I have messed around a bit with the ModeSecurity configuration and made some progress but the use of the 'trick' was far simpler! Cheers, Andrew
Servers seen as going up and down using option httpchk
Hello, I have the following configuration for my webfarm: 7 servers, all running Centos 5. Two (LB1 and LB2) are HAproxy load balancers, Web1-5 are Apache2 and Lighttpd web servers (I tried both as a method of elimination test on the side of the web servers). All of the servers are dual xeon Dell PE1850s with 4GB of ram, or better. I am using HAproxy 1.3.15.2. The webfarm is used for adserving using OpenX. Here is the problem I am having: Once I get in the neighborhood of 1000 sessions the stats page on the active LB starts showing that the web servers are going up and down randomly (all servers even those set as backup and not getting any traffic). When I monitor the servers in real time and check on the stats page of the standby LB (the hot spare that still monitors servers) everything looks fine and all servers are running green with no problems. As a result of HAproxy thinking that servers are going up and down, clients are getting errors. The apache servers are set up in prefork mode with a ServerLimit at 2000, MaxClients at 2000, and MaxRequestsPerChild at 4000. The server load per server has never gotten to 2000, and there is plenty of CPU and Ram to spare on each machine even during heavy load. I am guessing that the issue has to do with the active HAproxy server running out of something or other and losing the ability to poll the web servers under this load. There is barely any usage of CPU or Ram on the LB itself, so I don't think its a hardware issue. Below is my HAproxy config file. Please note that because this is an ad server distribution system, I don't need the proxy to keep track of sessions or send the same user to the same web server as each request is a full self contained operation. global log 127.0.0.1 local0 notice #log 127.0.0.1 local1 notice #log loghostlocal0 info maxconn 2 #debug #quiet ulimit-n 25000 user root set because lit says ulimit-n above requires group root set because lit says ulimit-n above requires defaults log global modehttp option httplog option dontlognull retries 3 redispatch maxconn 2 contimeout 6 clitimeout 10 srvtimeout 10 listen webfarm bind :80 mode http stats enable stats auth ***:*** balance roundrobin #cookie SERVERID insert indirect nocache option httpclose option forwardfor option httpchk GET /check.txt HTTP/1.0 #server web1..com ***.***.***.***:80 weight 1 check maxconn 200 inter 1000 rise 2 fall 5 --- commented out, reserved for admin interface server web2..com ***.***.***.***:80 weight 3 check maxconn 3000 inter 1000 rise 2 fall 5 backup server web3..com ***.***.***.***:80 weight 3 check maxconn 3000 inter 1000 rise 2 fall 5 server web4..com ***.***.***.***:80 weight 3 check maxconn 3000 inter 1000 rise 2 fall 5 and so forth, same for the rest of the servers... I have tried using HTTP/1.1 for the httpchk option, but that results in all servers being shown as down in the stats. I have also tried varying the inter variable from 500 to 4000 with no change in behavior. Please let me know if you can suggest something. I am guessing some operating system variables need to be tweaked. Thank you, Michael
RE: option httpchk is reporting servers as down when they're not
Hi Willy, We now have HAProxy running over our freshly released website: http://www.infrastructurereportcard.org/ Thanks for this great piece of software and all the help! Only two connection errors in 3 connections thus far, one of which was due to me cancelling a long-running page load in the admin. Thomas Allen Web Developer, ASCE 703.295.6355 -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Monday, March 09, 2009 5:26 PM To: Allen, Thomas Cc: Jeffrey 'jf' Lim; haproxy@formilux.org Subject: Re: option httpchk is reporting servers as down when they're not Hi Thomas, On Mon, Mar 09, 2009 at 05:20:49PM -0400, Allen, Thomas wrote: Hi Willy, Hm, changing to 60s for each gave me 100% 504 errors, I removed all three. Bad idea, I know, but at least it works then. then use 6, that's the old way of doing it :-) I'm running 1.2.18 because the HAProxy homepage calls it the Latest version. Ah OK, version 1.2 did not have the time units. Well, in fact it's not exactly marked as the only latest version, it's the latest version of branch 1.2, and 1.2 is the only one not tainted by development I admit. I've removed all cookies from this IP, cleared my cache, and still it seems that only one server is being hit. But the stats page reports an equal distribution, so it's anybody's guess. What would be a simple way to log the distribution? I find it difficult to determine this even in debug mode (I'm running the proxy in daemon mode, of course). it is in the logs, you have the server's name (assuming you're logging with option httplog). Something is possible if you're playing with only once client. If the number of objects on a page is a multiple of the number of servers and you're in round-robin mode, then each time you'll fetch a page, you'll alternatively fetch objects from both servers and come back to the first one for the next click. Of course that does not happen as soon as you have at least another client. And since I saw 20 sessions on your stats after my access, I'm tempted to think that it could be related. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
Hi Thomas, On Wed, Mar 25, 2009 at 12:57:41PM -0400, Allen, Thomas wrote: Hi Willy, We now have HAProxy running over our freshly released website: http://www.infrastructurereportcard.org/ thanks for the heads up ! Thanks for this great piece of software and all the help! Only two connection errors in 3 connections thus far, one of which was due to me cancelling a long-running page load in the admin. fine ! anyway, you should expect to get some error requests due to such activities from your clients. In general, various sites report request error rates ranging from 0.1 to 0.6%, so what you observe is almost perfect :-) Cheers, Willy
RE: option httpchk is reporting servers as down when they're not
That, along with specifying HTTP1.1, did it, so thanks! What should I load into Host: ? It seems to work fine with www, but I'd prefer to use something I understand. Please keep in mind that none of this is yet associated with a domain, so www.mydomain.com would be inaccurate. Beginning very recently, I get a 504 Gateway Timeout for about 30% of all requests. What could be causing this? More importantly, I'm not convinced that HAProxy is successfully forwarding requests to both servers, although I could wrong. As you can see on the two app instances, each reports a separate internal IP to help diagnose. It appears that only SAMP1 receives requests, although both pass health checks now. Load balancer: http://174.129.240.119/ and stats (temporarily unblocked) http://174.129.240.119/status/lb SAMP1: http://174.129.251.234/ SAMP2: http://174.129.244.252/ Thanks, Thomas Allen Web Developer, ASCE 703.295.6355 -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Friday, March 06, 2009 1:39 PM To: Allen, Thomas Cc: Jeffrey 'jf' Lim; haproxy@formilux.org Subject: Re: option httpchk is reporting servers as down when they're not Hi Thomas, On Thu, Mar 05, 2009 at 08:45:20AM -0500, Allen, Thomas wrote: Hi Jeff, The thing is that if I don't include the health check, the load balancer works fine and each server receives equal distribution. I have no idea why the servers would be reported as down but still work when unchecked. It is possible that your servers expect the Host: header to be set during the checks. There's a trick to do it right now (don't forget to escape spaces) : option httpchk GET /index.php HTTP/1.0\r\nHost:\ www.mydomain.com Also, you should check the server's logs to see why it is reporting the service as down. And as a last resort, a tcpdump of the traffic between haproxy and a failed server will show you both the request and the complete error from the server. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
Hi Thomas, just replying quick, as I'm in a hurry. On Mon, Mar 09, 2009 at 04:01:29PM -0400, Allen, Thomas wrote: That, along with specifying HTTP1.1, did it, so thanks! What should I load into Host: ? It seems to work fine with www, but I'd prefer to use something I understand. Please keep in mind that none of this is yet associated with a domain, so www.mydomain.com would be inaccurate. Of course, www.mydomain.com was an example. Often web servers are fine with just www but normally you should use the same host name that your server will respond to. Sometimes you can also put the server's IP address. Some servers also accept an empty header (so just Host: and nothing else). Beginning very recently, I get a 504 Gateway Timeout for about 30% of all requests. What could be causing this? responses taking too much time. Are you sure that your timeout server is properly set ? Maybe you have put times in milliseconds there thinking they were in seconds ? More importantly, I'm not convinced that HAProxy is successfully forwarding requests to both servers, although I could wrong. As you can see on the two app instances, each reports a separate internal IP to help diagnose. It appears that only SAMP1 receives requests, although both pass health checks now. I see both servers receiving 20 sessions, so that seems fine. Among possible reasons for what you observe : - ensure you're using balance roundrobin and not any sort of hash or source-based algorithm - ensure that you have not enabled cookie stickiness, or that you close your browser before retrying. - ensure that you have option httpclose and that your browser is not simply pushing all requests in the same session tunnelled to the first server haproxy connected to. Regards, Willy
RE: option httpchk is reporting servers as down when they're not
I used the unit 'S' for my timeouts, as in clitimeout 60S contimeout 60S srvtimeout 60S Is that to be avoided? I assumed it meant seconds. I'm using roundrobin and adding the httpclose option. I've been using cookie stickiness (which will be important for this website), but after disabling this stickiness, I get the same results. I tried clearing out the server cookie before and opening the page in multiple browsers, and still got these results. Thanks, Thomas Allen Web Developer, ASCE 703.295.6355 -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Monday, March 09, 2009 4:09 PM To: Allen, Thomas Cc: Jeffrey 'jf' Lim; haproxy@formilux.org Subject: Re: option httpchk is reporting servers as down when they're not Hi Thomas, just replying quick, as I'm in a hurry. On Mon, Mar 09, 2009 at 04:01:29PM -0400, Allen, Thomas wrote: That, along with specifying HTTP1.1, did it, so thanks! What should I load into Host: ? It seems to work fine with www, but I'd prefer to use something I understand. Please keep in mind that none of this is yet associated with a domain, so www.mydomain.com would be inaccurate. Of course, www.mydomain.com was an example. Often web servers are fine with just www but normally you should use the same host name that your server will respond to. Sometimes you can also put the server's IP address. Some servers also accept an empty header (so just Host: and nothing else). Beginning very recently, I get a 504 Gateway Timeout for about 30% of all requests. What could be causing this? responses taking too much time. Are you sure that your timeout server is properly set ? Maybe you have put times in milliseconds there thinking they were in seconds ? More importantly, I'm not convinced that HAProxy is successfully forwarding requests to both servers, although I could wrong. As you can see on the two app instances, each reports a separate internal IP to help diagnose. It appears that only SAMP1 receives requests, although both pass health checks now. I see both servers receiving 20 sessions, so that seems fine. Among possible reasons for what you observe : - ensure you're using balance roundrobin and not any sort of hash or source-based algorithm - ensure that you have not enabled cookie stickiness, or that you close your browser before retrying. - ensure that you have option httpclose and that your browser is not simply pushing all requests in the same session tunnelled to the first server haproxy connected to. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
On Mon, Mar 09, 2009 at 04:15:34PM -0400, Allen, Thomas wrote: I used the unit 'S' for my timeouts, as in clitimeout 60S contimeout 60S srvtimeout 60S Is that to be avoided? I assumed it meant seconds. OK it's just a minor problem. You have to use a lower-case s : 60s. It's stupid that the parser did not catch this mistake. I should improve it. By default, it ignores unknown chars, you you clearly had 60 ms here. BTW, there's no use in setting large contimeouts. You should usually stay with lower values such as 5-10s. Oh BTW, what version are you running ? Your stats page looks old. The time units were introduced in 1.3.14, so I hope you're at least at this level. I'm using roundrobin and adding the httpclose option. I've been using cookie stickiness (which will be important for this website), but after disabling this stickiness, I get the same results. I tried clearing out the server cookie before and opening the page in multiple browsers, and still got these results. Then it is possible that haproxy could not manage to connect to your server in 60ms, then immediately retried on the other one, and sticked to that one. Regards, Willy
RE: option httpchk is reporting servers as down when they're not
Hi Willy, Hm, changing to 60s for each gave me 100% 504 errors, I removed all three. Bad idea, I know, but at least it works then. I'm running 1.2.18 because the HAProxy homepage calls it the Latest version. I've removed all cookies from this IP, cleared my cache, and still it seems that only one server is being hit. But the stats page reports an equal distribution, so it's anybody's guess. What would be a simple way to log the distribution? I find it difficult to determine this even in debug mode (I'm running the proxy in daemon mode, of course). Thanks, Thomas Allen Web Developer, ASCE 703.295.6355 -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Monday, March 09, 2009 4:58 PM To: Allen, Thomas Cc: Jeffrey 'jf' Lim; haproxy@formilux.org Subject: Re: option httpchk is reporting servers as down when they're not On Mon, Mar 09, 2009 at 04:15:34PM -0400, Allen, Thomas wrote: I used the unit 'S' for my timeouts, as in clitimeout 60S contimeout 60S srvtimeout 60S Is that to be avoided? I assumed it meant seconds. OK it's just a minor problem. You have to use a lower-case s : 60s. It's stupid that the parser did not catch this mistake. I should improve it. By default, it ignores unknown chars, you you clearly had 60 ms here. BTW, there's no use in setting large contimeouts. You should usually stay with lower values such as 5-10s. Oh BTW, what version are you running ? Your stats page looks old. The time units were introduced in 1.3.14, so I hope you're at least at this level. I'm using roundrobin and adding the httpclose option. I've been using cookie stickiness (which will be important for this website), but after disabling this stickiness, I get the same results. I tried clearing out the server cookie before and opening the page in multiple browsers, and still got these results. Then it is possible that haproxy could not manage to connect to your server in 60ms, then immediately retried on the other one, and sticked to that one. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
Hi Thomas, On Mon, Mar 09, 2009 at 05:20:49PM -0400, Allen, Thomas wrote: Hi Willy, Hm, changing to 60s for each gave me 100% 504 errors, I removed all three. Bad idea, I know, but at least it works then. then use 6, that's the old way of doing it :-) I'm running 1.2.18 because the HAProxy homepage calls it the Latest version. Ah OK, version 1.2 did not have the time units. Well, in fact it's not exactly marked as the only latest version, it's the latest version of branch 1.2, and 1.2 is the only one not tainted by development I admit. I've removed all cookies from this IP, cleared my cache, and still it seems that only one server is being hit. But the stats page reports an equal distribution, so it's anybody's guess. What would be a simple way to log the distribution? I find it difficult to determine this even in debug mode (I'm running the proxy in daemon mode, of course). it is in the logs, you have the server's name (assuming you're logging with option httplog). Something is possible if you're playing with only once client. If the number of objects on a page is a multiple of the number of servers and you're in round-robin mode, then each time you'll fetch a page, you'll alternatively fetch objects from both servers and come back to the first one for the next click. Of course that does not happen as soon as you have at least another client. And since I saw 20 sessions on your stats after my access, I'm tempted to think that it could be related. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
On Sat, Mar 7, 2009 at 2:38 AM, Willy Tarreau w...@1wt.eu wrote: Hi Thomas, On Thu, Mar 05, 2009 at 08:45:20AM -0500, Allen, Thomas wrote: Hi Jeff, The thing is that if I don't include the health check, the load balancer works fine and each server receives equal distribution. I have no idea why the servers would be reported as down but still work when unchecked. It is possible that your servers expect the Host: header to be set during the checks. There's a trick to do it right now (don't forget to escape spaces) : option httpchk GET /index.php HTTP/1.0\r\nHost:\ www.mydomain.com you know Thomas, Willy may be very right here. And I just realized as well - u say u're using 'option httpchk /index.php'? - without specifying the 'GET' verb? -jf Also, you should check the server's logs to see why it is reporting the service as down. And as a last resort, a tcpdump of the traffic between haproxy and a failed server will show you both the request and the complete error from the server. Regards, Willy
Re: option httpchk is reporting servers as down when they're not
Hi Thomas, On Thu, Mar 05, 2009 at 08:45:20AM -0500, Allen, Thomas wrote: Hi Jeff, The thing is that if I don't include the health check, the load balancer works fine and each server receives equal distribution. I have no idea why the servers would be reported as down but still work when unchecked. It is possible that your servers expect the Host: header to be set during the checks. There's a trick to do it right now (don't forget to escape spaces) : option httpchk GET /index.php HTTP/1.0\r\nHost:\ www.mydomain.com Also, you should check the server's logs to see why it is reporting the service as down. And as a last resort, a tcpdump of the traffic between haproxy and a failed server will show you both the request and the complete error from the server. Regards, Willy
RE: option httpchk is reporting servers as down when they're not
Thanks, once I figure out logging I'll let you guys know what I discover :^) Thomas Allen Web Developer, ASCE 703.295.6355 -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Friday, March 06, 2009 1:39 PM To: Allen, Thomas Cc: Jeffrey 'jf' Lim; haproxy@formilux.org Subject: Re: option httpchk is reporting servers as down when they're not Hi Thomas, On Thu, Mar 05, 2009 at 08:45:20AM -0500, Allen, Thomas wrote: Hi Jeff, The thing is that if I don't include the health check, the load balancer works fine and each server receives equal distribution. I have no idea why the servers would be reported as down but still work when unchecked. It is possible that your servers expect the Host: header to be set during the checks. There's a trick to do it right now (don't forget to escape spaces) : option httpchk GET /index.php HTTP/1.0\r\nHost:\ www.mydomain.com Also, you should check the server's logs to see why it is reporting the service as down. And as a last resort, a tcpdump of the traffic between haproxy and a failed server will show you both the request and the complete error from the server. Regards, Willy
RE: option httpchk is reporting servers as down when they're not
Hi Jeff,
The thing is that if I don't include the health check, the load balancer works
fine and each server receives equal distribution. I have no idea why the
servers would be reported as down but still work when unchecked.
Thanks,
Thomas Allen
Web Developer, ASCE
703.295.6355
-Original Message-
From: Jeffrey 'jf' Lim [mailto:jfs.wo...@gmail.com]
Sent: Wednesday, March 04, 2009 8:11 PM
To: Allen, Thomas
Cc: haproxy@formilux.org
Subject: Re: option httpchk is reporting servers as down when they're not
well, looks like ur servers are actually down then. Do a curl from
your haproxy machine to both servers. What do you get?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
On Wed, Mar 4, 2009 at 9:40 PM, Allen, Thomas tal...@asce.org wrote:
Never mind, I got it going. My stats page simply says that both servers are
down. What else should I be looking for?
Thomas Allen
Web Developer, ASCE
703.295.6355
-Original Message-
From: Jeffrey 'jf' Lim [mailto:jfs.wo...@gmail.com]
Sent: Wednesday, March 04, 2009 2:22 AM
To: Allen, Thomas
Cc: haproxy@formilux.org
Subject: Re: option httpchk is reporting servers as down when they're not
- Show quoted text -
On Wed, Mar 4, 2009 at 4:05 AM, Allen, Thomas tal...@asce.org wrote:
Hi,
I like the idea of having HAProxy check server health, but for some reason,
it reports all of my servers as down. Here's my full config:
listen http_proxy :80
mode http
balance roundrobin
option httpchk
server webA {IP} cookie A check
server webB {IP} cookie B check
I tried option httpchk /index.php just to be sure, and got the same
result. If I remove the httpchk option, HAProxy has no problem proxying
these servers. What am I doing wrong?
what's listed under Status for these servers when viewing your
haproxy status page?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not
help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
RE: option httpchk is reporting servers as down when they're not
What's a status page? I don't recall anything in the docs about where this is
located or how to configure it, but I probably glossed over something.
Thanks,
Thomas Allen
Web Developer, ASCE
703.295.6355
-Original Message-
From: Jeffrey 'jf' Lim [mailto:jfs.wo...@gmail.com]
Sent: Wednesday, March 04, 2009 2:22 AM
To: Allen, Thomas
Cc: haproxy@formilux.org
Subject: Re: option httpchk is reporting servers as down when they're not
On Wed, Mar 4, 2009 at 4:05 AM, Allen, Thomas tal...@asce.org wrote:
Hi,
I like the idea of having HAProxy check server health, but for some reason,
it reports all of my servers as down. Here's my full config:
listen http_proxy :80
mode http
balance roundrobin
option httpchk
server webA {IP} cookie A check
server webB {IP} cookie B check
I tried option httpchk /index.php just to be sure, and got the same
result. If I remove the httpchk option, HAProxy has no problem proxying
these servers. What am I doing wrong?
what's listed under Status for these servers when viewing your
haproxy status page?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
RE: option httpchk is reporting servers as down when they're not
Never mind, I got it going. My stats page simply says that both servers are
down. What else should I be looking for?
Thomas Allen
Web Developer, ASCE
703.295.6355
-Original Message-
From: Jeffrey 'jf' Lim [mailto:jfs.wo...@gmail.com]
Sent: Wednesday, March 04, 2009 2:22 AM
To: Allen, Thomas
Cc: haproxy@formilux.org
Subject: Re: option httpchk is reporting servers as down when they're not
On Wed, Mar 4, 2009 at 4:05 AM, Allen, Thomas tal...@asce.org wrote:
Hi,
I like the idea of having HAProxy check server health, but for some reason,
it reports all of my servers as down. Here's my full config:
listen http_proxy :80
mode http
balance roundrobin
option httpchk
server webA {IP} cookie A check
server webB {IP} cookie B check
I tried option httpchk /index.php just to be sure, and got the same
result. If I remove the httpchk option, HAProxy has no problem proxying
these servers. What am I doing wrong?
what's listed under Status for these servers when viewing your
haproxy status page?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
Re: option httpchk is reporting servers as down when they're not
well, looks like ur servers are actually down then. Do a curl from
your haproxy machine to both servers. What do you get?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
On Wed, Mar 4, 2009 at 9:40 PM, Allen, Thomas tal...@asce.org wrote:
Never mind, I got it going. My stats page simply says that both servers are
down. What else should I be looking for?
Thomas Allen
Web Developer, ASCE
703.295.6355
-Original Message-
From: Jeffrey 'jf' Lim [mailto:jfs.wo...@gmail.com]
Sent: Wednesday, March 04, 2009 2:22 AM
To: Allen, Thomas
Cc: haproxy@formilux.org
Subject: Re: option httpchk is reporting servers as down when they're not
- Show quoted text -
On Wed, Mar 4, 2009 at 4:05 AM, Allen, Thomas tal...@asce.org wrote:
Hi,
I like the idea of having HAProxy check server health, but for some reason,
it reports all of my servers as down. Here's my full config:
listen http_proxy :80
mode http
balance roundrobin
option httpchk
server webA {IP} cookie A check
server webB {IP} cookie B check
I tried option httpchk /index.php just to be sure, and got the same
result. If I remove the httpchk option, HAProxy has no problem proxying
these servers. What am I doing wrong?
what's listed under Status for these servers when viewing your
haproxy status page?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not
help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
Re: option httpchk is reporting servers as down when they're not
On Wed, Mar 4, 2009 at 4:05 AM, Allen, Thomas tal...@asce.org wrote:
Hi,
I like the idea of having HAProxy check server health, but for some reason,
it reports all of my servers as down. Here's my full config:
listen http_proxy :80
mode http
balance roundrobin
option httpchk
server webA {IP} cookie A check
server webB {IP} cookie B check
I tried option httpchk /index.php just to be sure, and got the same
result. If I remove the httpchk option, HAProxy has no problem proxying
these servers. What am I doing wrong?
what's listed under Status for these servers when viewing your
haproxy status page?
-jf
--
In the meantime, here is your PSA:
It's so hard to write a graphics driver that open-sourcing it would not help.
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
