Re: Haproxy failover: DNS RR vs Virtual IP (heartbeat, keepalived)
DNS RR offers at least some measure of protection in case the whole data center loses its connection(s) to the world, which happens from time to time. Of course, if you have another data center to fall to. ;) So ideally you would have keepalived-like solution inside each data center, but DNS RR if you deploy your solution to multiple data centers. Anze On Thursday 06 January 2011, Angelo Höngens wrote: Round robin is not the same as random. If a host name has 4 A records, then most DNS servers (if not all) will return it round-robin. So first a.a.a.1, then a.a.a.2, then a.a.a.1, then a.a.a.2, etc.. Of course there are multiple dns servers involved and thousands of clients over the world, but in the end we almost always get a perfect 25-25-25-25 balancing. -Original Message- From: David [mailto:da...@silveregg.co.jp] Sent: donderdag 6 januari 2011 9:45 To: haproxy@formilux.org Subject: Re: Haproxy failover: DNS RR vs Virtual IP (heartbeat, keepalived) On 01/06/2011 05:01 PM, Angelo Höngens wrote: (sorry for top posting, damn Outlook) Just to second Willy's story, this is how a lot of people do it, including us. We use pacemaker for high availability, and dnr rr for loadbalancing. For example we have a 4-node cluster running varnish and haproxy. In this case I have 4 virtual ipv4-addresses and 4 virtual ipv6 addresses on the cluster. We use pacemaker to keep the virtual ip's up, and we use dns round-robin to balance the load. We get nice equal load balancing this way, and if a node is down (or I want to do maintenance), the vip's move to other nodes, and they take the extra load. Thanks for the information. Both Willy and you refer to DNS RR as a load balancing solution, but I don't really understand that point: if caching, etc... means hostname-load balancer resolution is random, the load balancing will likely be very unbalanced, no ? cheers, David
Re: Include client IP address in rewritten URL
This might be obvious, but still:
all of the programming languages I know of can access REMOTE_ADDR even if they
don't get it via GET vars. For instance, in PHP you can get it from
$_SERVER['REMOTE_ADDR']. So there should be no need to use GET vars, just
change the way you access the data. The change should be trivial.
If the code is not yours and you are not allowed to change it, there could
still be some way to fix this. For instance, in PHP you could try
autoprepending this code:
?php
$_REQUEST['REMOTE_ADDR'] = $_GET['REMOTE_ADDR'] = $_SERVER['REMOTE_ADDR'];
?
That's ugly, of course, but it should still be much faster than Apache
rewrite.
YMMV.
Anze
On Tuesday 14 December 2010, Willy Tarreau wrote:
Hi Shawn,
On Mon, Dec 13, 2010 at 07:29:20PM -0700, Shawn Heisey wrote:
I am trying to use haproxy 1.4.8 to replace an apache URL rewriting
engine that crumbles under load. I have to duplicate the following part
of it:
ipaddr=%{REMOTE_ADDR}
The IP address is not part of the original URL, but it must be included
in the rewritten one. Looking at the documentation, I cannot figure out
how to do this. Can anyone help?
You can't with current version. A variable-based system is planned for
version 1.5, but its development has not started yet so I doubt we'd
get anything by the end of the year :-/
Regards,
Willy
Re: Performance Question
I hope I am not saying something stupid (haven't met `ab` before), but wouldn't non-2xx responses mean HTTP errors? In that case you have the same amount of errors (actually, a bit more???) than requests sent. You should fix those first before running any tests. Also, as Hank said, you should probably run ab for more than just a few seconds. Anze On Thursday 07 October 2010, Les Stroud wrote: My goal in this testing was to produce a report for management showing how haproxy with the four tomcats increased capacity and improved scale vs having direct access to tomcat. I would think it would be pretty straight forward to produce that graph. Unfortunately, the results that I received did not line up. So, I figure that I have a misconfiguration of some sort. Here are the details; So, I have 3 scenarios: Scenario 1: /———tomcat1 /——— machine 3 — / \———tomcat2 ab on machine1 —— haproxy on machine 2 —— \ /———-tomcat3 \——— machine 4 — \———tomcat4 Scenario 2: /———tomcat1 /——— machine 3 — / \———tomcat2 ab and haproxy on machine 2 —— \ /———-tomcat3 \——— machine 4 — \———tomcat4 Scenario 3: /———tomcat1 ab and haproxy on machine 3 —— \———tomcat2 For all of the scenarios, each box is a vm with 4 procs, 12 Gb of RAM, and fiber channel io. They are all using the latest drivers from vmware and are patched up redhat EL 5.5 (latest). Here is the uname string: 2.6.18-194.17.1.el5 #1 SMP Mon Sep 20 07:12:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux In all cases I tested by running the following test: ab -n 1 -c 100 http://tomcat1:8080/examples/jsp/ ab -n 1 -c 100 http://haproxy:81/examples/jsp/ I also ran these tests at various levels of concurrency (actually ran them at powers of 2) and received the same results. Here is an example of the results that I receive: Scenario 1: ab -n 1 -c 100 http://tomcat1:8080/examples/jsp/ Concurrency Level: 100 Time taken for tests: 1.602049 seconds Complete requests: 1 Failed requests:0 Write errors: 0 Non-2xx responses: 10001 Total transferred: 1630163 bytes HTML transferred: 0 bytes Requests per second:6242.01 [#/sec] (mean) Time per request: 16.020 [ms] (mean) Time per request: 0.160 [ms] (mean, across all concurrent requests) Transfer rate: 993.10 [Kbytes/sec] received ab -n 1 -c 100 http://haproxy:81/examples/jsp/ Concurrency Level: 100 Time taken for tests: 1.821838 seconds Complete requests: 1 Failed requests:0 Write errors: 0 Non-2xx responses: 1 Total transferred: 193 bytes HTML transferred: 0 bytes Requests per second:5488.96 [#/sec] (mean) Time per request: 18.218 [ms] (mean) Time per request: 0.182 [ms] (mean, across all concurrent requests) Transfer rate: 1034.12 [Kbytes/sec] received Scenario 2: ab -n 1 -c 100 http://tomcat1:8080/examples/jsp/ Concurrency Level: 100 Time taken for tests: 1.558019 seconds Complete requests: 1 Failed requests:0 Write errors: 0 Non-2xx responses: 10002 Total transferred: 1630326 bytes HTML transferred: 0 bytes Requests per second:6418.41 [#/sec] (mean) Time per request: 15.580 [ms] (mean) Time per request: 0.156 [ms] (mean, across all concurrent requests) Transfer rate: 1021.81 [Kbytes/sec] received ab -n 1 -c 100 http://haproxy:81/examples/jsp/ Concurrency Level: 100 Time taken for tests: 1.915394 seconds Complete requests: 1 Failed requests:0 Write errors: 0 Non-2xx responses: 10012 Total transferred: 1932316 bytes HTML transferred: 0 bytes Requests per second:5220.86 [#/sec] (mean) Time per request: 19.154 [ms] (mean) Time per request: 0.192 [ms] (mean, across all concurrent requests) Transfer rate: 985.18 [Kbytes/sec] received Scenario 3: ab -n 1 -c 100 http://tomcat1:8080/examples/jsp/ Concurrency Level: 100 Time taken for tests: 1.765041 seconds Complete requests: 1 Failed requests:0 Write errors: 0 Non-2xx responses: 10010 Total transferred: 1631630 bytes HTML transferred: 0
Re: HAProxy Setup not talking to Passenger/Apache
Nick, try rather: $ curl 10.202.197.103:7000 $ curl 10.208.202.70:7000 You are not using localhost:7000 in your config, but web02:7000 and web01:7000 instead. Anze On Tuesday 03 August 2010, Willy Tarreau wrote: Hi Nick, On Mon, Aug 02, 2010 at 11:13:29PM -0500, Nick Hilem wrote: I have HAProxy, 1.3.15, on the frontend of a few Ubuntu 9.04 instances with the following haproxy.cfg that are distributing to a couple apache/passenger instances. My problem is that if I... curl http://localhost/ It sits for awhile then returns a 504 Gateway Timeout. However if I... curl http://localhost:7000/ Which passenger is listening on, it returns fairly quickly with the page I would expect to see. That's kinda weird. One (unlikely) possibility would be that the server responds differently when it does not see its port in the request. Could you force it to see if it changes anything : curl -H Host: localhost http://localhost:7000/ curl -H Host: localhost:7000 http://localhost/ I've also included the lsof results and hosts file on web01, everything is the same on web02. Anyone have any guess as to why my haproxy and passenger setup aren't talking to each other? Is my haproxy.cfg file misconfigured somehow? No, it looks fine. You don't even have option httpclose, so that means that the connection passes unmodified. Please run it in debug mode if you can (using -d) and check the output for request/responses. Maybe we'll see something odd, but I have no idea what. Also you could try to switch to mode tcp and see if that changes anything. If so, that means that it blocks in the HTTP protocol. If it does not work either, then it will be more likely a problem handling the Host header. Regards, Willy
Re: HAProxy Setup not talking to Passenger/Apache
Nick, try rather: $ curl 10.202.197.103:7000 $ curl 10.208.202.70:7000 You are not using localhost:7000 in your config, but web02:7000 and web01:7000 instead. Anze On Tuesday 03 August 2010, Willy Tarreau wrote: Hi Nick, On Mon, Aug 02, 2010 at 11:13:29PM -0500, Nick Hilem wrote: I have HAProxy, 1.3.15, on the frontend of a few Ubuntu 9.04 instances with the following haproxy.cfg that are distributing to a couple apache/passenger instances. My problem is that if I... curl http://localhost/ It sits for awhile then returns a 504 Gateway Timeout. However if I... curl http://localhost:7000/ Which passenger is listening on, it returns fairly quickly with the page I would expect to see. That's kinda weird. One (unlikely) possibility would be that the server responds differently when it does not see its port in the request. Could you force it to see if it changes anything : curl -H Host: localhost http://localhost:7000/ curl -H Host: localhost:7000 http://localhost/ I've also included the lsof results and hosts file on web01, everything is the same on web02. Anyone have any guess as to why my haproxy and passenger setup aren't talking to each other? Is my haproxy.cfg file misconfigured somehow? No, it looks fine. You don't even have option httpclose, so that means that the connection passes unmodified. Please run it in debug mode if you can (using -d) and check the output for request/responses. Maybe we'll see something odd, but I have no idea what. Also you could try to switch to mode tcp and see if that changes anything. If so, that means that it blocks in the HTTP protocol. If it does not work either, then it will be more likely a problem handling the Host header. Regards, Willy
acl matching cookies
Hi all! I hope this is not too newbie question, but... is it possible to redirect to a specified backend whenever a predefined value is found in a cookie? (or at least, when a predefined cookie is found at all?) I have read through the docs and ACLs don't seem to be able to match the cookies... I am trying to find a way to test new versions of my app (final checks) before they go live. The idea was this: - remove check.txt, thus taking server down - upgrade web app on that server - set a special cookie in my browser - go to haproxy, it should see the cookie and redirect to down server - test the app - when satisfied: `touch check.txt` and upgrade others too Is this possible? 1) can haproxy redirect to down server? 2) can it redirect based on a cookie? (I wouldn't want to hardcode IPs and URLs can't be changed) How do others cope with the problem of upgrading the app? I would appreciate some ideas. Thanks, Anze
Re: Truncated health check response from real servers
Hi Willy, On Saturday 10 July 2010, Willy Tarreau wrote: Actually, I am not that much interested in the patch per-se, I would just like to check the body of the check-response and compare it to a string for more reliable checks. It's what the patch does. Hmmm, I thought it was bigger... I have included my original mail (and reasoning) on the subject below. It's not on top of my priority list right now either, still... sooner or later we might get bitten by it. I'd love to help out coding, but sockets and C++ are not my strong points (though I am comfortable in C). So you're welcome, haproxy is plain C, and I don't understand C++ either ;-) How did I miss that? :-D Ok, I'll take a look at it then. Thanks for updating the patch and for pointers on where to look for problematic sections. I can't promise it will be soon though, we have a lot of work right now... but I'll do my best. Thanks, Anze
Re: Truncated health check response from real servers
Indeed it never made it in. Each time I look at it, I realize that cleaning it up and fixing it will take more time than what I have available to work on it :-/ I can understand that completely. :) Actually, I am not that much interested in the patch per-se, I would just like to check the body of the check-response and compare it to a string for more reliable checks. I have included my original mail (and reasoning) on the subject below. It's not on top of my priority list right now either, still... sooner or later we might get bitten by it. I'd love to help out coding, but sockets and C++ are not my strong points (though I am comfortable in C). I can help with testing though if needed. Let me know if it helps and if/when you find time for this. :) Or, if you wish, I can take Nick's patch and extract just the parts with checking the response body? Can you point out some of the errors so I can avoid them? Thanks, Anze On Friday 12 March 2010, Anze wrote: Hi! First of all: Willy Tarreau others, thank you for a great piece of software! It just works. :) PHP has a bug when errors in PHP code do not always issue HTTP error status code to be sent, but instead return status 200 (http://bugs.php.net/bug.php?id=50921) The problem is that when haproxy calls check.php, the scrpt itself is not run if there is an error in it (or in the includes it needs for the checks). So this check is not really reliable. I know this is not really haproxy's problem (the bug is on the PHP side), but reliability would be much higher if haproxy could compare the response body to some string (for instance, server must return really ok in request body). That way the bug in interpreter could never cause a false positive. Example PHP script: ? // this fails: call_some_undefined_function(); // this is not executed: echo really ok ? This fails silently and return an empty document (or document with error description, depending on PHP config) with status 200. As a side note, that would also allow us to catch PHP configuration errors, such as short_tags set to off - in which case the previous code would be just listed (with status 200!), not executed. When compared to really ok, it would fail. Any chance this can be done with haproxy? Thanks, Anze On Thursday 08 July 2010, Willy Tarreau wrote: On Wed, Jul 07, 2010 at 11:37:48AM +0200, Anze wrote: Hi all, We have tried upgrading our haproxy version to 1.4.8 and found than this configuration setting fails: http-check expect string check_is_ok with message: 'http-check' only supports 'disable-on-404' I guess the patch never made it to stable (yet?). Are there any plans to add it to the trunk? Indeed it never made it in. Each time I look at it, I realize that cleaning it up and fixing it will take more time than what I have available to work on it :-/ It's still available online though, but I would really not run it if I were you, as it still lacks some memory bounds checkings. Regards, Willy
Re: Truncated health check response from real servers
Hi all, We have tried upgrading our haproxy version to 1.4.8 and found than this configuration setting fails: http-check expect string check_is_ok with message: 'http-check' only supports 'disable-on-404' I guess the patch never made it to stable (yet?). Are there any plans to add it to the trunk? Thanks, Anze On Wednesday 17 March 2010, Cyril Bonté wrote: Hi all, Le Mardi 16 Mars 2010 21:35:10, Willy Tarreau a écrit : I'm now gathering my changes and committing your patch with the small fixes above. That way we can concentrate on ECV. I've played a little with this commits. I added some traces to see what happens when a timeout occurs and noticed that the response is concatenated after the response of the previous timed out check, until the buffer is full. This can occur when the server started to send a response but was too long to finish or if a check uses HTTP/1.1 on a server with a HTTP Keep-Alive longer than the check timeout (adding Connection: close could help prevent this, maybe it should be added in the option httpchk documentation). The expect option from the ECV patch can be affected by this. The patch provided in attachment resets the buffer after a timeout bur I wonder if it can't be done in set_server_check_status(). Also, to save a little memory, the check_data buffer is only allocated for the servers that are checked.
Re: dynamic referer protecion
Hello Hervé, Thanks for the link, it reminded me to read it again and refresh my memory. :) The manual is a really good start to using haproxy, but it doesn't cover the topics that Willy wrote in previous mail (maxconn, setting timeouts, tarpit,...). I was thinking more about what to do if... kind of guide / book (depends on volume and the will to deal with publishers). Thanks, Anze On Tuesday 16 March 2010, Hervé COMMOWICK wrote: Hello Anze, Willy already wrote this a while ago : http://1wt.eu/articles/2006_lb/ Hervé On 03/16/2010 12:39 PM, Anze wrote: Willy: thanks for the explanation on tactics below! Ever thought of writing a book on the subject? I don't mean haproxy configuration but its possible deployments, common setups,...
Re: httpchk - checking request body
On Saturday 13 March 2010, Willy Tarreau wrote: On Fri, Mar 12, 2010 at 11:55:59PM +0100, Anze wrote: On Friday 12 March 2010, Willy Tarreau wrote: It is possible with the patch that Nick Chalk recently posted in the thread Truncated health check response from real servers. It is a rework of an ... You seem to be a good candidate to test it :-) I'm having trouble getting this to work. Without option httpchk - no problem. Then I try with the check and I get this: no, your config is not clear to me now. Could you please post it ? Sorry, it was late. :) Here is my config: (I replaced all sensitive info, I hope I haven't made a mistake) - global log 127.0.0.1 local0 log 127.0.0.1 local1 notice #log loghostlocal0 info maxconn 4096 #chroot /usr/share/haproxy user haproxy group haproxy daemon #debug #quiet defaults log global modehttp option httplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 5 srvtimeout 5 listen webfarm 123.123.123.123:80 mode http stats enable balance roundrobin stats auth user:pass cookie SERVERID insert indirect option httpclose option forwardfor except 123.123.123.123 # option httpchk HEAD /check.php HTTP/1.0 # option httpchk /check.php # http-check expect string all_is_ok server a0 10.0.0.1:8080 cookie a0 check server a1 10.0.0.2:8080 cookie a1 check - I have played with the 3 uncommented lines in listen section - these are the results: With 1.4.1 WITHOUT patch: - all 3 lines disabled: system works (serves content, no problem, but no check) - only 1. line enabled: system works (serves content, performs checks) - only 2. line enabled: system works (serves content, performs checks) So the system works and it is correctly configured. With 1.4.1 WITH patch applied: - all 3 lines disabled: system works (serves content, no problem, but no check) - only 1. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - only 2. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - 2. and 3. line or 1. and 3. line enabled: ERROR as above. Hope that clears things up... Any ideas? Anze
Re: httpchk - checking request body
On Saturday 13 March 2010, Willy Tarreau wrote: On Sat, Mar 13, 2010 at 02:50:32PM +, Nick Chalk wrote: With 1.4.1 WITH patch applied: - all 3 lines disabled: system works (serves content, no problem, but no check) - only 1. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - only 2. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - 2. and 3. line or 1. and 3. line enabled: ERROR as above. On the ToDo list is better logging of what httpchk is doing - along with working out all the options to the original patch. As a quick test, try adding the following options in place of the commented-out lines above: option httpchk GET /check.php HTTP/1.0 http-check expect rstring all_is_ok The 'string' test should work, but it's had less testing than 'rstring'. I've not tried the HEAD method; I can't see it working with 'expect'. Ah indeed, I did not notice the method was HEAD. Obviously it will not get any contents, explaining the issue. However, Anze indicated that as soon as the patch was included, option httpchk stopped working. I suspect that the status code check is always enabled and does not match what is implicitly configured (eg: 0). I may be wrong, that's pure guess. Nick, the above configuration doesn't work. Actually, none of these work: - option httpchk GET /check.php HTTP/1.0 http-check expect rstring all_is_ok - option httpchk GET /check.php HTTP/1.0 http-check expect string all_is_ok - option httpchk GET /check.php HTTP/1.0 http-check expect rstring all_is_ok - option httpchk GET /check.php HTTP/1.0 - option httpchk HEAD /check.php HTTP/1.0 - I think it is time for that logging feature... ;-D I will try to trace where the check fails and why. I guess it is easiest for me to do it because i can replicate the error anytime. If you have some guidelines where in the code to add printf()-s however, I'd appreciate it. Or I'll just take a look at what the patch changes... I love open source software. :) Thanks, Anze
Re: httpchk - checking request body
I will try to trace where the check fails and why. I guess it is easiest
for me to do it because i can replicate the error anytime. ...
Ok, looks like the culprit is this piece of code:
-
/* Close the transmit channel, leaving the connection half-open... */
int shut = shutdown(fd, SHUT_WR);
int err = errno;
if (shut == -1)
{
Alert(event_srv_chk_w(): err = %i, %s\n, err, strerror(err));
}
-
If I disable this (as it was before) everything works as it should. It even
figures out (correctly) that one server returns all_is_ok and another just
empty string - and takes the other one down. Nice, exactly what I need. :D
However, I have no idea why it works now. My knowledge of sockets is very
limited. Also, I have found this in the first 10 minutes of looking at the
code - I found that httpchk_expect() is not even called, so I checked other
stuff in the patch and this looked very suspicious to me so I disabled it. And
it just started working. I also figured that shutting down socket for writes
only is... well, a bit black-magic-ish and not really necessary. ;) You must
shutdown the whole socket sometime later anyway.
In other words: I can't say that this is a correct way to deal with the
problem - I haven't got a clue. But it works now.
Does it shed any light on the issue?
Thanks,
Anze
Re: httpchk - checking request body
If I disable this (as it was before) everything works as it should. It even figures out (correctly) that one server returns all_is_ok and another just empty string - and takes the other one down. Interesting. Have you run haproxy -d from the command line to check the debugging output? option log-health-checks is also useful. I was running it from the command line (too) because logging was not working (rsyslogd's UDP support was disabled). That was the output I have sent in the first mails. It's a half-close. HAProxy only sends one request, and then waits for the response, so we might as well close the transmit channel. Are you sure nothing is being sent anymore through this socket? Never? When tcpdump'ing the problem system that led me to work on this, I found that HAProxy was not closing the transmit channel properly. The real servers were sending FIN to close their side, and HAProxy was responding with RST. So I put in the half-close to make sure the connection was cleanly shut down. I think this is not the proper way to deal with this, but as I said, I am no expert in sockets programming. I would say that haproxy still tries to sent RST but can't - which leads to an error somewhere. It's interesting that this has caused problems in your system. As well as running HAProxy in debug mode, would you be able to do a tcpdump on the system running haproxy? If you could do that both with and without the half-close, that would be very useful. Hmmm, I am a bit out of depths here. I would love to help out, but please send me more specific instructions on what you need. What arguments should I use with tcpdump so it helps you? Also, if you don't mind, I'd like to send it privately to you and Willy (so I don't need to clean up too much private / security sensitive info from it). If required, I can send you a version of checks.c with debugging code in it - that will dump a lot of data on the checks if you run it in debug. Sure, bring it on. ;) option httpchk GET /check.php HTTP/1.0 ... option httpchk HEAD /check.php HTTP/1.0 Those two will not check the content - I'll need to trawl through the config code to check whether it defaults to checking the HTTP return code in these cases. I have tested GET with the fixed version (no half-closed socket) and it works. But it would still be good if you checked it out for sure. There are a lot of existing configs that are setup like this - it wouldn't be good if they broke down in next version. ;) Thanks, Anze
Re: setup with Oracle and SSL
On Saturday 13 March 2010, Craig Carl wrote: Anne - Your would need an application to handle SSL and forward HTTP. I use stunnel for that with no problem. This is the guide I used, the basics are the same on any distro - Also, there is a patch for stunnel on haproxy homepage. You can find more info about haproxy (not) supporting HTTPS there too. Enjoy! Anze
Re: httpchk - checking request body
Hi Nick, it's a bit late now and I'm not sure I will be able to test this tomorrow... I have quite a lot to do. Will try asap, if not sooner, by Monday. But I'll try to do it tomorrow. Thanks for your help debugging this. Thanks for your help in solving this. :D Thanks, Anze On Saturday 13 March 2010, Nick Chalk wrote: Hello Anze. On 13 March 2010 20:08, Anze anzen...@volja.net wrote: Interesting. Have you run haproxy -d from the command line to check the debugging output? option log-health-checks is also useful. I was running it from the command line (too) because logging was not working (rsyslogd's UDP support was disabled). That was the output I have sent in the first mails. Ah, missed that email. It's a half-close. HAProxy only sends one request, and then waits for the response, so we might as well close the transmit channel. Are you sure nothing is being sent anymore through this socket? Never? Not in my packet traces. HAProxy sends one request, receives the response from the real server, real server closes its channel, HAProxy replies with RST. Future tests bind a new socket. When tcpdump'ing the problem system that led me to work on this, I found that HAProxy was not closing the transmit channel properly. The real servers were sending FIN to close their side, and HAProxy was responding with RST. So I put in the half-close to make sure the connection was cleanly shut down. I think this is not the proper way to deal with this, but as I said, I am no expert in sockets programming. Neither am I - my background is in System/Network Admin. I'm also just learning the HAProxy code, so there may be a better way to ensure that the check connection is closed properly. I would say that haproxy still tries to sent RST but can't - which leads to an error somewhere. The RST should not be sent - that's a response to an already-closed socket. However, HAProxy is not signalling that the connection should be closed by sending a FIN. It's interesting that this has caused problems in your system. As well as running HAProxy in debug mode, would you be able to do a tcpdump on the system running haproxy? If you could do that both with and without the half-close, that would be very useful. Hmmm, I am a bit out of depths here. I would love to help out, but please send me more specific instructions on what you need. What arguments should I use with tcpdump so it helps you? Also, if you don't mind, I'd like to send it privately to you and Willy (so I don't need to clean up too much private / security sensitive info from it). I have no problem with you sending it to me directly. My usual tcpdump invocation is: tcpdump -i any -s 0 -w output_file Substitute an appropriate filename for output_file. You won't want to leave this running for too long, as it listens on all interfaces and captures the whole of every packet. I'd suggest starting the tcpdump, then start HAProxy, wait for a few health checks to be sent, then stop the tcpdump with Ctrl-C. If required, I can send you a version of checks.c with debugging code in it - that will dump a lot of data on the checks if you run it in debug. Sure, bring it on. ;) I'll send that directly to you, rather than the list... Those two will not check the content - I'll need to trawl through the config code to check whether it defaults to checking the HTTP return code in these cases. I have tested GET with the fixed version (no half-closed socket) and it works. Interesting. I'll test that locally. Thanks for your help debugging this. Nick.
httpchk - checking request body
Hi! First of all: Willy Tarreau others, thank you for a great piece of software! It just works. :) PHP has a bug when errors in PHP code do not always issue HTTP error status code to be sent, but instead return status 200 (http://bugs.php.net/bug.php?id=50921) The problem is that when haproxy calls check.php, the scrpt itself is not run if there is an error in it (or in the includes it needs for the checks). So this check is not really reliable. I know this is not really haproxy's problem (the bug is on the PHP side), but reliability would be much higher if haproxy could compare the response body to some string (for instance, server must return really ok in request body). That way the bug in interpreter could never cause a false positive. Example PHP script: ? // this fails: call_some_undefined_function(); // this is not executed: echo really ok ? This fails silently and return an empty document (or document with error description, depending on PHP config) with status 200. As a side note, that would also allow us to catch PHP configuration errors, such as short_tags set to off - in which case the previous code would be just listed (with status 200!), not executed. When compared to really ok, it would fail. Any chance this can be done with haproxy? Thanks, Anze
httpchk - checking request body
Hi! First of all: Willy Tarreau others, thank you for a great piece of software! It just works. :) PHP has a bug when errors in PHP code do not always issue HTTP error status code to be sent, but instead return status 200 (http://bugs.php.net/bug.php?id=50921) The problem is that when haproxy calls check.php, the scrpt itself is not run if there is an error in it (or in the includes it needs for the checks). So this check is not really reliable. I know this is not really haproxy's problem (the bug is on the PHP side), but reliability would be much higher if haproxy could compare the response body to some string (for instance, server must return really ok in request body). That way the bug in interpreter could never cause a false positive. Example PHP script: ? // this fails: call_some_undefined_function(); // this is not executed: echo really ok ? This fails silently and return an empty document (or document with error description, depending on PHP config) with status 200. As a side note, that would also allow us to catch PHP configuration errors, such as short_tags set to off - in which case the previous code would be just listed (with status 200!), not executed. When compared to really ok, it would fail. Any chance this can be done with haproxy? Thanks, Anze
Re: httpchk - checking request body
On Friday 12 March 2010, Willy Tarreau wrote: On Fri, Mar 12, 2010 at 09:32:16AM +0100, Anze wrote: I know this is not really haproxy's problem (the bug is on the PHP side), but reliability would be much higher if haproxy could compare the response body to some string (for instance, server must return really ok in request body). That way the bug in interpreter could never cause a false positive. ... Any chance this can be done with haproxy? It is possible with the patch that Nick Chalk recently posted in the thread Truncated health check response from real servers. It is a rework of an ... You seem to be a good candidate to test it :-) I'd love to! We will be going live in a few weeks (hopefully :) ) and things must work then, so I'm on a tight schedule. Will try to apply the patch and let you know how it goes. From what I gathered from the sources, I just set this option: http-check expect string all_is_ok And then in check.php: ? // ... do some checks... echo all_is_ok; ? Am I correct? Thanks for a fast response, really appreciate it! Anze
Re: httpchk - checking request body
On Friday 12 March 2010, Willy Tarreau wrote: It is possible with the patch that Nick Chalk recently posted in the thread Truncated health check response from real servers. It is a rework of an ... You seem to be a good candidate to test it :-) I'm having trouble getting this to work. Without option httpchk - no problem. Then I try with the check and I get this: - Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. Using sepoll() as the polling mechanism. [WARNING] 070/224249 (2731) : Server webfarm/a0 is DOWN, reason: Layer7 invalid response, check duration: 4ms. [WARNING] 070/224250 (2731) : Server webfarm/a1 is DOWN, reason: Layer7 invalid response, check duration: 0ms. [ALERT] 070/224250 (2731) : proxy 'webfarm' has no server available! - I haven't even gotten to http-check expect string ... yet. The servers (and check.php files on them) are available, I can `wget` the check files. Any idea what is going on or how I could get more info about why it is failing? Thanks, Anze
Re: httpchk - checking request body
On Friday 12 March 2010, Anze wrote: The servers (and check.php files on them) are available, I can `wget` the check files. Any idea what is going on or how I could get more info about why it is failing? Additional info: I have tried version 1.4.1 without patch and httpchk works with it. Version 1.4.1 with the patch however fails with previously noted symptoms. So the patch is not perfect yet. ;) Thanks, Anze
