Re: redispatch request if 502 received
Thanks for all the input guys. Sounds like haproxy is the wrong place to solve the problem, so we've instead decided to remove nginx from the equation (which was only providing gzip capabilities that we can implement directly in our webserver). Thus, requests failing on the webserver will actually result in disconnects, which will allow haproxy to redispatch the request to a new server. On Tue, Aug 24, 2010 at 2:53 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Aug 24, 2010 at 12:52:20AM -0600, Jerry Champlin wrote: In that case I'm not sure how to make it work. Seems like you will need to modify http_proto.c to cause the request to attempt to be redispatched when haproxy receives a valid 502 http response from the server. I doubt that this is the only thing to change, but it probably involves changing this section of code and others: it won't work. Haproxy tries to redispatch as far as it can. If you see a 502, that means that the request was *really* forwarded and that it does not have it anymore. So it won't be able to redispatch it to anyone else. That's one reason some people like to put caches such as nginx or varnish before or after haproxy. Caches by their nature are able to replay a request if they consider it is not risky, because they keep a copy of it. Regards, Willy
Re: redispatch request if 502 received
Hey Jerry, can you elaborate? To be clear what happens is: 1/ proxy receives requests 2/ proxy sends request to webserver 3/ webserver sends 502 response to proxy 4/ proxy forwards 502 response to the end user I do have other webservers that I want to re-dispatch the request to - is that what you mean by backup servers? In any case, it seems like the proxy must be the one responsible for handling the failure. On Fri, Aug 20, 2010 at 12:08 PM, Jerry Champlin j...@absolute-performance.com wrote: It seems like you could configure backup servers within the pool to accomplish this although it's not quite as clean as what I suspect you are looking for. -Jerry Jerry Champlin Absolute Performance Inc. -- Enabling businesses to deliver critical applications at lower cost and higher value to their customers. On Fri, Aug 20, 2010 at 12:54 PM, Dustin Moskovitz mos...@asana.comwrote: Hi all, We run haproxy in front of a bank of nginx servers, which each proxy requests to a custom web server on the same box. I was wondering if it is possible to configure haproxy such that it will redispatch requests that receive a 502 from nginx to a new server. I looked through the docs but could not find anything that seemed relevant. Dustin
Re: haproxy websockets
Ok great, I've got things working with stunnel in front now. Thanks for all the help. On Wed, Apr 28, 2010 at 10:23 PM, Willy Tarreau w...@1wt.eu wrote: Hi Dustin, On Wed, Apr 28, 2010 at 04:51:41PM -0700, Dustin Moskovitz wrote: Actually, I should have mentioned at the beginning that we are using websockets to communicate with a *stateful* server, so we don't want to close the connection. I'm not speaking about closing the websockets connection at all. And that's what's nice with websockets, compared to earliser long-polling methods, it's that once the server asks to switch to websockets, a bidirectional tunnel is established over the HTTP connection between the client and the server. At this point, haproxy does not care anymore about what passes into it. However, explicitly declaring mode http in either the defaults section or the frontend backend sections has made things happy. Yes, that's expected. Now, a new problem (please let me know if/when I should start new threads). If I introduce ssl into the mix, I cannot get it to work with http over a websockets connection (I have it working without websockets, so it is otherwise ok). haproxy does not decrypt SSL by itself. You need something like stunnel, nginx or pound in front of it to handle it. I like stunnel for that because it does not mangle the HTTP part at all. However, if you don't want to decrypt the SSL traffic, you can make this instance work in TCP mode and have the server do the job. You won't have any persistence though, and the server will not get the client's IP address. Using the config at the bottom of this email, if I hit port 80 with a request or two and then move to ssl, haproxy will systematically *seg fault*, like so: Yes, this was recently reported and fixed. It's only a debug mode issue. When running in debug mode, haproxy tries to display all headers it receives. When it receives invalid requests or responses (SSL traffic being invalid from an HTTP point of view), it dereferences a NULL pointer. The fix is pending in the tree. It is not a problem for you anyway, because when you see the segfault, it indicates that haproxy received something invalid which would not have worked in non-debug mode. Regards, Willy
Re: haproxy websockets
leastconn server ws_0 prod_ws0.ec2.:8080 check backend beta_ws mode http option tcplog log /dev/log local0 debug balance leastconn server beta_ws_0 beta_ws0.ec2.:8080 check backend lists_ws mode http option tcplog log /dev/log local0 debug balance leastconn server ws_2 prod_ws2.ec2.:8080 check listen stats 0.0.0.0:8090 mode http stats enable stats uri /haproxy stats refresh 5s On Tue, Apr 27, 2010 at 10:09 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Apr 27, 2010 at 05:36:12PM -0700, Dustin Moskovitz wrote: Actually, I spoke too soon. When I create a config similar to Laurie's (see below), I find that I still don't see headers and my default backend is always utilized. I originally mistook this for working. Laurie, are you positive you are actually routing traffic based on the hdr rule? This implies to me the default is tcp? When I explicitly declare mode http in the backends, the client never sees the response, as I mentioned in the first email. You don't have any mode http statement, so haproxy does not wait for any HTTP contents to decide what backend to use. You have no httpclose/http-server-close option, so you're working in tunnel mode for backwards compatibility, where only the first request is processed and the rest is considered as data. Please just add mode http and option http-server-close to your defaults section and it should be OK. Regards, Willy
Re: haproxy websockets
Actually, I spoke too soon. When I create a config similar to Laurie's (see below), I find that I still don't see headers and my default backend is always utilized. I originally mistook this for working. Laurie, are you positive you are actually routing traffic based on the hdr rule? This implies to me the default is tcp? When I explicitly declare mode http in the backends, the client never sees the response, as I mentioned in the first email. haproxy.cfg: global maxconn 4096 user nobody group nobody #chroot /var/run/haproxy_jail defaults retries 3 maxconn 2000 contimeout 5000 clitimeout 5 srvtimeout 5 frontend ws_in bind *:80 acl is_lists hdr_end(host) -i lists.asana.com acl is_beta hdr_end(host) -i beta.asana.com use_backend beta_ws if is_beta use_backend lists_ws if is_lists default_backend ws backend ws option tcplog log /dev/log local0 debug balance leastconn server ws_0 prod_ws0.ec2.:8080 check server ws_1 prod_ws1.ec2.:8080 check backend beta_ws option tcplog log /dev/log local0 debug balance leastconn server beta_ws_0 beta_ws0.ec2.:8080 check backend lists_ws option tcplog log /dev/log local0 debug balance leastconn server ws_2 prod_ws2.ec2.:8080 check server ws_3 prod_ws3.ec2.:8080 check listen stats 0.0.0.0:8090 mode http stats enable stats uri /haproxy stats refresh 5s On Tue, Apr 27, 2010 at 5:11 PM, Dustin Moskovitz mos...@asana.com wrote: Yea, this was due to a failure to clean between builds, sorry about that. I think we've got what we need now, thanks! On Tue, Apr 27, 2010 at 3:11 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Apr 27, 2010 at 02:54:24PM -0700, Dustin Moskovitz wrote: Yea, I should have thought of that - certainly we are on the cutting edge :) Unfortunately, when I try to upgrade from 1.4.0, I am unable to get the proxy up and handling requests (forgetting the virtual hosts thing, just trying to use the config I already had working). The only thing I see in the log output is: Available polling systems : poll : pref=200, test result OK select : pref=150, test result OK Total: 2 (2 usable), will use poll. Using poll() as the polling mechanism. First, depending on the system you'll use it on, you could get better performance. On Linux, use TARGET=linux26 to get support for epoll(), and on FreeBSD/OpenBSD, use the appropriate makefile to get support for kqueue(). [ALERT] 116/214946 (671) : socket for logger #1 failed: Address family not supported by protocol (errno=97) Huh ? It's the first time I ever see that ! Looks like you can't create a UDP socket to send your logs ! Did you perform a make clean before rebuilding ? It is possible that some struct members have moved between the two versions and that you rebuilt without replacing all of your .o files. Requests just seem to hang from the client's perspective and I don't see anything on the console indicating they are being handled. Depending on the reason for the anomaly above, it could be anything. It's possible that the process is spinning or waiting forever or doing anything stupid. There is nothing in the log, which makes sense given the logging alert. I will get a separate alert for each backend block I have defined, regardless of whether it actually has a log line or what options I try using there. Is it possible that the new haproxy somehow enforces ipv6 addresses? No, there is no reason for this. And you would not get such a strange message anyway. If you want, you can send me your config in private, I'll retry here, just in case. If I can reproduce, whatever the reason we have to work on it so that either it cannot happen anymore or at least there are indications of the reason ! Regards, Willy
haproxy websockets
Our company is attempting to use haproxy to balance websocket requests and ran into a snag when trying to set up virtual hosts. Using acl rules to divide traffic between two backends is very straightforward and we're able to make it work when using http mode, but we need to use tcp for websockets. Otherwise the proxy never sends the response back to the client. Does anybody know of a way either to a) use haproxy in http mode with websockets (e.g. tell the proxy to send packets onward as it gets them from the backend) or b) create a host-based rule in tcp mode? Thanks, Dustin
