Re: HAProxy sometimes selects wrong acl
On Mon, Oct 15, 2012 at 02:35:57PM +0200, Milan Babuskov wrote:
On Mon, Oct 15, 2012 at 1:25 PM, Willy Tarreau w...@1wt.eu wrote:
I'll try Windows version of Safari to see if I could reproduce the problem.
Well, Safari 5.1.2 on Windows works without any problems both with and
without http-server-close, so no luck there.
So most likely it's only old outdated safari using an experimental protocol
version which is affected.
I have rechecked the old versions to refresh my memories :
- draft 75 was about an exact byte-matching handshake which is totally
incompatible with HTTP relaying since it doesn't talk about headers
or HTTP messaging but really bytes on the wire. This one was eliminated
2.5 years ago for this precise reason. Anyone still using this has no
chance to ever see his client work ; Also the framing used was subject
to some vulnerabilities which could be used to corrupt intermediary
caches.
- draft 76 was even more incompatible with relaying as it broke the
HTTP sequencing by expecting the proxies or servers to forward some
data that were not advertised as content-length. The result is that
the client gets stuck waiting for the server to respond, and the
server gets stuck waiting for the client to send the data. The framing
had the same issues as above. Similarly people using such an outdated
browser will never be able to reach a websocket site.
It's worth understanding that these were early design versions which have
nothing to do with the WebSocket protocol as it is now and which cannot
work by design. It's the principle of experimental specs. So there is really
nothing we can do to become compatible with these clients.
Maybe we should work instead on detecting such invalid clients and quickly
reject their requests instead of leaving them waiting for a timeout to
occur ?
They can be identified by the presence of the Upgrade header without the
Sec-WebSocket-Key header. So something like this would cleanly reject them :
http-request deny if { hdr_cnt(Upgrade) gt 0 } { hdr_cnt(Sec-WebSocket-Key)
eq 0 }
I don't know if such clients are used to display the response to the client,
otherwise we could even redefine the 403 error to send them a message inviting
them to replace their websocket-incompatible browser.
Regards,
Willy
Re: HAProxy sometimes selects wrong acl
On Mon, Oct 15, 2012 at 7:47 AM, Willy Tarreau w...@1wt.eu wrote: However it's better to figure what is causing the issue, even if this means that we should develop a workaround. The issue is the following: a) if I use http-server-close then websocket connections are broken for Safari users because Safari uses the old websocket spec. However, I cannot switch to tcp mode because I have multiple node.js servers running one the same machine, with users sharded accross (i.e. a particular user needs to go to a specific server). I need http to be able to do L7 routing to correct server. b) if I do not use http-server-close, then users behind Mikrotik routers have problem, because Mikrotik decides to reuse the same http connection for the same IP address even when a different hostname is used. The only thing HAProxy could do here is to read headers on each data exchange and if it detects hostname change which would trigger different ACL then HAProxy would switch connection to other backend. I don't know how hard would be to implement this, but I assume it messes with core functionality. Regards, -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
On Mon, Oct 15, 2012 at 1:25 PM, Willy Tarreau w...@1wt.eu wrote: I'll try Windows version of Safari to see if I could reproduce the problem. Well, Safari 5.1.2 on Windows works without any problems both with and without http-server-close, so no luck there. -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
Oops. I read your email wrong. Re-reading your email. I see that you are having problem with haproxy + long polling and websockets. There was a discussion earlier in this thread on websockets breaking http protocol (See http://marc.info/?l=haproxym=127803250504340w=2. Also see http://stackoverflow.com/questions/4360221/haproxy-websocket-disconnection) For Comet (long polling), haproxy might be closing the connection due to server timeout. I personally use websockets and comets by connecting client directly to the backend server for websocket and comet connections bypassing the load balancer all together. However, there might be better ways to do it which I am not aware of. Regards, Vivek On Sun, Oct 14, 2012 at 7:46 PM, Vivek Malik vivek.ma...@gmail.com wrote: I am not sure of how mikrokit could be having problems with http-server-close. http-server-close only affects the haproxy - backend connection, so it hidden from any front end. Having said that, I can think of one way to do http-server-close for specific conditions. It might not be the best method, but here it is. option http-server-close can be specified in backend. So, you can have a normal backend with http-server-close enabled and a mikrotik backend with http-server-close disabled. You can then use acl to choose the backend depending on http headers. Regards, Vivek On Sun, Oct 14, 2012 at 4:05 PM, Milan Babuskov milan.babus...@gmail.comwrote: On Fri, Oct 12, 2012 at 4:51 PM, Vivek Malik vivek.ma...@gmail.com wrote: Yes, option http-server-close would make haproxy run acls for every request. It will also enable keep-alive between client (or proxy) and haproxy. Thanks, this fixed to problem for Mikrotik users, but now I have problems with Safari users who cannot establish websockets or xhr-polling connections anymore :( So, I reverted to old config and have a new question. Is it possible to activate http-server-close option only for Mikrotik users? I read the docs, but appears I can only access http headers withing acls for frontend selection? I would like to achieve something like: option http-server-close WHEN http headers CONTAIN Mikrotik Thanks, -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
On Fri, Oct 12, 2012 at 4:08 AM, Vivek Malik vivek.ma...@gmail.com wrote: You need to add option httpclose or option http-server-close so that haproxy doesn't act in the tunnel mode (default behavior). Thank you. Could you please explain why does it work for 99.8% of the clients and only 0.2% fail. I'm trying to understand what is exactly happening? Regards, -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
because only 0.2% of users tried to switch domain name using the same TCP connection. cheers On Fri, Oct 12, 2012 at 2:44 PM, Milan Babuskov milan.babus...@gmail.com wrote: On Fri, Oct 12, 2012 at 4:08 AM, Vivek Malik vivek.ma...@gmail.com wrote: You need to add option httpclose or option http-server-close so that haproxy doesn't act in the tunnel mode (default behavior). Thank you. Could you please explain why does it work for 99.8% of the clients and only 0.2% fail. I'm trying to understand what is exactly happening? Regards, -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
On Fri, Oct 12, 2012 at 3:21 PM, Baptiste bed...@gmail.com wrote: because only 0.2% of users tried to switch domain name using the same TCP connection. Ah, I see. Their proxy sees that both domains have the same IP address, so it decides to reuse the TCP connection. I guess in this case http-server-close should be enough? Their proxy would still use the same connection, while HAProxy would close it and connect to a different backend server. Thanks, -- Milan Babuskov http://www.guacosoft.com
Re: HAProxy sometimes selects wrong acl
You need to add option httpclose or option http-server-close so that haproxy doesn't act in the tunnel mode (default behavior). Regards, Vivek On Thu, Oct 11, 2012 at 5:33 PM, Milan Babuskov milan.babus...@gmail.comwrote: Hello, I'm using HAProxy 1.4.22. I have the following haproxy.conf file: haproxy.conf --- global maxconn 10 daemon defaults mode http retries 1 contimeout 8000 clitimeout 12 srvtimeout 12 stats enable stats uri /haproxy-stats stats auth admin: option httpchk frontend http-in bind 16.9.13.39:80 maxconn 10 acl is_l1 hdr_end(host) -i l1.mydomain.com acl is_l2 hdr_end(host) -i l2.mydomain.com acl is_l3 hdr_end(host) -i l3.mydomain.com acl is_l0 hdr_end(host) -i mydomain.com use_backend lora1 if is_l1 use_backend lora2 if is_l2 use_backend lora3 if is_l3 use_backend lora0 if is_l0 default_backend lora0 backend lora0 balance roundrobin option forwardfor except 127.0.0.1 # stunnel already adds the header server s0 127.0.0.1:5000 check inter 6 backend lora1 balance source option forwardfor except 127.0.0.1 # stunnel already adds the header server s1 127.0.0.1:5001 check inter 6 backend lora2 balance source option forwardfor except 127.0.0.1 # stunnel already adds the header server s2 127.0.0.1:5002 check inter 6 backend lora3 balance source option forwardfor except 127.0.0.1 # stunnel already adds the header server s3 127.0.0.1:5003 check inter 6 haproxy.conf end --- It all works fine. Except, for some 0.2% of the clients. Sometimes when the request comes for l1, l2 or l3. For example: http://l3.mydomain.com/something and HAProxy does not match the domain name for some reason and uses default backend instead. I have set up logging in my application and it reports that hostname on the receiving end is in fact l3.mydomain.com. Here are the headers that my application receives: --- headers captured by Node.js/Express application -- host: 'l3.mydomain.com', 'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4', accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', referer: ' http://mydomain.com/menu/2034414/e2e1abb5500ed51391d6351b1cf03695', 'accept-encoding': 'gzip,deflate,sdch', 'accept-language': 'en-US,en;q=0.8', 'accept-charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'x-proxy-id': '1407537728', 'x-forwarded-for': '10.201.4.168', via: '1.1 10.201.255.254 (Mikrotik HttpProxy)' -- I don't know what are the actual headers HAProxy receives from the client, but I assume this Mikrotik proxy is mangling it somehow and HAProxy fails to match the hostname. Is there some way to enable logging only for situation when none of the acl rules is matched and default backend is selected? Thanks, -- Milan Babuskov guacosoft.com
