Re: odd 10 seconds in websocket negotiation with more than one backend server
On 6/12/2013 10:40 AM, Lukas Tribus wrote: Hi Ramin, When I have two servers in the backend haproxy makes connections to both websocket backends and exactly 10 seconds after the client had negotiated the connection to haproxy the client considers itself connected to socket.io application. This is very odd, but I also have to admit I don't know anything about websocket. Could you start haproxy in debug mode, reproduce it and post the debug output? Thanks Lukas and Willy for giving the config the once over. In regards to haproxy needing to hunt for a working server, that shouldn't be the case. Each server individually in the backend worked normally when it was the only server in rotation. I ran two different tests. One with just one server in the rotation and a second with all three in the rotation. I also did a tcpdump for the frontend and backend as well as haproxy in debug. I did turn off check in the backend to keep the tcpdumps cleaner. Files are in the URL below and 1_ is one server and 3_ is three servers. I bumped my haproxy 1.5 deb to the June 11th snapshot w/ pcre, ssl, and zlib. Ubuntu 12.04 LTS, 64bit, m1.small on EC2. http://badapple.net/haproxy/ After spending a number of hours on this, I'm leaning towards an application side problem as Willy suggested. There were some mentions of early Socket.io not working properly with haproxy. Our local version should have those fixes, but it is more than a year behind upstream. I think my next step is set up a current dead simple websocket application and try to replicate the problem without the unknowns our application brings. Ramin
RE: odd 10 seconds in websocket negotiation with more than one backend server
Hi Ramin, When I have two servers in the backend haproxy makes connections to both websocket backends and exactly 10 seconds after the client had negotiated the connection to haproxy the client considers itself connected to socket.io application. This is very odd, but I also have to admit I don't know anything about websocket. Could you start haproxy in debug mode, reproduce it and post the debug output? Lukas
odd 10 seconds in websocket negotiation with more than one backend server
I'm seeing an very odd and consistent 10 second time to negotiate a websocket connection when using HAproxy, but only when I have more than one server in the rotation. Going directly to the server from the browser, I connect immediately. With only one server in the haproxy backend, I connect immediately in the browser. When I have two servers in the backend haproxy makes connections to both websocket backends and exactly 10 seconds after the client had negotiated the connection to haproxy the client considers itself connected to socket.io application. I added a stick-table based on src thinking it might help which didn't change the behavior. I tested in haproxy 1.4 (whatever the latest 12.04 LTS is shipping) and the June 1st snapshot of 1.5. Same behavior in both cases. haproxy server is a 64bit m1.small instance on EC2. I suspect that there is something off if in my cobbled together config from various blogs and serverfault answers. I can also get some proper tcpdump later tonight too. Ramin # PUPPETHEADER: This file is owned by Puppet. # vim:syntax=haproxy #- # global settings, start global daemon gid 0 maxconn 4000 pidfile /var/run/haproxy.pid stats socket /var/lib/haproxy/stats uid 0 ulimit-n 8192 log 127.0.0.1 local0 notice # global settings, end #- #- # default settings, start defaults mode http log global option httplog option http-server-close option dontlognull option redispatch option contstats retries 3 backlog 1 timeout client 25s timeout connect 5s timeout server 25s # timeout tunnel available in ALOHA 5.5 or HAProxy 1.5-dev10 and higher # timeout tunnel3600s timeout http-keep-alive 1s timeout http-request15s timeout queue 30s timeout tarpit 60s default-server inter 3s rise 2 fall 3 option forwardfor # default settings, end #- #- # frontend settings, start of ws frontend ws bind *:80 default_backend ws # frontend settings, end of ws #- #- # backend settings, start of ws backend ws balance roundrobin stick store-request src stick-table type ip size 200k expire 20m server fe01 fe01.stage.usw2.example.com:8000 maxconn 1000 check server fe03 fe03.stage.usw2.example.com:8000 maxconn 1000 check # backend settings, end of ws #-
