Re: HAProxy is incompatible with WebSocket protocol revision 76?
Hi, On Thu, Jul 01, 2010 at 03:05:15PM +0300, Juhani Åhman wrote: Can you start wireshark or tcpdump session with a working websockets setup and offer a download of this file, so that we can take alook what's differrent. you can download wireshark dump here. In it I'm connecting straight to the backend server without haproxy. When I monitored the backend server with haproxy in the middle, I could not get any websocket related packets through. haproxy just discards them. http://a47.org/ws_packets It first establishes a websocket connection, receives message PING from server, send message PONG and then closes the connection. Thanks for the trace, but could you please take the same one between the browser and haproxy ? It's the most important one because it's the one where we should see why nothing happens (maybe a 400 bad request or something like this). Thanks, Willy
Re: HAProxy is incompatible with WebSocket protocol revision 76?
Hi again, On Thu, Jul 01, 2010 at 02:47:46PM +0200, Willy Tarreau wrote: Hi, On Thu, Jul 01, 2010 at 03:05:15PM +0300, Juhani Åhman wrote: Can you start wireshark or tcpdump session with a working websockets setup and offer a download of this file, so that we can take alook what's differrent. you can download wireshark dump here. In it I'm connecting straight to the backend server without haproxy. When I monitored the backend server with haproxy in the middle, I could not get any websocket related packets through. haproxy just discards them. http://a47.org/ws_packets It first establishes a websocket connection, receives message PING from server, send message PONG and then closes the connection. Thanks for the trace, but could you please take the same one between the browser and haproxy ? It's the most important one because it's the one where we should see why nothing happens (maybe a 400 bad request or something like this). When I send your trace through haproxy, I see correct exchanges between the client and the server. The only thing that *might* be causing trouble is the connection: close header which is added. Could you please try to configure your instance with both following options set : option http-server-close option http-pretend-keepalive and without option httpclose ? In my case, I get an *exact* copy of the client's request on the server, as well as an exact copy of the server's response on the client, and both can continue to talk uninterrupted, so I don't know why it does not work on your side. BTW I'm on haproxy 1.4.8 too. Regards, willy
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Thu, 2010-07-01 at 15:02 +0200, Willy Tarreau wrote: Thanks for the trace, but could you please take the same one between the browser and haproxy ? It's the most important one because it's the one where we should see why nothing happens (maybe a 400 bad request or something like this). When I send your trace through haproxy, I see correct exchanges between the client and the server. The only thing that *might* be causing trouble is the connection: close header which is added. Could you please try to configure your instance with both following options set : option http-server-close option http-pretend-keepalive and without option httpclose ? Here is new trace http://a47.org/ws_packets3 with no option httpclose option http-server-close option http-pretend-keepalive and otherwise the same config I pasted before. I still can't get anything through. In my case, I get an *exact* copy of the client's request on the server, as well as an exact copy of the server's response on the client, and both can continue to talk uninterrupted, so I don't know why it does not work on your side. BTW I'm on haproxy 1.4.8 too. I'm running Ubuntu's unstable binary (https://launchpad.net/ubuntu/maverick/amd64/haproxy/1.4.8-1), perhaps there's something wonky in it that's breaking it. I could try compiling from source later on and test with that.
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Thu, Jul 01, 2010 at 05:15:05PM +0300, Juhani Åhman wrote: On Thu, 2010-07-01 at 15:02 +0200, Willy Tarreau wrote: Thanks for the trace, but could you please take the same one between the browser and haproxy ? It's the most important one because it's the one where we should see why nothing happens (maybe a 400 bad request or something like this). When I send your trace through haproxy, I see correct exchanges between the client and the server. The only thing that *might* be causing trouble is the connection: close header which is added. Could you please try to configure your instance with both following options set : option http-server-close option http-pretend-keepalive and without option httpclose ? Here is new trace http://a47.org/ws_packets3 with no option httpclose option http-server-close option http-pretend-keepalive and otherwise the same config I pasted before. I still can't get anything through. In my case, I get an *exact* copy of the client's request on the server, as well as an exact copy of the server's response on the client, and both can continue to talk uninterrupted, so I don't know why it does not work on your side. BTW I'm on haproxy 1.4.8 too. I'm running Ubuntu's unstable binary (https://launchpad.net/ubuntu/maverick/amd64/haproxy/1.4.8-1), perhaps there's something wonky in it that's breaking it. I could try compiling from source later on and test with that. no there's no reason it would be broken. However I think I have found some clue. In the trace you sent, the data was sent from the server along with the headers, though no intermediate proxy nor reverse-proxy would do that because you only know you're forwarding a websocket connection once you get the response. So I'll have to recheck the recent websocket spec updates and maybe discuss with Ian Hickson about possible issues. If the server really *needs* to receive the data before the handshake is complete, then a content length must be provided, and probably that the spec mentions it, or maybe not, in which case it breaks HTTP again :-/ In the mean time, I think that the only thing you can do is to use plain TCP mode. But in TCP mode you can match HTTP traffic if required, and switch to an http backend (for the rest of the traffic). Regards, Willy
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Thu, Jul 01, 2010 at 04:47:33PM +0200, Willy Tarreau wrote: However I think I have found some clue. In the trace you sent, the data was sent from the server along with the headers, though no intermediate proxy nor reverse-proxy would do that because you only know you're forwarding a websocket connection once you get the response. So I'll have to recheck the recent websocket spec updates and maybe discuss with Ian Hickson about possible issues. Well, I got a response from Ian. In short, he says that the HTTP breakage is intentional in order to ensure that there is no intermediate between the client and the server, that it was a design mistake to make the protocol look like HTTP and that you have to use another port for your WebSocket traffic ! I'm trying to make him realize what what he says implies, but it's not easy, as it's not the first time that the HTTP compatibility is intentionally broken. There are people in the working group who believe they have a clue about security and who try to ensure that the handshake is very difficult because they believe it strengthens the protocol, which is terribly wrong. After all, there are already other protocols making use of HTTP+something else after an Upgrade header which work correctly simply because they respect HTTP semantics :-/ What a waste of energy. In the end, I think that websocket adoption might be broken into two sets : - the ones who need a load balancer and/or a reverse proxy and who will prefer to stick to their current protocols instead of using websocket - the ones who need a load balancer and/or a reverse proxy and who will hack them to support it anyway and who'll run into security issues I don't think there will be many users without such equipments in front of them, so the correctly working implementations will not be the most deployed ones. That's a shame because the original goal of the protocol *was* to put an end to current ugly hacks, and it seems it will finally fail because there's nobody concerned about HTTP in this working group ! Thus, I don't know what to say. I could just forward you Ian's recommendation, which is to use another port for your websocket, eventhough I find this stupid and counter-productive. But if the protocol takes this direction, I don't know what to do with it... Willy
Re: HAProxy is incompatible with WebSocket protocol revision 76?
Dear Juhani, On Mit 30.06.2010 15:09, Juhani Åhman wrote: Hi everyone I've recently tried to use HAProxy as a reverse proxy to a WebSocket server like Moskovitz in this thread (http://www.mail-archive.com/haproxy@formilux.org/msg02754.html), but I've run into problems. I can only get websockets proxied in http mode when I'm using a browser which speaks revision pre-76 draft websockets (eg. Chromium 5.0.375.70). When using a rev. 76 browser (eg. Chromium 6.0.414.0 or later) the websocket connections seem to always fail. I think the reason for this is that rev. 76 is not backwards compatible with earlier ones as it adds new headers HTTP_SEC_WEBSOCKET_KEY1, HTTP_SEC_WEBSOCKET_KEY2, HTTP_SEC_WEBSOCKET_PROTOCOL and maybe some others too. Apparently HAProxy doesn't like these new headers unless I'm running in tcp mode. I've tested this with Eventlet library which understands both pre-76 and 76 revisions. Is there anyway to get rev. 76 websockets working with HAProxy without using tcp mode? Please can you try to run haproxy with -p flag similar as in http://www.mail-archive.com/haproxy@formilux.org/msg02793.html Please can you also send us the output of haproxy -vv BR Aleks
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Wed, 2010-06-30 at 14:46 +0200, Aleksandar Lazic wrote: Dear Juhani, On Mit 30.06.2010 15:09, Juhani Åhman wrote: Hi everyone I've recently tried to use HAProxy as a reverse proxy to a WebSocket server like Moskovitz in this thread (http://www.mail-archive.com/haproxy@formilux.org/msg02754.html), but I've run into problems. I can only get websockets proxied in http mode when I'm using a browser which speaks revision pre-76 draft websockets (eg. Chromium 5.0.375.70). When using a rev. 76 browser (eg. Chromium 6.0.414.0 or later) the websocket connections seem to always fail. I think the reason for this is that rev. 76 is not backwards compatible with earlier ones as it adds new headers HTTP_SEC_WEBSOCKET_KEY1, HTTP_SEC_WEBSOCKET_KEY2, HTTP_SEC_WEBSOCKET_PROTOCOL and maybe some others too. Apparently HAProxy doesn't like these new headers unless I'm running in tcp mode. I've tested this with Eventlet library which understands both pre-76 and 76 revisions. Is there anyway to get rev. 76 websockets working with HAProxy without using tcp mode? Please can you try to run haproxy with -p flag similar as in http://www.mail-archive.com/haproxy@formilux.org/msg02793.html Please can you also send us the output of haproxy -vv BR Aleks haproxy -vv output fuzzyb...@stacker:~$ haproxy -vv HA-Proxy version 1.4.8 2010/06/16 Copyright 2000-2010 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. my config global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 16384 #chroot /usr/share/haproxy user haproxy group haproxy daemon #debug #quiet defaults mode http log global option httplog option dontlognull retries 3 option redispatch #option httpclose option forwardfor maxconn 1 timeout connect 5s timeout client 60s timeout server 60s frontend all 0.0.0.0:80 timeout client 7d option nolinger acl websocket hdr(Upgrade) -i WebSocket AND hdr(Connection) -i Upgrade use_backend ws if websocket default_backend django #backend nginx # server media localhost:8080 check backend django server django localhost:8081 check backend ws timeout server 7d option nolinger server ws localhost:8081 check Enabling httpclose seems to break websockets with both rev 76 and pre-76. Disabling option nolinger does not have effect to websockets. here's the debug output with -p flag. tough, I don't understand what it was supposed to do. It didn't output anything to anywhere. r...@stacker:/var/run# haproxy -f /etc/haproxy/haproxy.cfg -d -p /var/run/haproxy-private.pid -sf $(/var/run/haproxy-private.pid) bash: /var/run/haproxy-private.pid: No such file or directory Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. Using sepoll() as the polling mechanism. :all.accept(0004)=0006 from [127.0.0.1:50520] :all.clireq[0006:]: GET /35/ HTTP/1.1 :all.clihdr[0006:]: Host: localhost :all.clihdr[0006:]: Connection: keep-alive :all.clihdr[0006:]: Referer: http://localhost/ :all.clihdr[0006:]: Cache-Control: max-age=0 :all.clihdr[0006:]: Accept: application/xml,application/xhtml +xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 :all.clihdr[0006:]: User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/6.0.452.0 Safari/534.2 :all.clihdr[0006:]: Accept-Encoding: gzip,deflate,sdch :all.clihdr[0006:]: Accept-Language: en-US,en;q=0.8 :all.clihdr[0006:]: Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 :all.clihdr[0006:]: Cookie: csrftoken=4a6ff9be8c08d063178b5b0f0efde0a7 :django.srvrep[0006:0007]: HTTP/1.1 200 OK :django.srvhdr[0006:0007]: Vary: Cookie, Accept-Encoding :django.srvhdr[0006:0007]: Content-Length: 11003 :django.srvhdr[0006:0007]: Content-Type: text/html; charset=utf-8 :django.srvhdr[0006:0007]: Content-Encoding: gzip :django.srvhdr[0006:0007]: Date: Wed, 30 Jun 2010 13:51:25 GMT :django.srvhdr[0006:0007]: Connection: keep-alive
Re: HAProxy is incompatible with WebSocket protocol revision 76?
Dear Juhani, On Mit 30.06.2010 16:59, Juhani Åhman wrote: On Wed, 2010-06-30 at 14:46 +0200, Aleksandar Lazic wrote: Dear Juhani, On Mit 30.06.2010 15:09, Juhani Åhman wrote: Hi everyone I've recently tried to use HAProxy as a reverse proxy to a WebSocket server like Moskovitz in this thread (http://www.mail-archive.com/haproxy@formilux.org/msg02754.html), but I've run into problems. I can only get websockets proxied in http mode when I'm using a browser which speaks revision pre-76 draft websockets (eg. Chromium 5.0.375.70). When using a rev. 76 browser (eg. Chromium 6.0.414.0 or later) the websocket connections seem to always fail. [snipp] Please can you try to run haproxy with -p flag similar as in http://www.mail-archive.com/haproxy@formilux.org/msg02793.html Please can you also send us the output of haproxy -vv BR Aleks haproxy -vv output fuzzyb...@stacker:~$ haproxy -vv HA-Proxy version 1.4.8 2010/06/16 Copyright 2000-2010 Willy Tarreau w...@1wt.eu [snipp] my config global [snipp] defaults mode http log global option httplog option dontlognull retries 3 option redispatch #option httpclose option http-server-close option forwardfor maxconn 1 [snipp] I think you will need the client-keepalive option. I'am not sure if this is the only option to activate this, I hope anybody else on this list can say more about this. Hth Aleks
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Wed, 2010-06-30 at 16:18 +0200, Aleksandar Lazic wrote: I think you will need the client-keepalive option. I'am not sure if this is the only option to activate this, I hope anybody else on this list can say more about this. Hth Aleks I tested enabling option httpclose or option http-server-close, but enabling either one or both would break websockets with both rev 76 and pre-76.
Re: HAProxy is incompatible with WebSocket protocol revision 76?
On Mit 30.06.2010 17:38, Juhani Åhman wrote:
On Wed, 2010-06-30 at 16:18 +0200, Aleksandar Lazic wrote:
I think you will need the client-keepalive option. I'am not sure if
this is the only option to activate this, I hope anybody else on this
list can say more about this.
Hth
Aleks
I tested enabling option httpclose or option http-server-close, but
enabling either one or both would break websockets with both rev 76 and
pre-76.
Why does the client connect 3 times to the server, with differnt keys?
Is this normal behaviour?
:all.accept(0004)=0006 from [127.0.0.1:50520]
0001:all.accept(0004)=0008 from [127.0.0.1:50522]
0001:all.clihdr[0008:]: Sec-WebSocket-Key1: V(7823t 6E{)8U8K48^%
0001:all.clihdr[0008:]: Sec-WebSocket-Key2: 3M5=wYx1y1818t9j:O82
G
0002:all.accept(0004)=000a from [127.0.0.1:50535]
0002:all.clihdr[000a:]: Sec-WebSocket-Key1: 196J;n 7^1M101o50
0002:all.clihdr[000a:]: Sec-WebSocket-Key2: 2 7 90M 5B 0 8
E69 2
#
Can you start wireshark or tcpdump session with a working websockets
setup and offer a download of this file, so that we can take alook
what's differrent.
BR
Aleks
