Re: Hello from Mozilla
On Wed, 5 Aug 2009, Amos Jeffries wrote: On Wed, 5 Aug 2009 00:57:18 + (UTC), Ian Hickson i...@hixie.ch wrote: On Thu, 30 Jul 2009, Robert Collins wrote: On Wed, 2009-07-29 at 23:48 +, Ian Hickson wrote: Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. 10.1.2 101 Switching Protocols The server understands and is willing to comply with the client's request, via the Upgrade message header field (section 14.42), for a change in the application protocol being used on this connection. The server will switch protocols to those defined by the response's Upgrade header field immediately after the empty line which terminates the 101 response. So the server has to send a response before switching. The above text has no normative conformance criteria (the server will is an informative statement that is unsupported as far as I can tell). Point to note: if webserver fails to switch the WebSockets handshake to verify said switch actually occurred will fail. Detecting such bad servers and acting properly. If web server successfully switched then WebSockets client needs to handle the HTTP response correctly and begin to do a WebSockets native handshake and verify the link is safe _after_ the 101 message is received. I don't understand what you are trying to say here. In any case, the WebSocket protocol handshake does do a complete HTTP Upgrade including the empty line, it just requires that the handshake be done in a particular way. It's profiling the HTTP spec. No, our whole point repeated over and over in different ways is that the specd description for the upgrade request is _incomplete_. And does not account for an enormous number of byte-level variations. All of which can be safely discarded without affecting security. Being sensitive to whether the server replies 101 Blah versus 101 blah absolutely cripples WebSockets. We want to help you fix this problem. I still do not understand why anything gets crippled. Maybe you could show an example of how you expect this problem to occur? This is because if it doesn't, how can it say 'I won't upgrade'. It can just not upgrade. Returning anything but the correct handshake will be treated as a failed connection by the WebSocket client. Correct. So why would it need to say I won't upgrade? On Thu, 30 Jul 2009, Robert Collins wrote: Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Can't the web page just send an ajax request to corp.example.com anyway? It can't read the response, no. Can you explain this please? AFAIK, Sending a request then closing the connection immediately is the only way said response might be unreadable. I don't understand what you mean. A script on a Web page can cause a GET request to be sent to an arbitrary URL, but it has no way to obtain the contents of the result of that request unless that server opts in (using CORS) to allowing the script to see the contents of the response. Dumping the ability to look like HTTP altogether is your best bet IMO: even on port 443. For port 443, someone running a websocket server should just not run an HTTPS server there. Or define port 815 as websocket/s. Port 815 is the preferred port, but in certain situations, only ports 80 and 443 are allowed, and port 80 is mitm'ed, so that only leaves 443. You don't always have the option of using a different host, that's why we need to share the port sometimes. (There have also been statements to the effect that just sending a new protocol over ports 80 or 443 without using the Upgrade mechanism would be a violation of the semantics of those ports.) Understood. However strange protocols (torrent is one example) on port 80 can be detected and dealt with easily. Something that appears to be perfect HTTP cannot be detected as non-HTTP. When HTTP operations are done to the traffic it will break badly and in nasty invisible ways. Could you explain how a WebSocket connection can break in an invisible way? An example would be very helpful here. If its
Re: Hello from Mozilla
Ian Hickson wrote: On Wed, 5 Aug 2009, Amos Jeffries wrote: On Wed, 5 Aug 2009 00:57:18 + (UTC), Ian Hickson i...@hixie.ch wrote: On Thu, 30 Jul 2009, Robert Collins wrote: On Wed, 2009-07-29 at 23:48 +, Ian Hickson wrote: Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. 10.1.2 101 Switching Protocols The server understands and is willing to comply with the client's request, via the Upgrade message header field (section 14.42), for a change in the application protocol being used on this connection. The server will switch protocols to those defined by the response's Upgrade header field immediately after the empty line which terminates the 101 response. So the server has to send a response before switching. The above text has no normative conformance criteria (the server will is an informative statement that is unsupported as far as I can tell). Point to note: if webserver fails to switch the WebSockets handshake to verify said switch actually occurred will fail. Detecting such bad servers and acting properly. If web server successfully switched then WebSockets client needs to handle the HTTP response correctly and begin to do a WebSockets native handshake and verify the link is safe _after_ the 101 message is received. I don't understand what you are trying to say here. In any case, the WebSocket protocol handshake does do a complete HTTP Upgrade including the empty line, it just requires that the handshake be done in a particular way. It's profiling the HTTP spec. No, our whole point repeated over and over in different ways is that the specd description for the upgrade request is _incomplete_. And does not account for an enormous number of byte-level variations. All of which can be safely discarded without affecting security. Being sensitive to whether the server replies 101 Blah versus 101 blah absolutely cripples WebSockets. We want to help you fix this problem. I still do not understand why anything gets crippled. Maybe you could show an example of how you expect this problem to occur? Your protocol definition used byte-level. At the byte-level 'b' (0x97 IIRC) does not equal 'B' (0x65 IIRC). Thus the response is a different byte pattern and a failed WebSocket connection. One very real example of this would be the web server or an fully WebSocket capable intermediary sending back bytes Your spec section 3.1 sub 12 says: 12. Read the first 85 bytes from the server. If the connection closes before 85 bytes are received, or if the first 85 bytes aren't exactly equal to the following bytes, then fail the Web [ note the words _exactly equal_ ] Socket connection and abort these steps. 48 54 54 50 2f 31 2e 31 20 31 30 31 20 57 65 62 20 53 6f 63 6b 65 74 20 50 72 6f 74 6f 63 6f 6c 20 48 61 6e 64 73 68 61 6b 65 0d 0a 55 70 67 72 61 64 65 3a 20 57 65 62 53 6f 63 6b 65 74 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 61 64 65 0d 0a example #1 suppose there was an intermediary translating websockets-over-http to websockets-port-81 which used HTTP to format said headers of confirmation. In all other ways it is fully WebSockets compliant. But sends byte 18 as 73 (s) instead of 53 (S). Boom! The entire application is not WebSockets compliant and will fail every single transaction that goes through it. example #2 is where the traffic is processed by an HTTP-only intermediary which sees the 'Upgrade:' header and flags the connection for transparent pass-thru (This by the way is the desirable method of making Squid support WebSockets). Being a good HTTP relay it accepts these bytes: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade It violates HTTP by omitting the Via and other headers your spec omits to handle. And passes these on: HTTP/1.1 101 Web Socket Protocol Handshake Connection: Upgrade Upgrade: WebSocket then moves to tunnel mode for you. Bang. Another whole network isolated from WebSockets. This is because if it doesn't, how can it say 'I won't upgrade'. It can just not upgrade. Returning anything but the correct handshake will be treated as a failed connection by the WebSocket client. Correct. So why would it need to say I won't upgrade? To inform MITM that the upgrade is not going to happen and the links they have open maybe used for other HTTP things without wasting network resources tearing them down and rebuilding. On Thu, 30 Jul 2009, Robert Collins wrote: Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a
Re: Hello from Mozilla
On Thu, 30 Jul 2009, Robert Collins wrote: On Wed, 2009-07-29 at 23:48 +, Ian Hickson wrote: Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. 10.1.2 101 Switching Protocols The server understands and is willing to comply with the client's request, via the Upgrade message header field (section 14.42), for a change in the application protocol being used on this connection. The server will switch protocols to those defined by the response's Upgrade header field immediately after the empty line which terminates the 101 response. So the server has to send a response before switching. The above text has no normative conformance criteria (the server will is an informative statement that is unsupported as far as I can tell). In any case, the WebSocket protocol handshake does do a complete HTTP Upgrade including the empty line, it just requires that the handshake be done in a particular way. It's profiling the HTTP spec. This is because if it doesn't, how can it say 'I won't upgrade'. It can just not upgrade. Returning anything but the correct handshake will be treated as a failed connection by the WebSocket client. On Thu, 30 Jul 2009, Robert Collins wrote: Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Can't the web page just send an ajax request to corp.example.com anyway? It can't read the response, no. And if it can't, why doesn't the same browser security model prevent websockets being used in the same way? The same-origin model would prevent any WebSocket traffic at all, since it's a different scheme than the document. Now, Web Socket has a multi-layered approach to dealing with this. - there is the handshake, which requires that the server respond with a very specific set of bytes, thus guaranteeing that the server is in fact WebSocket-aware. Any wildcard part to this handshake increases the risk that there will be a server somewhere that can be tricked. For example, if the handshake were HTTP followed by anything followed by WebSocket, then some HTTP servers could be tricked into doing the handshake -- for example, the response to GET /WebSocket on the ietf.org host (not www.ietf.org) includes the word WebSocket in the response. HTTP servers aren't the only concern, of course; we want the handshake to be as secure as possible against any other protocol that may exist on any server that may be deployed. We don't know what's out there (especially in intranets), so the handshake has to be pretty stringent to make it as unlikely as possible. Dumping the ability to look like HTTP altogether is your best bet IMO: even on port 443. For port 443, someone running a websocket server should just not run an HTTPS server there. Or define port 815 as websocket/s. Port 815 is the preferred port, but in certain situations, only ports 80 and 443 are allowed, and port 80 is mitm'ed, so that only leaves 443. You don't always have the option of using a different host, that's why we need to share the port sometimes. (There have also been statements to the effect that just sending a new protocol over ports 80 or 443 without using the Upgrade mechanism would be a violation of the semantics of those ports.) - there is origin checking and location checking, and the location checking isn't just an echo of the original request's data. This makes causing the server to send back particular data harder. (It's also part of our cross-origin security model and our shared hosting support; but that's a separate discussion.) - there is the framing, which does provide a modicum of protection by forcing another byte in front of the first author-controlled packet sent. (This isn't really a security feature, it's just a lucky accident of the framing that we needed to turn TCP streams into packets.) To summarise, if I have it right, you want to make sure that *the browser* can be sure that corp.example.com really was intending to be a websocket server when it opened the connection. Right. If its not, it should bail. That seems reasonable :). We can definitely (and have
Re: Hello from Mozilla
On Wed, 5 Aug 2009 00:57:18 + (UTC), Ian Hickson i...@hixie.ch wrote: On Thu, 30 Jul 2009, Robert Collins wrote: On Wed, 2009-07-29 at 23:48 +, Ian Hickson wrote: Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. 10.1.2 101 Switching Protocols The server understands and is willing to comply with the client's request, via the Upgrade message header field (section 14.42), for a change in the application protocol being used on this connection. The server will switch protocols to those defined by the response's Upgrade header field immediately after the empty line which terminates the 101 response. So the server has to send a response before switching. The above text has no normative conformance criteria (the server will is an informative statement that is unsupported as far as I can tell). Point to note: if webserver fails to switch the WebSockets handshake to verify said switch actually occurred will fail. Detecting such bad servers and acting properly. If web server successfully switched then WebSockets client needs to handle the HTTP response correctly and begin to do a WebSockets native handshake and verify the link is safe _after_ the 101 message is received. In any case, the WebSocket protocol handshake does do a complete HTTP Upgrade including the empty line, it just requires that the handshake be done in a particular way. It's profiling the HTTP spec. No, our whole point repeated over and over in different ways is that the specd description for the upgrade request is _incomplete_. And does not account for an enormous number of byte-level variations. All of which can be safely discarded without affecting security. Being sensitive to whether the server replies 101 Blah versus 101 blah absolutely cripples WebSockets. We want to help you fix this problem. This is because if it doesn't, how can it say 'I won't upgrade'. It can just not upgrade. Returning anything but the correct handshake will be treated as a failed connection by the WebSocket client. Correct. On Thu, 30 Jul 2009, Robert Collins wrote: Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Can't the web page just send an ajax request to corp.example.com anyway? It can't read the response, no. Can you explain this please? AFAIK, Sending a request then closing the connection immediately is the only way said response might be unreadable. I don't understand what you mean. And if it can't, why doesn't the same browser security model prevent websockets being used in the same way? The same-origin model would prevent any WebSocket traffic at all, since it's a different scheme than the document. Now, Web Socket has a multi-layered approach to dealing with this. - there is the handshake, which requires that the server respond with a very specific set of bytes, thus guaranteeing that the server is in fact WebSocket-aware. Any wildcard part to this handshake increases the risk that there will be a server somewhere that can be tricked. For example, if the handshake were HTTP followed by anything followed by WebSocket, then some HTTP servers could be tricked into doing the handshake -- for example, the response to GET /WebSocket on the ietf.org host (not www.ietf.org) includes the word WebSocket in the response. HTTP servers aren't the only concern, of course; we want the handshake to be as secure as possible against any other protocol that may exist on any server that may be deployed. We don't know what's out there (especially in intranets), so the handshake has to be pretty stringent to make it as unlikely as possible. Dumping the ability to look like HTTP altogether is your best bet IMO: even on port 443. For port 443, someone running a websocket server should just not run an HTTPS server there. Or define port 815 as websocket/s. Port 815 is the preferred port, but in certain situations, only ports 80 and 443 are allowed, and port 80 is mitm'ed, so that only leaves 443. You don't always have the option of using a different host, that's why we need to share the port
Re: Hello from Mozilla
tor 2009-07-30 klockan 10:26 +1000 skrev Robert Collins: In corporate networking, TLS MITM is a 'feature': company signed certificates are used to sign the TLS connection to the corporate firewall, and the firewall validate the SSL connection to the outside world. I haven't personally used this, so can't really say much more about it. Squid's ssl-bump feature can be used for this, but I believe browser config is needed (again on a corporate basis) to tell it that this is expected. Squid ssl-bump isn't yet very polished, but also far from alone in doing this. It is a fairly common thing in large corporations. The configuration needed for this SSL interception thing to run smoothly (in the eyes of users browsing https) is apart from the middleman doing it that the corporation adds a private CA to the browsers list of accepted CAs. Once that is done the corporation can freely spoof certificates for any server, completely breaking SSL end-to-end in ways the X.509 trust model is supposed to prevent.. but also throwing certificate based client authentication out the window as this requires end-to-end. In essence the level of trust you can place in SSL without manually inspecting each received certificate is the least amount of trust you have in any of the CAs installed in your browsers list of universally trusted CAs and all their CA delegations, official and unofficial.. Regards Henrik
Re: Hello from Mozilla
On Sat, 18 Jul 2009, Amos Jeffries wrote: But per HTTP specifications such body data is part of the HTTP Upgrade request and should be ignored if switching protocols as part of ignoring the HTTP request which contained the Upgrade request. Consider for example if the request is a POST request, containing a alternative simplex representation of the communication meant to be used by a non-WebSockets server. Ian: is that a relevant case? you can spec it in or out as needed. But I'm thinking out, since the idea here is to setup a TCP-link tunnel then perform authentication handshake inside it. The upgrade request is only to get the MITM/Surrogates to pass it correctly or fail it cleanly. I'm not sure what you're asking here, sorry. 3.1.5.4 A server receiving a valid HTTP Upgrade request to WebSockets and which does not accept that WebSockets upgrade MUST send back a 4xx or 5xx HTTP response as per section 3.1.5.2. There is no such requirement in HTTP. Handling Upgrade is purely optional and the server is free to ignore Upgrade. Upgrade is a please upgrade to one of these protocols if supported. Meaning HTTP does not care what gets returned... At the non-101 reply point its not usable as a WebSockets link. This bit is going slightly beyond HTTP in a compatible way and refers to the WebSockets-enabled server rejecting the socket connection. Ian mentioned many times WebSockets being 'opt-in'. This clause permits 'opt-in' to be denied by the server. It's irrelevant to non-WebSocket intermediaries and applies only to WebSocket origin servers. Perhapse that needs to be worded clearer. I'm not sure exactly what part isn't clear. If you mean some text in the Web Socket protocol draft, could you elaborate on what change you are requesting? The server SHOULD send only the following lines: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade While it's true HTTP does not mandate any headers for 1xx responses, including things like Server is a good idea in 101 responses, and there may also be other headers required by the HTTP operation such as Authentication-Info and possibly other headers as well in future specifications. Ah yes. Okay. Ian: anything else you find useful as required here as well. This is where you get the is the mystery server capable of WebSockets info. Being the _right_ one needs must separate and handled by the WebSockets handshake. Just only specify it at the string level. Specifying the order sent is fine but confusing because it implies order received. Specifying the order received beyond 101 is broken. I really don't understand what you are asking here. Do you mean that the handshake should allow the Upgrade and Connection lines to be sent in any order? Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Wed, 2009-07-15 at 00:51 +, Ian Hickson wrote: If there are any bytes allowed from the client or the server before the handshake starts, then it is no longer secure. The idea is to make sure you can't smuggle though payloads from other protocols, since otherwise you could use WebSocket to connect to services that aren't expecting it. Then don't use port 80! If you use port 80 you must expect the following: - many users will be unable to connect directly to your service - many users will think they are connecting directly to your service but will not actually be doing so AIUI websockets is: * TCP + * authentication I don't really see this having *anything* to do with HTTP. Perhaps I'm missing something fundamental, but as it stands, I think it would be more robust, and more secure to say: Websockets is on IANA port the authentication handshake for a websocket server is after that its a bidirectional stream of octects just like TCP If a browser needs to get through a firewall to connect to the websocket server, we recommend the use of a SOCKS proxy an HTTP proxy supporting the CONNECT method. What drives the desire to live on port 80? -Rob signature.asc Description: This is a digitally signed message part
Re: Hello from Mozilla
On Wed, 2009-07-29 at 23:48 +, Ian Hickson wrote: Surely once the HTTP server has decided that it can Upgrade, it doesn't actually need to worry about sending back something HTTP-valid at all, since we can just say the entire connection was always using the WebSocket protocol, and was never HTTP. 10.1.2 101 Switching Protocols The server understands and is willing to comply with the client's request, via the Upgrade message header field (section 14.42), for a change in the application protocol being used on this connection. The server will switch protocols to those defined by the response's Upgrade header field immediately after the empty line which terminates the 101 response. So the server has to send a response before switching. This is because if it doesn't, how can it say 'I won't upgrade'. -Rob signature.asc Description: This is a digitally signed message part
Re: Hello from Mozilla
On Thu, 30 Jul 2009, Robert Collins wrote: What drives the desire to live on port 80? Nothing. WebSocket will be on ports 81 and 815. The desire to share a port with HTTP comes from port 443 being the only port that is usable in some settings due to firewalls and mitm proxies. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Thu, 2009-07-30 at 00:00 +, Ian Hickson wrote: On Thu, 30 Jul 2009, Robert Collins wrote: What drives the desire to live on port 80? Nothing. WebSocket will be on ports 81 and 815. The desire to share a port with HTTP comes from port 443 being the only port that is usable in some settings due to firewalls and mitm proxies. Ok thanks. I hadn't read all the intervening messages :(. -Rob signature.asc Description: This is a digitally signed message part
Re: Hello from Mozilla
On Fri, 2009-07-17 at 10:00 +, Ian Hickson wrote: On Fri, 17 Jul 2009, Adrian Chadd wrote: 2009/7/17 Ian Hickson i...@hixie.ch: That way you are still speaking HTTP right until the protocol change occurs, so any and all HTTP compatible changes in the path(s) will occur. As mentioned earlier, we need the handshake to be very precisely defined because otherwise people could trick unsuspecting servers into opting in, or rather appearing to opt in, and could then send all kinds of commands down to those servers. Would you please provide an example of where an unsuspecting server is tricked into doing something? Sure. Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Can't the web page just send an ajax request to corp.example.com anyway? And if it can't, why doesn't the same browser security model prevent websockets being used in the same way? Now, Web Socket has a multi-layered approach to dealing with this. - there is the handshake, which requires that the server respond with a very specific set of bytes, thus guaranteeing that the server is in fact WebSocket-aware. Any wildcard part to this handshake increases the risk that there will be a server somewhere that can be tricked. For example, if the handshake were HTTP followed by anything followed by WebSocket, then some HTTP servers could be tricked into doing the handshake -- for example, the response to GET /WebSocket on the ietf.org host (not www.ietf.org) includes the word WebSocket in the response. HTTP servers aren't the only concern, of course; we want the handshake to be as secure as possible against any other protocol that may exist on any server that may be deployed. We don't know what's out there (especially in intranets), so the handshake has to be pretty stringent to make it as unlikely as possible. Dumping the ability to look like HTTP altogether is your best bet IMO: even on port 443. For port 443, someone running a websocket server should just not run an HTTPS server there. Or define port 815 as websocket/s. - there is origin checking and location checking, and the location checking isn't just an echo of the original request's data. This makes causing the server to send back particular data harder. (It's also part of our cross-origin security model and our shared hosting support; but that's a separate discussion.) - there is the framing, which does provide a modicum of protection by forcing another byte in front of the first author-controlled packet sent. (This isn't really a security feature, it's just a lucky accident of the framing that we needed to turn TCP streams into packets.) To summarise, if I have it right, you want to make sure that *the browser* can be sure that corp.example.com really was intending to be a websocket server when it opened the connection. If its not, it should bail. That seems reasonable :). We can definitely (and have been :)) advising on how to do that with HTTP, but it means *having a decent HTTP stack*. The response to a HTTP request is always an HTTP response. Upgrade: changes the way the wire behaves after that. Yes, but the client is a WebSocket client, not an HTTP client, so why would it send anything but the WebSocket handshake? Because its asking an HTTP server to upgrade. That makes it an HTTP client and a WebSocket client... as in fact you note below .. .. No. I'm intending to do stuff over port 81. There is a desire in certain cases to be able to share this traffic with servers running on port 80, and in those cases, the content sent and the content required to be returned by the server is valid HTTP until the Upgrade succeeds, at which point it isn't HTTP. Ian, are you absolutely certain that everywhere you use the internet, there is no man in the middle between you and the server you're speaking to? In the case of TLS connections, yes. In corporate networking, TLS MITM is a 'feature': company signed certificates are used to sign the TLS connection to the corporate firewall, and the firewall validate the SSL connection to the outside world. I haven't personally used this, so can't really say much more about it. Squid's ssl-bump feature can be used for this, but I believe browser config is needed (again on a corporate basis) to tell it that this is expected. -Rob signature.asc
Re: Hello from Mozilla
I missed that Ian was still talking about using port 80. I think that's broken / more trouble than it's worth, for the reasons Adri is going through. If you have to tunnel using an existing port, use 443 (with null encryption if you're worried about overhead, but still want to authenticate the endpoint). Even then, Wifi hotspots are probably going to redirect you, but using 443 should be a last-gasp measure anyway. Cheers, On 17/07/2009, at 3:18 PM, Adrian Chadd wrote: 2009/7/17 Ian Hickson i...@hixie.ch: That way you are still speaking HTTP right until the protocol change occurs, so any and all HTTP compatible changes in the path(s) will occur. As mentioned earlier, we need the handshake to be very precisely defined because otherwise people could trick unsuspecting servers into opting in, or rather appearing to opt in, and could then send all kinds of commands down to those servers. Would you please provide an example of where an unsuspecting server is tricked into doing something? Ian, don't you see and understand the semantic difference between speaking HTTP and speaking a magic bytecode that is intended to look HTTP-enough to fool a bunch of things until the upgrade process occurs ? Don't you understand that the possible set of things that can go wrong here is quite unbounded ? Don't you understand the whole reason for known ports and protocol descriptions in the first place? Apparently not. Ok. Look at this. The byte sequence GET / HTTP/1.0\r\nHost: foo\r\nConnection: close\r\n\r\n is not byte equivalent to the sequence GET / HTTP/1.0\r\nConnection: close\r\nHost: foo\r\n\r\n The same byte sequence interpreted as a HTTP protocol exchange is equivalent. There's a mostly-expected understanding that what happens over port 80 is HTTP. The few cases where that has broken (specifically Shoutcast, but I do see other crap on port 80 from time to time..) has been by people who have implemented a mostly HTTP looking protocol, tested that it mostly works via a few gateways/firewalls/proxies, and then deployed it. My suggestion is to completely toss the whole pretend to be HTTP thing out of the window and look at extending or adding a new HTTP mechanism for negotiating proper tunneling on port 80. If this involves making CONNECT work on port 80 then so be it. Redesigning HTTP is really much more work than I intend to take on here. HTTP already has an Upgrade mechanism, reusing it seems the right thing to do. What you intend to take on here and what should be taken on here is very relevant. You're intending to do stuff over tcp/80 which looks like HTTP but isn't HTTP. Everyone who implements anything HTTP gateway related (be it a transparent proxy, a firewall, a HTTP router, etc) suddenly may have to implement your websockets stuff as well. So all of a sudden your attempt to not extend HTTP ends up extending HTTP. The point is, there may be a whole lot of stuff going on with HTTP implementations that you're not aware of. Sure, but with the except of man-in-the-middle proxies, this isn't a big deal -- the people implementing the server side are in control of what the HTTP implementation is doing. That may be your understanding of how the world works, but out here in the rest of the world, the people who deploy the edge and the people who deploy the core may not be the same people. There may be a dozen layers of red tape, equipment lifecycle, security features, etc, that need to be handled before websockets happy stuff can be deployed everywhere it needs to. Please don't discount man-in-the-middle -anything- as being easy to deal with. In all cases except a man-in-the-middle proxy, this seems to be what we do. I'm not sure how we can do anything in the case of such a proxy, since by definition the client doesn't know it is present. .. so you're still not speaking HTTP? Ian, are you absolutely certain that everywhere you use the internet, there is no man in the middle between you and the server you're speaking to? Haven't you ever worked at any form of corporate or enterprise environment? What about existing captive portal deployments like wifi hotspots, some of which still use squid-2.5 (eww!) as their http firewall/proxy to control access to the internet? That stuff is going to need upgrading sure, but I'd rather see the upgrade happen once to a well thought out and reasonably well designed protocol, versus having lots of little upgrades need to occur because your HTTP but not quite HTTP exchange on port 80 isn't thought out enough. Adrian -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
fre 2009-07-17 klockan 16:32 +1200 skrev Amos Jeffries: 3.1.5.1 Connect to the server given by /host/ on port 80 and ask it to upgrade from HTTP to WebSockets. The client must send only the following lines: GET /thepath/ HTTP/1.1 Host: /host/ Upgrade: WebSockets Connection: Upgrade terminated by two CRLF as per HTTP specification. Per HTTP specifications the client SHOULD send more lines, in specific the User-Agent. 3.1.5.2 A server receiving the request SHOULD ignore any other headers received with the request. If any data following the headers is received then fail the WebSocket connection. The server MUST discard any body data received with an Upgrade request. It MAY respond with a 4xx or 5xx HTTP error code from the HTTP specification. No, only 4xx. It's the request that is in error for being an WebSockets Upgrade request, not the server. But per HTTP specifications such body data is part of the HTTP Upgrade request and should be ignored if switching protocols as part of ignoring the HTTP request which contained the Upgrade request. Consider for example if the request is a POST request, containing a alternative simplex representation of the communication meant to be used by a non-WebSockets server. 3.1.5.4 A server receiving a valid HTTP Upgrade request to WebSockets and which does not accept that WebSockets upgrade MUST send back a 4xx or 5xx HTTP response as per section 3.1.5.2. There is no such requirement in HTTP. Handling Upgrade is purely optional and the server is free to ignore Upgrade. Upgrade is a please upgrade to one of these protocols if supported. The server SHOULD send only the following lines: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade While it's true HTTP does not mandate any headers for 1xx responses, including things like Server is a good idea in 101 responses, and there may also be other headers required by the HTTP operation such as Authentication-Info and possibly other headers as well in future specifications. Regards Henrik
Re: Hello from Mozilla
On Fri, 17 Jul 2009, Amos Jeffries wrote: Um, okay. Have a read of this alternate Spec and see if you can stil find the security hole you are woried about: I don't understand in what way it substantially differs from what the spec says today. There are minor differences in wording, but other than that, and other than the requirement that two TCP connections be established to port 80 when using port 80 instead of just one, it seems equivalent. Could you elaborate on what the important difference you had in mind is? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Fri, 17 Jul 2009, Adrian Chadd wrote: 2009/7/17 Ian Hickson i...@hixie.ch: That way you are still speaking HTTP right until the protocol change occurs, so any and all HTTP compatible changes in the path(s) will occur. As mentioned earlier, we need the handshake to be very precisely defined because otherwise people could trick unsuspecting servers into opting in, or rather appearing to opt in, and could then send all kinds of commands down to those servers. Would you please provide an example of where an unsuspecting server is tricked into doing something? Sure. Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Now, Web Socket has a multi-layered approach to dealing with this. - there is the handshake, which requires that the server respond with a very specific set of bytes, thus guaranteeing that the server is in fact WebSocket-aware. Any wildcard part to this handshake increases the risk that there will be a server somewhere that can be tricked. For example, if the handshake were HTTP followed by anything followed by WebSocket, then some HTTP servers could be tricked into doing the handshake -- for example, the response to GET /WebSocket on the ietf.org host (not www.ietf.org) includes the word WebSocket in the response. HTTP servers aren't the only concern, of course; we want the handshake to be as secure as possible against any other protocol that may exist on any server that may be deployed. We don't know what's out there (especially in intranets), so the handshake has to be pretty stringent to make it as unlikely as possible. - there is origin checking and location checking, and the location checking isn't just an echo of the original request's data. This makes causing the server to send back particular data harder. (It's also part of our cross-origin security model and our shared hosting support; but that's a separate discussion.) - there is the framing, which does provide a modicum of protection by forcing another byte in front of the first author-controlled packet sent. (This isn't really a security feature, it's just a lucky accident of the framing that we needed to turn TCP streams into packets.) Ian, don't you see and understand the semantic difference between speaking HTTP and speaking a magic bytecode that is intended to look HTTP-enough to fool a bunch of things until the upgrade process occurs ? Don't you understand that the possible set of things that can go wrong here is quite unbounded ? Don't you understand the whole reason for known ports and protocol descriptions in the first place? Apparently not. Ok. Look at this. The byte sequence GET / HTTP/1.0\r\nHost: foo\r\nConnection: close\r\n\r\n is not byte equivalent to the sequence GET / HTTP/1.0\r\nConnection: close\r\nHost: foo\r\n\r\n The same byte sequence interpreted as a HTTP protocol exchange is equivalent. Yes, but the client is a WebSocket client, not an HTTP client, so why would it send anything but the WebSocket handshake? The only case I can see where the handshake gets changed is MITM proxies, but as far as I understand it, there's no way to ever get a reliable bidirectional non-HTTP TCP/IP connection through a Squid MITM proxy over port 80 to a remote server that normally acts like an HTTP server, so it doesn't matter anyway, since whatever we do, it won't work on that port. There's a mostly-expected understanding that what happens over port 80 is HTTP. The few cases where that has broken (specifically Shoutcast, but I do see other crap on port 80 from time to time..) has been by people who have implemented a mostly HTTP looking protocol, tested that it mostly works via a few gateways/firewalls/proxies, and then deployed it. Is there a way to get a reliable bidirectional non-HTTP TCP/IP connection through a Squid MITM proxy over port 80 to a remote server that normally acts like an HTTP server? If not, then sending any data over port 80 on such a network wouldn't work, right? So as far as I can tell, port 80 in such a scenario isn't relevant. Authors in such scenarios would use port 443, like Mark said. You're intending to do stuff over tcp/80 which looks like HTTP but isn't HTTP. No. I'm intending to do stuff over port 81. There is a desire in certain cases to be able to share this traffic with servers running on port
Re: Hello from Mozilla
On Fri, 17 Jul 2009, Mark Nottingham wrote: I missed that Ian was still talking about using port 80. I think that's broken / more trouble than it's worth, for the reasons Adri is going through. If you have to tunnel using an existing port, use 443 (with null encryption if you're worried about overhead, but still want to authenticate the endpoint). Even then, Wifi hotspots are probably going to redirect you, but using 443 should be a last-gasp measure anyway. I agree -- you would only use port 80 if the network wasn't intercepting your requests (which you can figure out pretty quickly by just trying to connect, since that will fail quickly in the case of a proxy). I would expect uses of this protocol that don't want encryption to try port 81, then port 80, then port 443 with encryption, in that order; I would expect uses of this protocol that _do_ want encryption to just try ports 815 and 443, in that order. 443 should always work if anything is going to work, as far as I can tell. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
fre 2009-07-17 klockan 10:00 + skrev Ian Hickson: HTTP servers aren't the only concern, of course; we want the handshake to be as secure as possible against any other protocol that may exist on any server that may be deployed. We don't know what's out there (especially in intranets), so the handshake has to be pretty stringent to make it as unlikely as possible. The HTTP/1.x 101[...]Upgrade: WebSocket[CR][NL][...][CR][NL][CR][NL] sequence is pretty stringent imho, without hardcoding the exact sequence at octet level, where [...] is any sequence of ascii characters except for [CR][NL][CR][NL] Yes, but the client is a WebSocket client, not an HTTP client, so why would it send anything but the WebSocket handshake? Because it's designed to also talk to a web server, and if I understand your intended use case those clients will by nature almost certainly also be HTTP clients, or at least embedded in HTTP clients. The only case I can see where the handshake gets changed is MITM proxies, but as far as I understand it, there's no way to ever get a reliable bidirectional non-HTTP TCP/IP connection through a Squid MITM proxy over port 80 to a remote server that normally acts like an HTTP server, so it doesn't matter anyway, since whatever we do, it won't work on that port. The relevant use case you need to make sure to support in specification is client - HTTP Surrogate - Internal origin don't bend over too much for transparently intercepting proxies, it's not your job to specify how those should behave. Is there a way to get a reliable bidirectional non-HTTP TCP/IP connection through a Squid MITM proxy over port 80 to a remote server that normally acts like an HTTP server? As I already said, generally no. But at some point transparently intercepting Squids will have the option to switch to tunnel mode when seeing non-HTTP traffic on port 80 (missing the HTTP signature), and at the same time forwarding of Upgrade requests is likely to be implemented. If not, then sending any data over port 80 on such a network wouldn't work, right? So as far as I can tell, port 80 in such a scenario isn't relevant. Authors in such scenarios would use port 443, like Mark said. Correct. No. I'm intending to do stuff over port 81. There is a desire in certain cases to be able to share this traffic with servers running on port 80, and in those cases, the content sent and the content required to be returned by the server is valid HTTP until the Upgrade succeeds, at which point it isn't HTTP. Which is the point I have been making all the time. As that is processed by HTTP stacks your protocol need to be prepared to deal with the aspects that the HTTP stack may introduce on you - HTTP authentication with it's set of status codes and HTTP headers - Via headers - Reordering of headers - Possible addition of Date headers in responses - Custom headers added to the request/response - And maybe other similar things as well Up until the 101 response has finished the sequence is HTTP and if your intention is that this should be processed by HTTP infrastructure then you need to accept it being HTTP. I don't understand the relevance of your question. It doesn't matter what you're talking; in the case of a MITM proxy, the client doesn't know it isn't talking straight to the server. Talking HTTP more than we already do doesn't suddenly mean we can get a two-way pipe set up. Talking proper HTTP means the MITM will handle it for you assuming the MITM can handle HTTP/1.1 properly. Sure, Squid can not yet, and we don't claim Squid to be HTTP/1.1 either as we don't like lying about our capabilities. But we will at some not too distant point in time. I really don't understand the problem you are alluding to here. What needs upgrading in such a scenario? The company infrastructure sitting between Internet and the web server(s) in question where some webmaster want's to deply a WebSocket application needs to add support for the WebSocket protocol, which is one of the primary things people who wants to run stuff like this over port 80 want to avoid.. To them the WebSocket communication is just part of the web application, not a separate entity which lives it's own life requiring it's own infrastructure in firewall rules, load balancers, servers etc. Regards Henrik
Re: Hello from Mozilla
Henrik Nordstrom wrote: fre 2009-07-17 klockan 16:32 +1200 skrev Amos Jeffries: 3.1.5.1 Connect to the server given by /host/ on port 80 and ask it to upgrade from HTTP to WebSockets. The client must send only the following lines: GET /thepath/ HTTP/1.1 Host: /host/ Upgrade: WebSockets Connection: Upgrade terminated by two CRLF as per HTTP specification. Per HTTP specifications the client SHOULD send more lines, in specific the User-Agent. Ah, thanks. Yes there are others that need to be added here for the formal draft. 3.1.5.2 A server receiving the request SHOULD ignore any other headers received with the request. If any data following the headers is received then fail the WebSocket connection. The server MUST discard any body data received with an Upgrade request. It MAY respond with a 4xx or 5xx HTTP error code from the HTTP specification. No, only 4xx. It's the request that is in error for being an WebSockets Upgrade request, not the server. I'm thinking 5xx if something went wrong, same as HTTP cases. So that the WebSockets spec is only limiting the 'good' response, but accepting of any HTTP error states as also its failure state. But per HTTP specifications such body data is part of the HTTP Upgrade request and should be ignored if switching protocols as part of ignoring the HTTP request which contained the Upgrade request. Consider for example if the request is a POST request, containing a alternative simplex representation of the communication meant to be used by a non-WebSockets server. Ian: is that a relevant case? you can spec it in or out as needed. But I'm thinking out, since the idea here is to setup a TCP-link tunnel then perform authentication handshake inside it. The upgrade request is only to get the MITM/Surrogates to pass it correctly or fail it cleanly. 3.1.5.4 A server receiving a valid HTTP Upgrade request to WebSockets and which does not accept that WebSockets upgrade MUST send back a 4xx or 5xx HTTP response as per section 3.1.5.2. There is no such requirement in HTTP. Handling Upgrade is purely optional and the server is free to ignore Upgrade. Upgrade is a please upgrade to one of these protocols if supported. Meaning HTTP does not care what gets returned... At the non-101 reply point its not usable as a WebSockets link. This bit is going slightly beyond HTTP in a compatible way and refers to the WebSockets-enabled server rejecting the socket connection. Ian mentioned many times WebSockets being 'opt-in'. This clause permits 'opt-in' to be denied by the server. It's irrelevant to non-WebSocket intermediaries and applies only to WebSocket origin servers. Perhapse that needs to be worded clearer. The server SHOULD send only the following lines: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade While it's true HTTP does not mandate any headers for 1xx responses, including things like Server is a good idea in 101 responses, and there may also be other headers required by the HTTP operation such as Authentication-Info and possibly other headers as well in future specifications. Ah yes. Okay. Ian: anything else you find useful as required here as well. This is where you get the is the mystery server capable of WebSockets info. Being the _right_ one needs must separate and handled by the WebSockets handshake. Just only specify it at the string level. Specifying the order sent is fine but confusing because it implies order received. Specifying the order received beyond 101 is broken. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE16 Current Beta Squid 3.1.0.9
Re: Hello from Mozilla
Ian Hickson wrote: On Fri, 17 Jul 2009, Amos Jeffries wrote: Um, okay. Have a read of this alternate Spec and see if you can stil find the security hole you are woried about: I don't understand in what way it substantially differs from what the spec says today. There are minor differences in wording, but other than that, and other than the requirement that two TCP connections be established to port 80 when using port 80 instead of just one, it seems equivalent. Could you elaborate on what the important difference you had in mind is? Two connections? no. Only one. Two METHODS of possible connection. 1) CONNECT and HTTP Upgrade are optional, and independent. One may be used or the other. They may be both tried in any order. 2) Specific mention is made that these are for failover use ONLY after port 81 and 815 have both already been tried and cannot be used. 3) The specific order of bytes is not mentioned _anywhere_ in the new text. 4) The order of headers _received_ is not mentioned past the 101 / 4xx /5xx line. HTTP varies order in-transit. 5) Specific mention is made to ignore non-understood headers added randomly by intermediaries. 6) Specific mention is made to drop random extra body data sent with headers or request and reply and how to handle it. Fixing that data-feeding issue you worry about. 7) Most importantly. The WebSocket handshake is not involved in this section. It happens after the Upgrade shift / CONNECT tunnel / port-81 TC link has succeeded. Not mid-way though the HTTP Upgrade processes. Between them these differences convert your section 3.1 from a detailed byte-level handshake, very flaky in any non-direct port-80 link, to an HTTP-level handshake which can be relayed or handled properly by existing MITM and Surrogates. HTTP clients and servers need very little change to operate with this as well and can leverage their broad base of HTTP code to process the headers properly and safely to the required end. As Henrik pointed out, there are HTTP things I've overlooked in the quick re-write. But their addition is minor editing touchups. HTTP Keep-Alive stuff may also be needed to encourage the WebSocket link to stay up long-term use for the handshake and data frames. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE16 Current Beta Squid 3.1.0.9
Re: Hello from Mozilla
ons 2009-07-15 klockan 04:26 + skrev Ian Hickson: On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? As already mentioned HTTP specs says it MUST add a Via header element. Regards Henrik
Re: Hello from Mozilla
ons 2009-07-15 klockan 07:34 + skrev Ian Hickson: On Wed, 15 Jul 2009, Mark Nottingham wrote: Upgrade is hop-by-hop, so it's pretty limiting. Do man-in-the-middle proxies count as a hop for the purposes of HTTP? When used as a surrogate it does. The transparently interceping case is outside of any specifications. As far as I can tell from the HTTP spec, the client is supposed to know whether it is speaking to a proxy or not, so man-in-the-middle proxies don't affect the hop-by-hop semantics... but it's not entirely clear. Yes, just as IP addresses are supposed to be unique and not shared or modified in transit (NAT)... Sure, but that's why we have the TLS-over-port-443 option. In the cases where there is uncooperative network infrastructure, the client can just switch to that, and then there's no way the connection can be affected. You think There is a funny technique implemented at many places today which unwinds TLS on port 443 and proxies the HTTP traffic which is supposed to be within. Used for policy enforcement and inspection in networks where end-to-end encrypted communication is forbidden by policy. Not doing so is unacceptably insecure for this use case, IMHO. We can't run the risk of Web pages hitting SMTP servers and sending spam, or poking at intranet servers, or who knows what else. You still haven¨t explained how running a proper HTTP Upgrade sequence may risk this. I just don't see it. Regards Henrik
Re: Hello from Mozilla
tor 2009-07-16 klockan 01:44 + skrev Ian Hickson: So Squid when used as a man-in-the-middle proxy will allow arbitrary traffic through port 80 if it isn't HTTP traffic? No, but support for tunneling WebSockets can be added, but not when it looks and feels like a HTTP message. This advice contradicts other advice we've gotten to the effect that if we send any content over port 80, it absolutely must be HTTP-compatible. And what we are saying is that if you make something that looks like HTTP and runs on port 80 it should be HTTP, not just something that looks like HTTP but isn't compatible with HTTP and which will break if processed as if it was HTTP. The point Amos tried to make is that if it isn't HTTP (just resembles HTTP but isn't compatible with HTTP) then it should not look like HTTP. For us as a proxy it's easier to deal with something that is easily identified as not-HTTP and switch stack entirely than to try to tweak the HTTP processing to not break looks-like-HTTP-but-isn't-HTTP applications. Could you elaborate on this? Why would a transparent proxy meddle with bytes after it has seen a successful Upgrade? Doesn't this contradict the requirements on transparent proxies in the HTTP spec? It won't. But it will touch the HTTP upgrade request, and possibly even the 101 response status description. If it at all supports Upgrade, but that is a different topic. For all intents and purposes, it _is_ an HTTP request. It's an Upgrade request. It seems that man-in-the-middle proxies would break all Upgrades, the way you've described them. The only part of your spec we talk about here is the binary coded looks-like-HTTP-Upgrade sequence. What happens after the looks-like-HTTP-101-response we don't care about. Right down to the HTTP/1.1 reserved protocol label (do get that changed please). If we're faking HTTP, then it has to look like HTTP. And our point is DO NOT FAKE HTTP. Use HTTP. To make the point explicit we already see one wire protocol (icecast, used by shoutcast streaming) which uses a similar fake HTTP mechanism like you. In this protocol the clients sends a quite well formed HTTP request, but the response is in a slightly different protocol. When this gets processed by an HTTP intermediary (proxy or frontend) things do break in subtle way because the application isn't expecting the transforms required by HTTP. For a long time noone noticed as Squid did not conform to a specific part the HTTP specifications, but when Squid was modified to follow HTTP specifications this application broke. I'm getting very mixed messages here. Is there a reliable way to open a bidirectional non-HTTP TCP/IP connection through a Squid man-in-the-middle proxy over port 80 to a remote server that normally acts like an HTTP server? If so, what is the sequence of bytes that will act in this way? Both yes and no. There is experimental support available for transparent interception operation to switch to tunnel mode when seeing non-HTTP traffic, but this is not yet part of the official distribution (needs a bit more work first). The use case for this is environments using transparent interception for caching reasons but which accepts that there may be non-HTTP applications also using port 80. At some point we will also support switching to tunnel mode after seeing a successful HTTP Upgrade sequence but we are not quite there yet. This discussion is to make sure WebSockets do not break when processed via a future Squid which do support HTTP Upgrade. The use case for this is both transparently interception in the client network and frontend intermediary setups in the web server network. In a normal forward proxy environment the supported method is CONNECT, but default policy (configuration) restricts this to port 443 and a few other well-known SSL ports blocking port 80. Related note to others following this discussion: Upgrade is hop-by-hop not end-to-end. Regards Henrik
Re: Hello from Mozilla
ons 2009-07-15 klockan 07:18 + skrev Ian Hickson: The reason we have a very strict handshake is because we don't want it to be possible to trick a non-WebSocket-aware server into accepting a connection (or similarly, having the client be tricked by the script into accepting a connection to a non-WebSocket-aware server). This is especially important for WebSockets because once there's a connection established, any data might be sent. Doesn't need to be octet-level strict for that. If you get a 101 response you know it's been accepted. If you get anything else you need to deal with that, as required by HTTP. Until the end of the 101 response it's all HTTP, per the rules of HTTP. HTTP is getting similar restrictions, by the way, in the form of the CORS mechanism. Without CORS, scripts can't ever read data back from cross-site HTTP requests they initiate. CORS allows the server to opt in to sending data back to a third-party script. Which is fine, as the HTTP part of CORS is a simple HTTP extension within the framework defined by HTTP. And I wouldn't call it that HTTP gets restrictions. In the case of CORS HTTP is extended to give client scripts authorization to access the servers data where such scripts before could not due to access restrictions implemented in the user-agent. For others following this thread: http://www.w3.org/TR/access-control/ http://hacks.mozilla.org/2009/07/cross-site-xmlhttprequest-with-cors/ Regards Henrik
Re: Hello from Mozilla
ons 2009-07-15 klockan 18:39 +1200 skrev Amos Jeffries: Byte 5 through to the first of: two CRLF or one NULL byte. Specified as step 1 through 11 by the looks of it. Correctly operating: * MUST remove the Upgrade: WebSocket\r\n bytes. Yes/No depending on the context. If a normal forward proxy sees such request then yes, but that's outside WebSocket specification as CONNECT is used in such case. In a surrogate intermediary situation it's essentially up to us how we deal with the upgrade. The operations of surrogate-origin is outside of HTTP specifications. Specifications only cover client-surrogate in such setups. In a transparent intercepting proxy it's our responsibility to deal with the Upgrade as appropriate. i.e. switch to tunnel mode, or declare it not supported by removing the header. Transparent interception mode is outside of HTTP specifications. * MUST add Via: 1.1 followed by an arbitrary length and content string. Yes. * SHOULD add a Date: header (position optional, suggested to be between GET and Host:). No. The Date requirement is only on responses, not requests. Conditionally: * Depending on the local network topology _sometimes_ proxies required to alter the section labeled the path in order for the request to even happen. Changing it from ( / path ) to ( http://; hostname / path ) Yes, but it will get changed back to the simple /path form before the requests hits an actual origin server. ** The http://hostname part MAY be removed again before passing to Server. Usually not. It MUST be removed per HTTP specifications (but servers MUST accept it if sent). Future HTTP specifications MAY be relaxed to allow absolute-URI form to be sent in requests as well but that's future when HTTP/1.0 servers is confirmed gone from the globe. To raise the extreme end of the problem: In HTTP a proxy MAY go so far as to sort the headers alphabetically and expect things to still work well. Yes. The solutions you need to be looking at are: * using HTTP Upgrade as per the HTTP spec, with full flexibility. * using ports other than 80. * sending requests as pure binary and dropping the HTTP look-alike bits Yes. Regards Henrik
Re: Hello from Mozilla
tor 2009-07-16 klockan 23:11 +1200 skrev Amos Jeffries: The faux-request begins with: GET /path-bytes HTTP/1.1 It also contains Connection: Upgrade. Only when not talking to a proxy. If a proxy is configured CONNECT is used for setting up a tunnel first. See 3.1 Handshake, step 3. For peers we already have the tricky case where admin may not set 'originserver' on web server links to trigger the un-map. Apache et al are compliant enough for this not to break the HTTP, but will completely break the WebSocket faux-HTTP byte-wise spec. True, but my point is that such setups is NOT HTTP/1.1 compliant. In HTTP/1.1 requests to origin server MUST be sent without the scheme://[u...@]host:port part, and servers MUST accept both forms. Regards Henrik
Re: Hello from Mozilla
Realise that server-side infrastructure often includes things like CDNs (even when the content is uncacheable), reverse proxies and L7 load balancers -- all of which can make the changes we're talking about. While it's true that these things are under control of the people who own the server, changing them is *much* harder than deploying an Apache module. Cheers, On 17/07/2009, at 9:53 AM, Ian Hickson wrote: The point is, there may be a whole lot of stuff going on with HTTP implementations that you're not aware of. Sure, but with the except of man-in-the-middle proxies, this isn't a big deal -- the people implementing the server side are in control of what the HTTP implementation is doing. -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
Yep. Just think it's worth pointing out in the draft, since you go to such efforts to make this possible. Cheers, On 17/07/2009, at 10:28 AM, Ian Hickson wrote: On Fri, 17 Jul 2009, Mark Nottingham wrote: Realise that server-side infrastructure often includes things like CDNs (even when the content is uncacheable), reverse proxies and L7 load balancers -- all of which can make the changes we're talking about. While it's true that these things are under control of the people who own the server, changing them is *much* harder than deploying an Apache module. People with that level of complexity can easily just not share the port. -- Ian Hickson U+1047E) \._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _ \ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'-- (,_..'`-.;.' -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
On Fri, 17 Jul 2009, Mark Nottingham wrote: On 17/07/2009, at 10:28 AM, Ian Hickson wrote: On Fri, 17 Jul 2009, Mark Nottingham wrote: Realise that server-side infrastructure often includes things like CDNs (even when the content is uncacheable), reverse proxies and L7 load balancers -- all of which can make the changes we're talking about. While it's true that these things are under control of the people who own the server, changing them is *much* harder than deploying an Apache module. People with that level of complexity can easily just not share the port. Yep. Just think it's worth pointing out in the draft, since you go to such efforts to make this possible. Done. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
2009/7/17 Ian Hickson i...@hixie.ch: That way you are still speaking HTTP right until the protocol change occurs, so any and all HTTP compatible changes in the path(s) will occur. As mentioned earlier, we need the handshake to be very precisely defined because otherwise people could trick unsuspecting servers into opting in, or rather appearing to opt in, and could then send all kinds of commands down to those servers. Would you please provide an example of where an unsuspecting server is tricked into doing something? Ian, don't you see and understand the semantic difference between speaking HTTP and speaking a magic bytecode that is intended to look HTTP-enough to fool a bunch of things until the upgrade process occurs ? Don't you understand that the possible set of things that can go wrong here is quite unbounded ? Don't you understand the whole reason for known ports and protocol descriptions in the first place? Apparently not. Ok. Look at this. The byte sequence GET / HTTP/1.0\r\nHost: foo\r\nConnection: close\r\n\r\n is not byte equivalent to the sequence GET / HTTP/1.0\r\nConnection: close\r\nHost: foo\r\n\r\n The same byte sequence interpreted as a HTTP protocol exchange is equivalent. There's a mostly-expected understanding that what happens over port 80 is HTTP. The few cases where that has broken (specifically Shoutcast, but I do see other crap on port 80 from time to time..) has been by people who have implemented a mostly HTTP looking protocol, tested that it mostly works via a few gateways/firewalls/proxies, and then deployed it. My suggestion is to completely toss the whole pretend to be HTTP thing out of the window and look at extending or adding a new HTTP mechanism for negotiating proper tunneling on port 80. If this involves making CONNECT work on port 80 then so be it. Redesigning HTTP is really much more work than I intend to take on here. HTTP already has an Upgrade mechanism, reusing it seems the right thing to do. What you intend to take on here and what should be taken on here is very relevant. You're intending to do stuff over tcp/80 which looks like HTTP but isn't HTTP. Everyone who implements anything HTTP gateway related (be it a transparent proxy, a firewall, a HTTP router, etc) suddenly may have to implement your websockets stuff as well. So all of a sudden your attempt to not extend HTTP ends up extending HTTP. The point is, there may be a whole lot of stuff going on with HTTP implementations that you're not aware of. Sure, but with the except of man-in-the-middle proxies, this isn't a big deal -- the people implementing the server side are in control of what the HTTP implementation is doing. That may be your understanding of how the world works, but out here in the rest of the world, the people who deploy the edge and the people who deploy the core may not be the same people. There may be a dozen layers of red tape, equipment lifecycle, security features, etc, that need to be handled before websockets happy stuff can be deployed everywhere it needs to. Please don't discount man-in-the-middle -anything- as being easy to deal with. In all cases except a man-in-the-middle proxy, this seems to be what we do. I'm not sure how we can do anything in the case of such a proxy, since by definition the client doesn't know it is present. .. so you're still not speaking HTTP? Ian, are you absolutely certain that everywhere you use the internet, there is no man in the middle between you and the server you're speaking to? Haven't you ever worked at any form of corporate or enterprise environment? What about existing captive portal deployments like wifi hotspots, some of which still use squid-2.5 (eww!) as their http firewall/proxy to control access to the internet? That stuff is going to need upgrading sure, but I'd rather see the upgrade happen once to a well thought out and reasonably well designed protocol, versus having lots of little upgrades need to occur because your HTTP but not quite HTTP exchange on port 80 isn't thought out enough. Adrian
Re: Hello from Mozilla
2009/7/15 Ian Hickson i...@hixie.ch: On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Anything which it can under the HTTP/1.x RFCs. Maybe I missed it - why exactly again aren't you just talking HTTP on the HTTP port(s), and doing a standard HTTP upgrade? Adrian
Re: Hello from Mozilla
Upgrade is hop-by-hop, so it's pretty limiting. Ian, an example: An intermediary (transparent, intercepting or whatever) can and often does add arbitrary headers; e.g., x-forwarded-for. This is completely legal in HTTP, where headers that are not understood are ignored, and additionally several headers have provisions to change values in headers as well (e.g., transfer-encoding, te, via, connection). They're also allowed to change the formatting of the message; e.g., re- wrap headers, normalise whitespace. Specifying a bit-for bit handshake is incredibly fragile. Cheers, On 15/07/2009, at 3:26 PM, Adrian Chadd wrote: 2009/7/15 Ian Hickson i...@hixie.ch: On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Anything which it can under the HTTP/1.x RFCs. Maybe I missed it - why exactly again aren't you just talking HTTP on the HTTP port(s), and doing a standard HTTP upgrade? Adrian -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
On Wed, 15 Jul 2009 04:26:42 + (UTC), Ian Hickson i...@hixie.ch wrote: On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Byte 5 through to the first of: two CRLF or one NULL byte. Specified as step 1 through 11 by the looks of it. Correctly operating: * MUST remove the Upgrade: WebSocket\r\n bytes. * MUST add Via: 1.1 followed by an arbitrary length and content string. * SHOULD add a Date: header (position optional, suggested to be between GET and Host:). Though Squid does not (yet) do this, other proxies and future fully HTTP compliant Squid might. Conditionally: * Depending on the local network topology _sometimes_ proxies required to alter the section labeled the path in order for the request to even happen. Changing it from ( / path ) to ( http://; hostname / path ) ** The http://hostname part MAY be removed again before passing to Server. Usually not. To raise the extreme end of the problem: In HTTP a proxy MAY go so far as to sort the headers alphabetically and expect things to still work well. The solutions you need to be looking at are: * using HTTP Upgrade as per the HTTP spec, with full flexibility. * using ports other than 80. * sending requests as pure binary and dropping the HTTP look-alike bits Amos
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Adrian Chadd wrote: 2009/7/15 Ian Hickson i...@hixie.ch: On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Anything which it can under the HTTP/1.x RFCs. Could you give some examples? What part of the handshake would Squid change? Maybe I missed it - why exactly again aren't you just talking HTTP on the HTTP port(s), and doing a standard HTTP upgrade? WebSocket isn't an HTTP protocol, or subprotocol, it's an independant protocol. The only reason we have an HTTP-Upgrade-like mechanism at all is so that people who want to can share a port with an HTTP server. (i.e. WebSocket doesn't use a standard HTTP Upgrade for the same reason FTP doesn't.) The reason we have a very strict handshake is because we don't want it to be possible to trick a non-WebSocket-aware server into accepting a connection (or similarly, having the client be tricked by the script into accepting a connection to a non-WebSocket-aware server). This is especially important for WebSockets because once there's a connection established, any data might be sent. HTTP is getting similar restrictions, by the way, in the form of the CORS mechanism. Without CORS, scripts can't ever read data back from cross-site HTTP requests they initiate. CORS allows the server to opt in to sending data back to a third-party script. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Mark Nottingham wrote: Upgrade is hop-by-hop, so it's pretty limiting. Do man-in-the-middle proxies count as a hop for the purposes of HTTP? As far as I can tell from the HTTP spec, the client is supposed to know whether it is speaking to a proxy or not, so man-in-the-middle proxies don't affect the hop-by-hop semantics... but it's not entirely clear. Ian, an example: An intermediary (transparent, intercepting or whatever) can and often does add arbitrary headers; e.g., x-forwarded-for. This is completely legal in HTTP, where headers that are not understood are ignored, and additionally several headers have provisions to change values in headers as well (e.g., transfer-encoding, te, via, connection). They're also allowed to change the formatting of the message; e.g., re-wrap headers, normalise whitespace. Sure, but that's why we have the TLS-over-port-443 option. In the cases where there is uncooperative network infrastructure, the client can just switch to that, and then there's no way the connection can be affected. Specifying a bit-for bit handshake is incredibly fragile. Not doing so is unacceptably insecure for this use case, IMHO. We can't run the risk of Web pages hitting SMTP servers and sending spam, or poking at intranet servers, or who knows what else. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Amos Jeffries wrote: Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Byte 5 through to the first of: two CRLF or one NULL byte. Specified as step 1 through 11 by the looks of it. Correctly operating: * MUST remove the Upgrade: WebSocket\r\n bytes. [...] This would cause the WebSocket connection to fail, which is the correct behaviour. After all, if the connection isn't upgraded, we don't want anything further to happen (in particular we don't want the client sending arbitrary bytes to the server or proxy, since that would open up the proxy to being abused to download content from any arbitrary server including intranet servers or other domains on shared-hosting servers). So loosening up the handshake wouldn't solve the problem described previously of Squid breaking an HTTP Upgrade to WebSocket in the case of a client behind a firewall that only allows port 80 and where all traffic through that port goes through a man-in-the-middle proxy. What solution would you recommend for such a case? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On 15/07/2009, at 5:34 PM, Ian Hickson wrote: On Wed, 15 Jul 2009, Mark Nottingham wrote: Upgrade is hop-by-hop, so it's pretty limiting. Do man-in-the-middle proxies count as a hop for the purposes of HTTP? As far as I can tell from the HTTP spec, the client is supposed to know whether it is speaking to a proxy or not, so man-in-the-middle proxies don't affect the hop-by-hop semantics... but it's not entirely clear. Intercepting (often called transparent, although that's confusing because HTTP defines semantically transparent as something completely different) do consider them to be separate hops (to be clear, the hops are on the wire, not on the devices, although sometimes there might be a virtual hop inside a device). There are a few L7 load balancers that don't act as a full intermediary in the HTTP sense (or at least, what they do is muddy enough that it's not clear), but do fiddle with headers (e.g., this is why you see things like Coenection instead of Connection). The important thing to remember is that interception *only* happens on port 80, except in the most pathological of networks (and you won't be able to do anything about them anyway). Notice that I'm making a distinction between intercepting and firewalling here; all bets are off when a firewall does stateful inspection of your protocol, unless you can convince it to pass an encrypted stream (the TLS-over-443 solution). They're also allowed to change the formatting of the message; e.g., re-wrap headers, normalise whitespace. Sure, but that's why we have the TLS-over-port-443 option. In the cases where there is uncooperative network infrastructure, the client can just switch to that, and then there's no way the connection can be affected. As long as you're using TLS, yes. That's going to limit scaling, of course. Specifying a bit-for bit handshake is incredibly fragile. Not doing so is unacceptably insecure for this use case, IMHO. We can't run the risk of Web pages hitting SMTP servers and sending spam, or poking at intranet servers, or who knows what else. Let me put that a slightly different way; specifying a bit-for-bit handshake is fragile *when you expect it to pass through HTTP infrastructure that has no reason to preserve those bits exactly as they are*. Can you remind me why you need the handshake to look like valid HTTP again? I think that's the crux here. Cheers, -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Mark Nottingham wrote: Can you remind me why you need the handshake to look like valid HTTP again? I think that's the crux here. Because in some cases, people will want to share the same port for their HTTP server as for their WebSocket server. For example, if they want to do TLS-WebSocket-over-port-443, in the case where that host also has an HTTPS server on port 443. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
Ian Hickson wrote: On Wed, 15 Jul 2009, Amos Jeffries wrote: Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? Byte 5 through to the first of: two CRLF or one NULL byte. Specified as step 1 through 11 by the looks of it. Correctly operating: * MUST remove the Upgrade: WebSocket\r\n bytes. [...] This would cause the WebSocket connection to fail, which is the correct behaviour. After all, if the connection isn't upgraded, we don't want anything further to happen (in particular we don't want the client sending arbitrary bytes to the server or proxy, since that would open up the proxy to being abused to download content from any arbitrary server including intranet servers or other domains on shared-hosting servers). So loosening up the handshake wouldn't solve the problem described previously of Squid breaking an HTTP Upgrade to WebSocket in the case of a client behind a firewall that only allows port 80 and where all traffic through that port goes through a man-in-the-middle proxy. What solution would you recommend for such a case? a) Getting a dedicated WebSocket port assigned. * You and the client needing it have an argument to get that port opened through the firewall. * Squid and other proxies can be altered to allow CONNECT through to safe defined ports (80 is not one). Or to do the WebSocket upgrade itself. b) accepting that the network being traversed is screwed beyond redemption by its own policy or admin. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE16 Current Beta Squid 3.1.0.9
Re: Hello from Mozilla
2009/7/15 Amos Jeffries squ...@treenet.co.nz: a) Getting a dedicated WebSocket port assigned. Â * You and the client needing it have an argument to get that port opened through the firewall. Â * Squid and other proxies can be altered to allow CONNECT through to safe defined ports (80 is not one). Or to do the WebSocket upgrade itself. b) accepting that the network being traversed is screwed beyond redemption by its own policy or admin. I think the fundamental mistake being made here by Ian (and potentially others) is breaking the assumption that specific protocols exist on the well-known ports. Suddenly treating stuff on port 80 as almost but not quite HTTP is bound to cause issues, both devices speaking valid HTTP (eg Squid) and firewalls etc which may treat the exchange as not HTTP and decide to start dropping things. Or worse - passing it through, sort of. Ian - I understand your motivations here but I think it shows a fundamental mis-understanding of the glue which keeps the internet mostly functioning together. Here's a question for you - would you run a mythical protocol, call it foonet, over IP, if it looked almost-but-not-quite like IP so people could run it on their existing IP networks? Can you see any particular issues with that? Other slots in the mythical OSI stack shouldn't be treated any differently. Adrian
Re: Hello from Mozilla
On 07/15/2009 01:59 AM, Ian Hickson wrote: On Wed, 15 Jul 2009, Mark Nottingham wrote: Can you remind me why you need the handshake to look like valid HTTP again? I think that's the crux here. Because in some cases, people will want to share the same port for their HTTP server as for their WebSocket server. For example, if they want to do TLS-WebSocket-over-port-443, in the case where that host also has an HTTPS server on port 443. Can such a dual-purpose port-sharing server implement both HTTP and WebSocket stacks and use the right stack depending on the first byte[s] of the incoming message? If port sharing is the primary motivation here, then dual protocol stack support seems like the right solution. This can be even implemented as a dumb TCP-level switch application that connects to either an HTTP server or a WebSocket server behind it. No binary HTTP messages, no risk of HTTP intermediaries screwing up with the not-really-HTTP-but-looking-like-one traffic. Just two completely different protocols sharing the same port. HTH, Alex.
Re: Hello from Mozilla
On Thu, 16 Jul 2009, Amos Jeffries wrote: What solution would you recommend for such a case? a) Getting a dedicated WebSocket port assigned. The plan is to ask for ports 81 and 815. * You and the client needing it have an argument to get that port opened through the firewall. * Squid and other proxies can be altered to allow CONNECT through to safe defined ports (80 is not one). Or to do the WebSocket upgrade itself. On the long term, that is the expected solution, yes. It doesn't work on the short term, though. b) accepting that the network being traversed is screwed beyond redemption by its own policy or admin. That's not really a solution. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Adrian Chadd wrote: I think the fundamental mistake being made here by Ian (and potentially others) is breaking the assumption that specific protocols exist on the well-known ports. Suddenly treating stuff on port 80 as almost but not quite HTTP is bound to cause issues, both devices speaking valid HTTP (eg Squid) and firewalls etc which may treat the exchange as not HTTP and decide to start dropping things. Or worse - passing it through, sort of. WebSocket really is its own protocol with its own ports. The talk about port 80 is only because in some environments, those other ports aren't going to work, and so there is interest in tunneling over ports 80 and 443, and this means sharing with HTTP servers, hence the Upgrade dance. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Wed, 15 Jul 2009, Alex Rousskov wrote: On 07/15/2009 01:59 AM, Ian Hickson wrote: On Wed, 15 Jul 2009, Mark Nottingham wrote: Can you remind me why you need the handshake to look like valid HTTP again? I think that's the crux here. Because in some cases, people will want to share the same port for their HTTP server as for their WebSocket server. For example, if they want to do TLS-WebSocket-over-port-443, in the case where that host also has an HTTPS server on port 443. Can such a dual-purpose port-sharing server implement both HTTP and WebSocket stacks and use the right stack depending on the first byte[s] of the incoming message? Isn't that exactly what an HTTP Upgrade is? If port sharing is the primary motivation here, then dual protocol stack support seems like the right solution. This can be even implemented as a dumb TCP-level switch application that connects to either an HTTP server or a WebSocket server behind it. No binary HTTP messages, no risk of HTTP intermediaries screwing up with the not-really-HTTP-but-looking-like-one traffic. Just two completely different protocols sharing the same port. That's exactly what WebSocket is doing. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Thu, 16 Jul 2009, Mark Nottingham wrote: As an alternative, why not: 1) specify a new port for normal WebSocket operation, and 2) specify that if there's a proxy configured, ask the proxy to CONNECT to your new port, and 3) specify that if #2 fails, they can CONNECT to 443 and ask for an Upgrade to WebSockets in the first HTTP request. We don't want to ever have a spec-mandated switch from port to port (since that implies an origin change), but basically, that's what the spec currently says, except it requires the script to detect the failure at #2 and requires the script to explicitly try again on port 443. (And except that the Upgrade has to be precisely constrained, not arbitrary HTTP, so that we never get to a situation where the client thinks it has done an upgrade but really the other side was tricked into sending the right bytes for that, letting the script speak to a poor unsuspecting server that didn't intentionally opt in.) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On 16/07/2009, at 12:28 PM, Ian Hickson wrote: On Thu, 16 Jul 2009, Mark Nottingham wrote: As an alternative, why not: 1) specify a new port for normal WebSocket operation, and 2) specify that if there's a proxy configured, ask the proxy to CONNECT to your new port, and 3) specify that if #2 fails, they can CONNECT to 443 and ask for an Upgrade to WebSockets in the first HTTP request. We don't want to ever have a spec-mandated switch from port to port (since that implies an origin change), Ah - that makes sense. but basically, that's what the spec currently says, except it requires the script to detect the failure at #2 and requires the script to explicitly try again on port 443. (And except that the Upgrade has to be precisely constrained, not arbitrary HTTP, so that we never get to a situation where the client thinks it has done an upgrade but really the other side was tricked into sending the right bytes for that, letting the script speak to a poor unsuspecting server that didn't intentionally opt in.) So, to be clear, the only time the byte-for-byte HTTP handshake is used is when it's over a TLS tunnel via CONNECT (i.e., it's not used to set up the tunnel, but only once it's established)? If that's the case, should be no problem. A bit weird, thought; speaking two protocols on the same port isn't really good practice... Cheers, -- Mark Nottingham m...@yahoo-inc.com
Re: Hello from Mozilla
On Thu, 16 Jul 2009, Mark Nottingham wrote: So, to be clear, the only time the byte-for-byte HTTP handshake is used is when it's over a TLS tunnel via CONNECT (i.e., it's not used to set up the tunnel, but only once it's established)? It's used whenever the client thinks it has a connection to the destination HTTP or WebSocket server, whether that's over TLS or not. If it _knows_ that it is talking to a proxy, then it does the CONNECT thing first (or whatever is appropriate; SOCKS proxies are preferred). If it thinks it is talking to the destination server but is being intercepted by a man-in-the-middle proxy, e.g. when it tries to connect over port 80 without knowing of any proxies (not a recommended practice, but it could happen), then you run into the problem that Adrian and I are discussing in the separate branch of this thread. If that's the case, should be no problem. A bit weird, thought; speaking two protocols on the same port isn't really good practice... Indeed, that's why it has (well, will have, they're not registered yet) its own ports. On the long term, I would hope that we could just use those, and not have to worry about HTTP at all. We're not there yet. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
tor 2009-07-02 klockan 17:59 -0700 skrev Jason Duell: 1) I have a quick question about the limitations that you put by default on the CONNECT method. squid.conf contains # Deny CONNECT to other than SSL ports http_access deny CONNECT !SSL_ports So squid by default only allows CONNECT to port 443. I assume this is a rule-of-thumb security measure, since you figured that port 443 is the only common legitimate case for CONNECT? How long has this been the default behavior? Do you know if other proxy servers out there also do this? As you already know it's a security measure. I would expect it to be implemented by quite many as it's mentioned in the original Netscape document where CONNECT was first specified. I am not sure if Netscape still has that document, but it was later published as an internet-draft http://tools.ietf.org/html/draft-luotonen-ssl-tunneling-03 Due to this fact, the proxy cannot verify that the protocol being spoken is really SSL, and so the proxy configuration should explicitly limit allowed connections to well-known SSL ports (such as 443 for HTTPS, 563 for SNEWS, as assigned by the Internet Assigned Numbers Authority). When CONNECT later entered HTTP standards-track via the TLS working group it was for a slightly different model however (HTTP/1.1 Upgrade mechanism), and that text ONLY talks about port 80 and the security risks of allowing other ports. http://tools.ietf.org/html/rfc2817#section-8.2 A generic TCP tunnel is fraught with security risks. First, such authorization should be limited to a small number of known ports. The Upgrade: mechanism defined here only requires onward tunneling at port 80. Second, since tunneled data is opaque to the proxy, there are additional risks to tunneling to other well-known or reserved ports. A putative HTTP client CONNECTing to port 25 could relay spam via SMTP, for example. And the HTTP Upgrade mechanism generally haven't taken off yet with most of the net stuck in the https model, so there haven't been much (any before you) demands for CONNECT to port 80. I noticed that in section 3.1.3 the spec relies implicitly on CONNECT being allowed to arbitrary ports. But this is not the case for default installs of squid, and thus I fear that the general approach may be flawed. As you can see above all specifications available for the CONNECT method recommends very strict rules regarding which ports is allowed. I suppose we could ask that you allow arbitrary CONNECT access (or at least to the well-known websockets ports: 80/81/815). But I'm guessing that wouldn't help much, as it would probably take many years for that change to roll out across the net. Indeed. And as discussed we would never ship a default config which allows arbitrary ports, just a carefully selected list of ports. For what it's worth neither 81 or 815 is registered with IANA, and the web-sockets draft you referenced has expired. Regards Henrik
Re: Hello from Mozilla
tis 2009-07-14 klockan 23:35 +0200 skrev Henrik Nordstrom: For what it's worth neither 81 or 815 is registered with IANA, and the web-sockets draft you referenced has expired. Correction, the draft is not expired as it was recently resubmitted to IETF, but very much a work in progress. The correct link for reading up on websockets is http://dev.w3.org/html5/websockets/ and/or http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol There was a very brief discussion regarding websockets in the HTTPbis IETF WG in January. But it didn't spur very much interest among the HTTP people and discussions quickly died, not too surprisingly as it's not HTTP even if the initial connection setup is wrapped in an HTTP style message to be able to piggy-back on port 80. http://lists.w3.org/Archives/Public/ietf-http-wg/2009JanMar/0008.html The actual discussions regarding this family of wire protocols now take place on the HYBI IETF mailing list https://www.ietf.org/mailman/listinfo/hybi and from a quick scan of the discussions there apparently is several (two to four) proposals being discussed. and it's still very early to say where things will end up. There is not yet an official IETF working group dealing with this, just an informal gathering to discuss the problem area to see in what direction an eventual standardization effort should go. Regards Henrik
Re: Hello from Mozilla
ons 2009-07-08 klockan 07:00 + skrev Ian Hickson: Well, the client is a WebSocket client, so it can always generate the exact byte sequence specified in the spec almost by definition. The server side is restrictive, but should be implementable without too much trouble so long as there is a way to register a protocol handler that can prevent the HTTP side of the server from sending any bytes at all as part of the response. I agree that this is suboptimal, but we're somewhat forced into this by one of our security design requirements (that the handshake be very specific to avoid enabling request smuggling or other such behaviour -- basically we really want to be sure we're talking to a WebSocket server at the end of the handshake). If you use HTTP Upgrade as the initial handshake mechanism and expect this to be layered ontop of HTTP servers then you SHOULD use HTTP for this sequence. What takes place after the HTTP/1.1 response to the Upgrade is entirely yours, but until then it's HTTP. Hardcoding this sequence is just silly, and will cause a lot of trouble later on. You already touch some aspects of HTTP such as authentication. Now throw things like NTLM or Digest authentication into the mix and you will quickly find that the current specification is a bit too limiting if the intention is that the upgrade requests should be processed by an HTTP server. Also regarding HTTP Upgrade semantics. If you want to follow that model then there is not really any relation between the initial Upgrade request and what takes place after the upgrade has completed. It's a complete switch of protocols, with their own semantics. Regards Henrik
Re: Hello from Mozilla
On 07/14/2009 04:38 PM, Henrik Nordstrom wrote: ons 2009-07-08 klockan 07:00 + skrev Ian Hickson: Well, the client is a WebSocket client, so it can always generate the exact byte sequence specified in the spec almost by definition. The server side is restrictive, but should be implementable without too much trouble so long as there is a way to register a protocol handler that can prevent the HTTP side of the server from sending any bytes at all as part of the response. I agree that this is suboptimal, but we're somewhat forced into this by one of our security design requirements (that the handshake be very specific to avoid enabling request smuggling or other such behaviour -- basically we really want to be sure we're talking to a WebSocket server at the end of the handshake). If you use HTTP Upgrade as the initial handshake mechanism and expect this to be layered ontop of HTTP servers then you SHOULD use HTTP for this sequence. What takes place after the HTTP/1.1 response to the Upgrade is entirely yours, but until then it's HTTP. Hardcoding this sequence is just silly, and will cause a lot of trouble later on. I strongly agree with this. Frankly, I suspect (hope?) that IETF gurus would prevent publishing an RFC that kind of reuses the HTTP Upgrade mechanism but then declares many compliant Upgrade exchanges invalid. If you think your approach is the right one, I would suggest openly discussing it with the right IETF folks as early as possible, to avoid wasting your time on an idea they will be blocked later. HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. Good luck, Alex. You already touch some aspects of HTTP such as authentication. Now throw things like NTLM or Digest authentication into the mix and you will quickly find that the current specification is a bit too limiting if the intention is that the upgrade requests should be processed by an HTTP server. Also regarding HTTP Upgrade semantics. If you want to follow that model then there is not really any relation between the initial Upgrade request and what takes place after the upgrade has completed. It's a complete switch of protocols, with their own semantics. Regards Henrik
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: If you think your approach is the right one, I would suggest openly discussing it with the right IETF folks as early as possible, to avoid wasting your time on an idea they will be blocked later. WebSocket is being discussed in the hybi IETF list. HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On 07/14/2009 06:01 PM, Ian Hickson wrote: On Tue, 14 Jul 2009, Alex Rousskov wrote: HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. Perhaps the WebSocket secure handshake should start _after_ the successful upgrade? Is not that how HTTP Upgrade was intended to be used? WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. I think I understand the intent behind the trick, but if a message looks like an HTTP message to the HTTP server or intermediary, the HTTP server or intermediary may start doing HTTP-valid things to it, and those things will not be valid from WebSocket point of view. Henrik provided a few examples of that already. I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. Cheers, Alex.
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: On 07/14/2009 06:01 PM, Ian Hickson wrote: On Tue, 14 Jul 2009, Alex Rousskov wrote: HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. Perhaps the WebSocket secure handshake should start _after_ the successful upgrade? Is not that how HTTP Upgrade was intended to be used? If there are any bytes allowed from the client or the server before the handshake starts, then it is no longer secure. The idea is to make sure you can't smuggle though payloads from other protocols, since otherwise you could use WebSocket to connect to services that aren't expecting it. WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. I think I understand the intent behind the trick, but if a message looks like an HTTP message to the HTTP server or intermediary, the HTTP server or intermediary may start doing HTTP-valid things to it, and those things will not be valid from WebSocket point of view. Right -- and those things will prevent the connection, exactly as intended. The idea is to make sure that if there is anything in between that _isn't_ WebSocket-aware, the connection be dropped before the author has any chance of sending data down the pipe. If we didn't do this, then it's possible that, e.g., Squid could be tricked into sending data from third-party servers in a way that the WebSocket client could not detect, thus breaching the same-origin security model. I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
ons 2009-07-15 klockan 00:51 + skrev Ian Hickson: Right -- and those things will prevent the connection, exactly as intended. The idea is to make sure that if there is anything in between that _isn't_ WebSocket-aware, the connection be dropped before the author has any chance of sending data down the pipe. If we didn't do this, then it's possible that, e.g., Squid could be tricked into sending data from third-party servers in a way that the WebSocket client could not detect, thus breaching the same-origin security model. Which is a clear sign that you MUST NOT use a message that even resembles a HTTP message for the WebSocket handshake if you don't think that the HTTP model can accomplish the protocol you need. As all processing MUST bypass the HTTP stack of the server anyway the message may in fact look any way you like as long as it can be identified.. But I honestly don't get what you are talking about here. How would using HTTP Upgrade as intended possibly break the WebSocket security model? The WebSocket client has full control over the handshake, and is responsible for sending it as a well formed HTTP message and also parsing the HTTP response and should guarantee that it DOES NOT deliver any non-websocket payload (i.e. 407 responses, 5xx responses etc) to the WebSocket API user... If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. Squid operates on HTTP messages. While doing so HTTP REQUIRES it to modify the message in certain ways, and allows it to modify it in several other ways. - Addition of Via header (MUST) - Various other informative headers added by Squid. (fully allowed) But it's not my point. What I said is not even related to Squid but general protocol design and your design goals with the websocket wire protocol handshake resembling HTTP. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) Err, a semantically transparent proxy still modifies the request in many ways. Via headers is the most obvious, but there is many many other small things going on. It's just not allowed to change the request or response in such way that the request/response means something else. But yes, a transparently intercepting proxy is a different thing and what Alex meant. Regards Henrik
Re: Hello from Mozilla
On 07/14/2009 06:51 PM, Ian Hickson wrote: I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) From HTTP point of view, some byte changes are not really changes and some are actually required. Squid changes HTTP headers today and will probably continue to do so 5 years from now, even when deployed as a man in the middle. If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) That is why I quoted transparent. There is little transparent about most transparent hijacking proxies. It is the reality that is not covered by the HTTP specs but WebSocket will have to deal with it. Moreover, I suspect similar arguments can be applied to surrogates and even forward proxies that are covered by the specs and the specs allow them to modify HTTP messages. Cheers, Alex.
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On Tue, 7 Jul 2009, Alex Rousskov wrote: What is the motivation behind adding more ports? WebSocket is its own protocol, so the right thing to do is to use new ports. However, WebSocket can be sent over 80 or 443 quite happily. Can websocket use port 80 (without CONNECT) and port 443 (with CONNECT)? Yes. I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? It actually does support a true HTTP Upgrade also, so that people who are already using ports 80 or 443 for HTTP can still use WebSocket if they are able to make their servers handle the protocol upgrade. Cheers, -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
ons 2009-07-08 klockan 15:16 +1200 skrev Amos Jeffries: CONNECT to port-80 by default is IMO not an option. It pretty much defeats all the other HTTP-level security measures. For what it's worth, RFC2817 Upgrading to TLS Within HTTP/1.1 requires CONNECT to be accepted to port 80. Not that I really know of anyone who uses this standards-track method of TLS/SSL encrypting HTTP, just plain old HTTP over TLS/SSL (https). Regards Henrik
Re: Hello from Mozilla
On Tue, 7 Jul 2009, Alex Rousskov wrote: If WebSocket is a new protocol on top of TCP, new TCP ports are appropriate. Right. Like virtually any protocol on top of TCP, WebSocket can then rely on HTTP CONNECT tunneling mechanism without trying to redefine or narrow it down. Right. If WebSocket is a new protocol on top of HTTP, then the talk of ports seems far less appropriate because HTTP already has commonly used ports. WebSocket is a protocol in its own right; however it happens to have a handshake that appears to HTTP servers as an HTTP Upgrade request so that it can share a port with HTTP servers. The fact that I could not tell which of the above statements is true based on a quick draft read and things like prohibiting valid HTTP messages in WebSocket responses worry me, but it is likely I just missed some important pieces or caveats. I've tried adding a section to the intro to make this clearer. Please let me know if it can be improved further. I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? It actually does support a true HTTP Upgrade also, so that people who are already using ports 80 or 443 for HTTP can still use WebSocket if they are able to make their servers handle the protocol upgrade. As far as I can tell, the protocol does not support true HTTP Upgrade sequence but just one specific exchange of hard-coded (down to each octet level!) HTTP messages. This clever hack smells like a layering violation and can be a real pain to support if the client or server HTTP stack just happens to generate slightly different messages. Well, the client is a WebSocket client, so it can always generate the exact byte sequence specified in the spec almost by definition. The server side is restrictive, but should be implementable without too much trouble so long as there is a way to register a protocol handler that can prevent the HTTP side of the server from sending any bytes at all as part of the response. I agree that this is suboptimal, but we're somewhat forced into this by one of our security design requirements (that the handshake be very specific to avoid enabling request smuggling or other such behaviour -- basically we really want to be sure we're talking to a WebSocket server at the end of the handshake). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On 07/02/2009 06:59 PM, Jason Duell wrote: Hi there! I'm doing some work for Mozilla on the network stack for Firefox, and there are a couple of items I wanted to run by the Squid development team. Hello Jason, I am forwarding your email to squid-dev mailing list where it will receive more attention. 1) I have a quick question about the limitations that you put by default on the CONNECT method. squid.conf contains # Deny CONNECT to other than SSL ports http_access deny CONNECT !SSL_ports So squid by default only allows CONNECT to port 443. I assume this is a rule-of-thumb security measure, since you figured that port 443 is the only common legitimate case for CONNECT? How long has this been the default behavior? Do you know if other proxy servers out there also do this? You have identified the main reason correctly. This default has been added many years ago but I do not know when. I would expect some other popular proxies to restrict tunneling as well, but I have not tested it. The reason I ask is because we're looking to take a patch that implements the IETF websockets protocol: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-17 I noticed that in section 3.1.3 the spec relies implicitly on CONNECT being allowed to arbitrary ports. But this is not the case for default installs of squid, and thus I fear that the general approach may be flawed. I suppose we could ask that you allow arbitrary CONNECT access (or at least to the well-known websockets ports: 80/81/815). But I'm guessing that wouldn't help much, as it would probably take many years for that change to roll out across the net. I would also expect many firewalls to block port 81 and 815 by default so even if Squid allows those ports, websocket clients that do not hide behind Squid will still have problems (unless websocket is restricted to proxies). Any thoughts you have on the matter would be much appreciated. In general, it seems that handling proxies correctly is turning out to be one of the trickier parts of implementing a websockets-like API, so we may want to pick your brains some more in the future with other ideas. What is the motivation behind adding more ports? Can websocket use port 80 (without CONNECT) and port 443 (with CONNECT)? I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? 2) A totally unrelated issue: I assume by now you're aware that Firefox (and all other major browsers besides Opera) now no longer renders replies from squid or other proxies for failed HTTPS requests. For details see https://bugzilla.mozilla.org/show_bug.cgi?id=479880 I meant to email you about this before the patch went in, but then I got busy :) I'm not sure that there's much to talk about now that the fix is in (it's gross, and we theoretically ought to be able to do better, but it would be a lot of work, so I don't think it's going to happen). But I wanted you to know. It's really just a UI issue (you can tell I'm a systems programmer...) Yeah, looks like a browser security bug has been replaced with usability and compliance bugs. There is nothing Squid can do about it but thank you for letting is know as this info may be useful when troubleshooting. P.S. So every time that I set up squid on my machine to test something, it always denies access to me out of the box. I finally figured out it's because you don't allow localhost connections by default. Should you be adding a line like acl localnet src localhost to squid.conf? Is there a reason why you're allowing 10.0.0.1, etc. to connect, but not localhost? Good question. I will let others answer it :-). Thank you, Alex.
Re: Hello from Mozilla
On Tue, 2009-07-07 at 17:01 -0600, Alex Rousskov wrote: The reason I ask is because we're looking to take a patch that implements the IETF websockets protocol: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-17 I noticed that in section 3.1.3 the spec relies implicitly on CONNECT being allowed to arbitrary ports. But this is not the case for default installs of squid, and thus I fear that the general approach may be flawed. I think it has several serious challenges; I doubt it could be deployed in (say) Australia *at all*. I suppose we could ask that you allow arbitrary CONNECT access (or at least to the well-known websockets ports: 80/81/815). But I'm guessing that wouldn't help much, as it would probably take many years for that change to roll out across the net. I would also expect many firewalls to block port 81 and 815 by default so even if Squid allows those ports, websocket clients that do not hide behind Squid will still have problems (unless websocket is restricted to proxies). So squid has two primary uses: - caching - security From a caching perspective, websockets are just overhead - uncacheable content simply adds overhead to what squid has to do. Just using TCP/IP would be a lot better and faster. Many ISP's use interception techniques to deploy squid and other caches. These operate by performing a MITM attack on TCP/IP traffic. Policy limits in these deployments are often turned off (because users using TCP/IP would expect to access any url), while at the same time lifetime heuristics tend to be turned way up (to maximise the hit rate the ISP can achieve). As such, I'd expect this spec to have requests fail all over the place. There are mechanisms for such requests to be reissued on fresh, direct TCP connections by the router that performed the interception - but the specific toolchain deployed will vary how successful that is. Note that this has nothing to do with the use of Connect: - squid, or other proxies, would see the 'Upgrade:' request. From a security perspective, there are two sub issues: - preventing malicous use (e.g. spam bouncing) - policy restrictions (acls, corporate policy, content filtering...) Malicious use covers things like not being an open proxy, and not permitting connections to SMTP servers for any clients. This is where the default Connect limit comes in - and also the prevention of http requests to port 25. Any thoughts you have on the matter would be much appreciated. In general, it seems that handling proxies correctly is turning out to be one of the trickier parts of implementing a websockets-like API, so we may want to pick your brains some more in the future with other ideas. What is the motivation behind adding more ports? Can websocket use port 80 (without CONNECT) and port 443 (with CONNECT)? I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? I'd like to understand the motivation behind not just using TCP/IP with a SOCKS proxy. The whole websockets thing seems like baroque cruft, TBH. A lot of complexity, more layers that things can go wrong in (and that security holes can be found in). 2) A totally unrelated issue: I assume by now you're aware that Firefox (and all other major browsers besides Opera) now no longer renders replies from squid or other proxies for failed HTTPS requests. For details see https://bugzilla.mozilla.org/show_bug.cgi?id=479880 I meant to email you about this before the patch went in, but then I got busy :) I'm not sure that there's much to talk about now that the fix is in (it's gross, and we theoretically ought to be able to do better, but it would be a lot of work, so I don't think it's going to happen). But I wanted you to know. It's really just a UI issue (you can tell I'm a systems programmer...) Thanks. It would be nice if mozilla would create a context *for the proxy* and use that to show the non 200 response. Debuggability is kindof important :). P.S. So every time that I set up squid on my machine to test something, it always denies access to me out of the box. I finally figured out it's because you don't allow localhost connections by default. Should you be adding a line like acl localnet src localhost to squid.conf? Is there a reason why you're allowing 10.0.0.1, etc. to connect, but not localhost? I'd be open to us changing this. It is a [small] risk for a bastion host to allow connections from itself because a different account being compromised then allows access via the proxy. I have no evidence to make an assertion about the frequency of deployments on a bastion host vs behind one, and so the only argument I have for preserving it is 'secure as possible by default', which while a good argument isn't the end of the discussion. -Rob
Re: Hello from Mozilla
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robert Collins wrote: P.S. So every time that I set up squid on my machine to test something, it always denies access to me out of the box. I finally figured out it's because you don't allow localhost connections by default. Should you be adding a line like acl localnet src localhost to squid.conf? Is there a reason why you're allowing 10.0.0.1, etc. to connect, but not localhost? I'd be open to us changing this. It is a [small] risk for a bastion host to allow connections from itself because a different account being compromised then allows access via the proxy. I have no evidence to make an assertion about the frequency of deployments on a bastion host vs behind one, and so the only argument I have for preserving it is 'secure as possible by default', which while a good argument isn't the end of the discussion. Your argument is subject to reductio ad absurdam: if you want secure as possible by default, then the default config shold not allow proxied access from *any host at all*. Any host other than localhost should be *less* trusted than localhost. I would argue that enabling only localhost for the default forward proxy configuration is a sane default: people configuring things like bastions ought not to expect to use out-of-the box configs without review / tweakage, while people using Squid as a personal cache ought not to have to do such tweaks. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKVARQ+gerLs4ltQ4RAhlJAKDWsjrr/7IT45r4IPXsXt5Xyfa0zwCffrfr hLbI2vMOIWeHA09Mf+Kdg2k= =bVwt -END PGP SIGNATURE-
Re: Hello from Mozilla
Robert Collins wrote: On Tue, 2009-07-07 at 17:01 -0600, Alex Rousskov wrote: The reason I ask is because we're looking to take a patch that implements the IETF websockets protocol: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-17 I noticed that in section 3.1.3 the spec relies implicitly on CONNECT being allowed to arbitrary ports. But this is not the case for default installs of squid, and thus I fear that the general approach may be flawed. I think it has several serious challenges; I doubt it could be deployed in (say) Australia *at all*. I suppose we could ask that you allow arbitrary CONNECT access (or at least to the well-known websockets ports: 80/81/815). But I'm guessing that wouldn't help much, as it would probably take many years for that change to roll out across the net. I would also expect many firewalls to block port 81 and 815 by default so even if Squid allows those ports, websocket clients that do not hide behind Squid will still have problems (unless websocket is restricted to proxies). I'm open to adding 815, maybe 81. Though as Robert says, the utility may not be as much as the authors think. CONNECT to port-80 by default is IMO not an option. It pretty much defeats all the other HTTP-level security measures. Port-81 is borderline on the risky side. Considering the number of proxies and web-apps which historically use it for regular internal LAN HTTP stuff. Admin who know they need CONNECT to those ports are free to do so of course. IIRC, there is also another HTTPS port 5-something, and some other protocols needing consideration in the same commit. So squid has two primary uses: - caching - security From a caching perspective, websockets are just overhead - uncacheable content simply adds overhead to what squid has to do. Just using TCP/IP would be a lot better and faster. Many ISP's use interception techniques to deploy squid and other caches. These operate by performing a MITM attack on TCP/IP traffic. Policy limits in these deployments are often turned off (because users using TCP/IP would expect to access any url), while at the same time lifetime heuristics tend to be turned way up (to maximise the hit rate the ISP can achieve). As such, I'd expect this spec to have requests fail all over the place. There are mechanisms for such requests to be reissued on fresh, direct TCP connections by the router that performed the interception - but the specific toolchain deployed will vary how successful that is. Note that this has nothing to do with the use of Connect: - squid, or other proxies, would see the 'Upgrade:' request. From a security perspective, there are two sub issues: - preventing malicous use (e.g. spam bouncing) - policy restrictions (acls, corporate policy, content filtering...) Malicious use covers things like not being an open proxy, and not permitting connections to SMTP servers for any clients. This is where the default Connect limit comes in - and also the prevention of http requests to port 25. Any thoughts you have on the matter would be much appreciated. In general, it seems that handling proxies correctly is turning out to be one of the trickier parts of implementing a websockets-like API, so we may want to pick your brains some more in the future with other ideas. What is the motivation behind adding more ports? Can websocket use port 80 (without CONNECT) and port 443 (with CONNECT)? I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? I'd like to understand the motivation behind not just using TCP/IP with a SOCKS proxy. The whole websockets thing seems like baroque cruft, TBH. A lot of complexity, more layers that things can go wrong in (and that security holes can be found in). 2) A totally unrelated issue: I assume by now you're aware that Firefox (and all other major browsers besides Opera) now no longer renders replies from squid or other proxies for failed HTTPS requests. For details see https://bugzilla.mozilla.org/show_bug.cgi?id=479880 I meant to email you about this before the patch went in, but then I got busy :) I'm not sure that there's much to talk about now that the fix is in (it's gross, and we theoretically ought to be able to do better, but it would be a lot of work, so I don't think it's going to happen). But I wanted you to know. It's really just a UI issue (you can tell I'm a systems programmer...) Thanks. It would be nice if mozilla would create a context *for the proxy* and use that to show the non 200 response. Debuggability is kindof important :). These are the reponse codes I'm aware of people trying to use with Squid CONNECT. For now the admins are hacking around the browsers that don't support custom errors. Taking 404 access denied as assumed. * the authenticated proxy (407) response needing a login popup. * the
Re: Hello from Mozilla
On 07/07/2009 05:39 PM, Ian Hickson wrote: On Tue, 7 Jul 2009, Alex Rousskov wrote: What is the motivation behind adding more ports? WebSocket is its own protocol, so the right thing to do is to use new ports. However, WebSocket can be sent over 80 or 443 quite happily. If WebSocket is a new protocol on top of TCP, new TCP ports are appropriate. Like virtually any protocol on top of TCP, WebSocket can then rely on HTTP CONNECT tunneling mechanism without trying to redefine or narrow it down. If WebSocket is a new protocol on top of HTTP, then the talk of ports seems far less appropriate because HTTP already has commonly used ports. The fact that I could not tell which of the above statements is true based on a quick draft read and things like prohibiting valid HTTP messages in WebSocket responses worry me, but it is likely I just missed some important pieces or caveats. I have noticed there is something about the Upgrade header in the websocket draft so perhaps that mechanism can be polished to implement a proper protocol switch? Or is that too late? It actually does support a true HTTP Upgrade also, so that people who are already using ports 80 or 443 for HTTP can still use WebSocket if they are able to make their servers handle the protocol upgrade. As far as I can tell, the protocol does not support true HTTP Upgrade sequence but just one specific exchange of hard-coded (down to each octet level!) HTTP messages. This clever hack smells like a layering violation and can be a real pain to support if the client or server HTTP stack just happens to generate slightly different messages. Again, perhaps general-purpose client and server stacks are simply not the targeted audience for the protocol and their limitation simply do not matter. Cheers, Alex.
Re: Hello from Mozilla
On Tue, 2009-07-07 at 22:28 -0400, Tres Seaver wrote: I would argue that enabling only localhost for the default forward proxy configuration is a sane default: people configuring things like bastions ought not to expect to use out-of-the box configs without review / tweakage, while people using Squid as a personal cache ought not to have to do such tweaks. Sure; as I said, a) we don't have really good data here, and b) I'm open to it being changed :) -Rob signature.asc Description: This is a digitally signed message part
Re: Hello from Mozilla
Robert Collins wrote: On Tue, 2009-07-07 at 22:28 -0400, Tres Seaver wrote: I would argue that enabling only localhost for the default forward proxy configuration is a sane default: people configuring things like bastions ought not to expect to use out-of-the box configs without review / tweakage, while people using Squid as a personal cache ought not to have to do such tweaks. Sure; as I said, a) we don't have really good data here, and b) I'm open to it being changed :) There changed already in HEAD. Will be effective from 3.1.0.10 and 3.0.STABLE17 Amos
