Re: Bad backend selected
Hi Tim, On Sat, Jun 26, 2021 at 08:20:20PM +0200, Tim Düsterhus wrote: > Willy, Amaury, > > On 6/18/21 5:55 PM, Willy Tarreau wrote: > > Amaury started something in this direction but it was only in H2 and > > I'd like that we explore the ability to do it the most generic way > > possible so as not to have different behaviours depending on where > > the requests enter and where they leave. I know it's tricky, which is > > one more reason for exploring this possibility. At least regardless > > of the final choice, we'll be certain not to have left unknown cases > > incorrectly handled. > > > > Thus no need to create a new issue for now, hopefully by next week > > we'll all agree on a durable solution. > > FYI: Another user in IRC (#haproxy on Libera.Chat) just stumbled upon this > issue as well. So it appears it's a reasonably common occurence with Firefox > and HAProxy 2.4. Just a quick heads up on this since nobody responded, we had a long discussion on the subject and Amaury proposed some interesting patches that we've been discussing and that will experience a few changes to preserve performance before being merged, but we're getting close to having them merged and backported now. Cheers, Willy
Re: Bad backend selected
Willy, Amaury, On 6/18/21 5:55 PM, Willy Tarreau wrote: Amaury started something in this direction but it was only in H2 and I'd like that we explore the ability to do it the most generic way possible so as not to have different behaviours depending on where the requests enter and where they leave. I know it's tricky, which is one more reason for exploring this possibility. At least regardless of the final choice, we'll be certain not to have left unknown cases incorrectly handled. Thus no need to create a new issue for now, hopefully by next week we'll all agree on a durable solution. FYI: Another user in IRC (#haproxy on Libera.Chat) just stumbled upon this issue as well. So it appears it's a reasonably common occurence with Firefox and HAProxy 2.4. Best regards Tim Düsterhus
Re: Bad backend selected
On Fri, Jun 18, 2021 at 05:08:56PM +0200, Tim Düsterhus wrote: > > So I had some thoughts about that discussion that started off-list. And > > now I think that the right thing to do is to always drop the port part > > of the authority when we have a scheme for which it's the default. I.e. > > if the scheme is "http" we drop ":80", and if the scheme is "https" we > > drop ":443". This will always be consistent with the standards, and by > > doing it early (i.e. during conversion to HTX) we're certain to address > > both the conversion of CONNECT to GET+Upgrade, and the hdr(host) match. > > Is this still on your radar? Should I file an issue for that? Amaury started something in this direction but it was only in H2 and I'd like that we explore the ability to do it the most generic way possible so as not to have different behaviours depending on where the requests enter and where they leave. I know it's tricky, which is one more reason for exploring this possibility. At least regardless of the final choice, we'll be certain not to have left unknown cases incorrectly handled. Thus no need to create a new issue for now, hopefully by next week we'll all agree on a durable solution. Thanks, Willy
Re: Bad backend selected
Willy, On 6/8/21 8:37 AM, Willy Tarreau wrote: However the only difference is the 443 port explicitly specified in the later request. I am not sure it's something specific to 2.4.0, but I've never seen it before. Is it an expected behaviour ? If so, how can I change my acls to correct it ? I encountered the same issue (incidentally also with socket.io). It's happening for WebSockets via HTTP/2. These are newly supported starting with HAProxy 2.4. The "broken" requests are most likely Firefox, while the working ones are not Firefox. I already have a private email thread with a few developers regarding this behavior. So I had some thoughts about that discussion that started off-list. And now I think that the right thing to do is to always drop the port part of the authority when we have a scheme for which it's the default. I.e. if the scheme is "http" we drop ":80", and if the scheme is "https" we drop ":443". This will always be consistent with the standards, and by doing it early (i.e. during conversion to HTX) we're certain to address both the conversion of CONNECT to GET+Upgrade, and the hdr(host) match. Is this still on your radar? Should I file an issue for that? Best regards Tim Düsterhus
Re: Bad backend selected
On Mon, Jun 07, 2021 at 07:13:43PM +0200, t...@bastelstu.be wrote: > Artur, > > [cc'ing Amaury] > > Am 2021-06-07 16:46, schrieb Artur: > > However the only difference is the 443 port explicitly specified in the > > later request. > > I am not sure it's something specific to 2.4.0, but I've never seen it > > before. > > Is it an expected behaviour ? If so, how can I change my acls to correct > > it ? > > I encountered the same issue (incidentally also with socket.io). It's > happening for WebSockets via HTTP/2. These are newly supported starting with > HAProxy 2.4. The "broken" requests are most likely Firefox, while the > working ones are not Firefox. I already have a private email thread with a > few developers regarding this behavior. So I had some thoughts about that discussion that started off-list. And now I think that the right thing to do is to always drop the port part of the authority when we have a scheme for which it's the default. I.e. if the scheme is "http" we drop ":80", and if the scheme is "https" we drop ":443". This will always be consistent with the standards, and by doing it early (i.e. during conversion to HTX) we're certain to address both the conversion of CONNECT to GET+Upgrade, and the hdr(host) match. We need to be careful when re-emitting a CONNECT request however, as the port is mandatory with it. But normally with this we should be fine and we should avoid any dirty hack in the middle. Willy
Re: Bad backend selected
Hello Tim, Le 07/06/2021 à 19:13, t...@bastelstu.be a écrit : > Artur, > > [cc'ing Amaury] > > Am 2021-06-07 16:46, schrieb Artur: >> However the only difference is the 443 port explicitly specified in the >> later request. >> I am not sure it's something specific to 2.4.0, but I've never seen it >> before. >> Is it an expected behaviour ? If so, how can I change my acls to correct >> it ? > > I encountered the same issue (incidentally also with socket.io). It's > happening for WebSockets via HTTP/2. These are newly supported > starting with HAProxy 2.4. The "broken" requests are most likely > Firefox, while the working ones are not Firefox. I already have a > private email thread with a few developers regarding this behavior. > > Best regards > Tim Düsterhus My problem was mainly caused by the port appearing in the url and an incorrect use of hdr(host) leading to bad backend choice. Once the websocket connection request correctly forwarded to the right backend I can't see any problem (request ended with 101 websocket upgrade http status code). However I have no idea how the 443 port appeared in request, I was unable to reproduce this kind of requests myself with Firefox or other browsers. Quite strange. FYI, our application use socket.io(-client) node.js/javascript modules. -- Best regards, Artur
Re: Bad backend selected
Artur, [cc'ing Amaury] Am 2021-06-07 16:46, schrieb Artur: However the only difference is the 443 port explicitly specified in the later request. I am not sure it's something specific to 2.4.0, but I've never seen it before. Is it an expected behaviour ? If so, how can I change my acls to correct it ? I encountered the same issue (incidentally also with socket.io). It's happening for WebSockets via HTTP/2. These are newly supported starting with HAProxy 2.4. The "broken" requests are most likely Firefox, while the working ones are not Firefox. I already have a private email thread with a few developers regarding this behavior. Best regards Tim Düsterhus
Re: Bad backend selected
Le 07/06/2021 à 17:22, Jarno Huuskonen a écrit : > Hello, > > On Mon, 2021-06-07 at 16:46 +0200, Artur wrote: >> Hello, >> >> I'm currently running haproxy 2.4.0 and I can see something strange in >> the way haproxy selects a backend for processing some requests. >> >> This is simplified frontend configuration that should select between >> static and dynamic (websocket) content URIs based on path_beg. >> >> frontend wwws >> bind 0.0.0.0:443 ssl crt /etc/haproxy/ssl/server.pem alpn >> h2,http/1.1 >> mode http >> >> acl is_static_prod31 path_beg /p31/ >> acl is_dynamic_prod31 path_beg /n/p31/ >> acl is_domain_name hdr(host) -i domain.name >> >> use_backend ws_be_prod31 if is_dynamic_prod31 is_domain_name >> use_backend www_be_prod if is_static_prod31 is_domain_name >> >> default_backend www_be_prod >> >> What I can see in logs is that some requests are correctly processed and >> redirected to dynamic backends (websockets servers) for processing : >> >> Jun 7 15:44:41 host haproxy[9384]: 1.2.3.4:56952 >> [07/Jun/2021:15:43:31.926] wwws~ ws_be_prod31/s1 5/0/1/3/70015 101 421 - >> - --VN 34/34/27/8/0 0/0 "GET https://domain.name/n/p31/socket.io/... >> HTTP/2.0" >> >> While others are wrongly processed by the static web server : >> >> Jun 7 15:50:06 host haproxy[9384]: 1.2.3.4:61037 >> [07/Jun/2021:15:50:06.157] wwws~ www_be_prod/web1 6/0/1/1/7 404 9318 - - >> 34/34/0/0/0 0/0 "GET https://domain.name:443/n/p31/socket.io/... >> HTTP/2.0" >> >> However the only difference is the 443 port explicitly specified in the >> later request. >> I am not sure it's something specific to 2.4.0, but I've never seen it >> before. >> Is it an expected behaviour ? If so, how can I change my acls to correct >> it ? > Does it work if you use > hdr_dom(https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#7.3.6-req.hdr) > for the host header acl: > (acl is_domain_name hdr_dom(host) -i domain.name) > (or some other match that ignores port in Host header). > > -Jarno Yes, it seems to work fine now. Thank you. I realized the port number is part of Host: header if explicitly specified in request. However as in my setup (removed part) I also have to check for dev* hostnames I would like to know the exact hdr_dom(host) behaviour. With this example : acl acl1 hdr_dom(host) -i domain.name 1) Host: domain.name:443 -> acl1 matches 2) Host: dimain.name -> acl1 matches 3) Host: dev.domain.name:443 -> acl1 does not match 4) Host: dev.domain.name -> acl1 does not match Am I right ? (I suppose I can also use hdr_beg(host) to check for the beginning of the hostname) -- Best regards, Artur
Re: Bad backend selected
Hello, On Mon, 2021-06-07 at 16:46 +0200, Artur wrote: > Hello, > > I'm currently running haproxy 2.4.0 and I can see something strange in > the way haproxy selects a backend for processing some requests. > > This is simplified frontend configuration that should select between > static and dynamic (websocket) content URIs based on path_beg. > > frontend wwws > bind 0.0.0.0:443 ssl crt /etc/haproxy/ssl/server.pem alpn > h2,http/1.1 > mode http > > acl is_static_prod31 path_beg /p31/ > acl is_dynamic_prod31 path_beg /n/p31/ > acl is_domain_name hdr(host) -i domain.name > > use_backend ws_be_prod31 if is_dynamic_prod31 is_domain_name > use_backend www_be_prod if is_static_prod31 is_domain_name > > default_backend www_be_prod > > What I can see in logs is that some requests are correctly processed and > redirected to dynamic backends (websockets servers) for processing : > > Jun 7 15:44:41 host haproxy[9384]: 1.2.3.4:56952 > [07/Jun/2021:15:43:31.926] wwws~ ws_be_prod31/s1 5/0/1/3/70015 101 421 - > - --VN 34/34/27/8/0 0/0 "GET https://domain.name/n/p31/socket.io/... > HTTP/2.0" > > While others are wrongly processed by the static web server : > > Jun 7 15:50:06 host haproxy[9384]: 1.2.3.4:61037 > [07/Jun/2021:15:50:06.157] wwws~ www_be_prod/web1 6/0/1/1/7 404 9318 - - > 34/34/0/0/0 0/0 "GET https://domain.name:443/n/p31/socket.io/... > HTTP/2.0" > > However the only difference is the 443 port explicitly specified in the > later request. > I am not sure it's something specific to 2.4.0, but I've never seen it > before. > Is it an expected behaviour ? If so, how can I change my acls to correct > it ? Does it work if you use hdr_dom(https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#7.3.6-req.hdr) for the host header acl: (acl is_domain_name hdr_dom(host) -i domain.name) (or some other match that ignores port in Host header). -Jarno -- Jarno Huuskonen
Bad backend selected
Hello, I'm currently running haproxy 2.4.0 and I can see something strange in the way haproxy selects a backend for processing some requests. This is simplified frontend configuration that should select between static and dynamic (websocket) content URIs based on path_beg. frontend wwws bind 0.0.0.0:443 ssl crt /etc/haproxy/ssl/server.pem alpn h2,http/1.1 mode http acl is_static_prod31 path_beg /p31/ acl is_dynamic_prod31 path_beg /n/p31/ acl is_domain_name hdr(host) -i domain.name use_backend ws_be_prod31 if is_dynamic_prod31 is_domain_name use_backend www_be_prod if is_static_prod31 is_domain_name default_backend www_be_prod What I can see in logs is that some requests are correctly processed and redirected to dynamic backends (websockets servers) for processing : Jun 7 15:44:41 host haproxy[9384]: 1.2.3.4:56952 [07/Jun/2021:15:43:31.926] wwws~ ws_be_prod31/s1 5/0/1/3/70015 101 421 - - --VN 34/34/27/8/0 0/0 "GET https://domain.name/n/p31/socket.io/... HTTP/2.0" While others are wrongly processed by the static web server : Jun 7 15:50:06 host haproxy[9384]: 1.2.3.4:61037 [07/Jun/2021:15:50:06.157] wwws~ www_be_prod/web1 6/0/1/1/7 404 9318 - - 34/34/0/0/0 0/0 "GET https://domain.name:443/n/p31/socket.io/... HTTP/2.0" However the only difference is the 443 port explicitly specified in the later request. I am not sure it's something specific to 2.4.0, but I've never seen it before. Is it an expected behaviour ? If so, how can I change my acls to correct it ? -- Best regards, Artur
