Re: Opinions desired on HTTP/2 config simplification
Ai Aleks, On Sat, Apr 15, 2023 at 10:12:00PM +0200, Aleksandar Lazic wrote: > Hi. > > On 15.04.23 11:32, Willy Tarreau wrote: > > Hi everyone, > > > > I was discussing with Tristan a few hours ago about the widespread > > deployment of H2 and H3, with Cloudflare showing that H1 only accounts > > for less than 7% of their traffic and H3 getting close to 30% [1], > > and the fact that on the opposite yesterday I heard someone say "we > > still have not tried H2, so H3..." (!). > > > > Tristan said something along the lines of "if only proxies would enable > > it by default by now", which resonated to me like when we decided to > > switch some defaults on (keep-alive, http-reuse, threads, etc). > > > > And it's true that at the beginning there was not even a question about > > enabling H2 by default on the edge, but nowadays it's as reliable as H1 > > and used by virtually everyone, yet it still requires admins to know > > about this TLS-specific extension called "ALPN" and the exact syntax of > > its declaration, in order to enable H2 over TLS, while it's already on > > by default for clear traffic. > > Is there any experience about the backends and what protocol they use? > As far as I can see is QUIC/h3 not yet there, what's the plan to add QUIC/h3 > as backend protocol? It's not there yet. It's planned for the long term (no emergency, it will only be relevant once h3 is common on the backends). And regarding H2 on the backend, it's not recommended if you're serving objects of moderate to large size, as it can cause head-of-line blocking. That's the reason why we've disabled connection sharing on the backend between multiple clients, because a slow client can cause some buffers to fill and slow other clients down. Using larger buffers is generally not a good option (it increases the memory footprint which often results in slowing down everything), and using smaller H2 windows to the server forces the server to make small frames and constantly switch between streams, which is not efficient either. Overall the only way to use H2 efficiently on the backend is either when it mostly delivers small objects in parallel (e.g. shipping cart thumb images, or map tiles), or when limiting it to one stream per connection to only benefit from the faster binary encoding and more resilient connection. H3 will make this incredibly better by the way, since all streams are independent. > > Thus you're seeing me coming with my question: does anyone have any > > objection against turning "alpn h2,http/1.1" on by default for HTTP > > frontends, and "alpn h3" by default for QUIC frontends, and have a new > > "no-alpn" option to explicitly turn off ALPN negotiation on HTTP > > frontends e.g. for debugging ? This would mean that it would no longer > > be necessary to know the ALPN strings to configure these protocols. I > > have not looked at the code but I think it should not be too difficult. > > ALPN is always driven by the client anyway so the option states what we > > do with it when it's presented, thus it will not make anything magically > > fail. > > Get a +1 for turning on the default settings. Thanks for the feedback! > This must be highlighted in the documentation as it could break some working > setups which have not activated H2 on some the listener for some specific > reasons. I know, and that's also why I preferred to ask. > > And if we change this default, do you prefer that we do it for 2.8 that > > will be an LTS release and most likely to be shipped with next year's > > LTS distros, or do you prefer that we skip this one and start with 2.9, > > hence postpone to LTS distros of 2026 ? > > +1 for 2.8 . OK. > > Even if I wouldn't share my feelings, some would consider that I'm > > trying to influence their opinion, so I'll share them anyway :-) I > > think that with the status change from "experimental-but-supported" to > > "production" for QUIC in 2.8, having to manually and explicitly deal > > with 3 HTTP versions in modern configs while the default (h1) only > > corresponds to 7% of what clients prefer is probably an indicator that > > it's the right moment to simplify these a little bit. But I'm open to > > any argument in any direction. > > As the history shows, that the a lot of peoples reuses some sample configs I > would also consider to add a example quic+h2 setup in the examples directory > because the current example quick config looks somehow wrong. > > http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/quick-test.cfg;h=f27eeff432de116132d2df36121356af0938b8a4;hb=HEAD Note that this one was called "quick test" to allow to quickly perform a test, it's not related to the QUIC protocol (no "k" in the name) :-) > I would be nice when the package owner of the distributions would also adopt > there config examples but this is a decision which is done outside of > haproxy :-) We tried this in the past and it revealed difficult. It appears that many load balancer users think they're
Re: Opinions desired on HTTP/2 config simplification
Hi Daniel, On Sat, Apr 15, 2023 at 02:37:06PM -0400, Daniel Corbett wrote: > Hi Willy, > > On Sat, Apr 15, 2023, at 8:02 AM, Willy Tarreau wrote: > > Hi Ionel, > > > > On Sat, Apr 15, 2023 at 01:52:27PM +0200, Ionel GARDAIS wrote: > > > Hi Willy, > > > > > > Agree with that. > > > However, maybe a "common H2 troubleshooting guide" should be provided so > > > options like h2-workaround-bogus-websocket-clients will be highlighted if > > > any > > > trouble arise. > > > > Good point, I even forgot about that one! Thank you for this feedback! > > > I too think this is a good idea. The less options to remember to configure at > runtime the better. > > However, to add to Ionel's point, it might be good to also highlight in this > troubleshooting guide how ` tune.h2.initial-window-size` plays into upload > speeds. The current documentation does a good job at covering this but it > could be difficult for some to find (and may not even realize it's a tunable > option). Yes this is a good point. I think that adding a "tuning" section in the doc could be useful to centralize certain such things. Thanks, Willy
Re: Opinions desired on HTTP/2 config simplification
Hi. On 15.04.23 11:32, Willy Tarreau wrote: Hi everyone, I was discussing with Tristan a few hours ago about the widespread deployment of H2 and H3, with Cloudflare showing that H1 only accounts for less than 7% of their traffic and H3 getting close to 30% [1], and the fact that on the opposite yesterday I heard someone say "we still have not tried H2, so H3..." (!). Tristan said something along the lines of "if only proxies would enable it by default by now", which resonated to me like when we decided to switch some defaults on (keep-alive, http-reuse, threads, etc). And it's true that at the beginning there was not even a question about enabling H2 by default on the edge, but nowadays it's as reliable as H1 and used by virtually everyone, yet it still requires admins to know about this TLS-specific extension called "ALPN" and the exact syntax of its declaration, in order to enable H2 over TLS, while it's already on by default for clear traffic. Is there any experience about the backends and what protocol they use? As far as I can see is QUIC/h3 not yet there, what's the plan to add QUIC/h3 as backend protocol? ``` podman run --rm --network host --name haproy-test --entrypoint /bin/bash -it haproxy:latest Available multiplexer protocols : (protocols marked as cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2flags=HTX|HOL_RISK|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG : mode=HTTP side=FE|BE mux=H1flags=HTX h1 : mode=HTTP side=FE|BE mux=H1flags=HTX|NO_UPG : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG ``` Do you see any benifit when there is a quic e2e connection? Something like: client - Q/H3 - HAProxy - Q/H3 - Backend Thus you're seeing me coming with my question: does anyone have any objection against turning "alpn h2,http/1.1" on by default for HTTP frontends, and "alpn h3" by default for QUIC frontends, and have a new "no-alpn" option to explicitly turn off ALPN negotiation on HTTP frontends e.g. for debugging ? This would mean that it would no longer be necessary to know the ALPN strings to configure these protocols. I have not looked at the code but I think it should not be too difficult. ALPN is always driven by the client anyway so the option states what we do with it when it's presented, thus it will not make anything magically fail. Get a +1 for turning on the default settings. This must be highlighted in the documentation as it could break some working setups which have not activated H2 on some the listener for some specific reasons. And if we change this default, do you prefer that we do it for 2.8 that will be an LTS release and most likely to be shipped with next year's LTS distros, or do you prefer that we skip this one and start with 2.9, hence postpone to LTS distros of 2026 ? +1 for 2.8 . Even if I wouldn't share my feelings, some would consider that I'm trying to influence their opinion, so I'll share them anyway :-) I think that with the status change from "experimental-but-supported" to "production" for QUIC in 2.8, having to manually and explicitly deal with 3 HTTP versions in modern configs while the default (h1) only corresponds to 7% of what clients prefer is probably an indicator that it's the right moment to simplify these a little bit. But I'm open to any argument in any direction. As the history shows, that the a lot of peoples reuses some sample configs I would also consider to add a example quic+h2 setup in the examples directory because the current example quick config looks somehow wrong. http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/quick-test.cfg;h=f27eeff432de116132d2df36121356af0938b8a4;hb=HEAD I would be nice when the package owner of the distributions would also adopt there config examples but this is a decision which is done outside of haproxy :-) It would be nice to be able to decide (and implement a change if needed) before next week's dev8, so that it leaves some time to collect feedback before end of May, so please voice in! Thanks! Willy [1] https://radar.cloudflare.com/adoption-and-usage Regards Alex
Re: Opinions desired on HTTP/2 config simplification
Hi Willy, On Sat, Apr 15, 2023, at 8:02 AM, Willy Tarreau wrote: > Hi Ionel, > > On Sat, Apr 15, 2023 at 01:52:27PM +0200, Ionel GARDAIS wrote: > > Hi Willy, > > > > Agree with that. > > However, maybe a "common H2 troubleshooting guide" should be provided so > > options like h2-workaround-bogus-websocket-clients will be highlighted if > > any > > trouble arise. > > Good point, I even forgot about that one! Thank you for this feedback! I too think this is a good idea. The less options to remember to configure at runtime the better. However, to add to Ionel's point, it might be good to also highlight in this troubleshooting guide how ` tune.h2.initial-window-size` plays into upload speeds. The current documentation does a good job at covering this but it could be difficult for some to find (and may not even realize it's a tunable option). I know I have seen in the past several issues arise for some after enabling HTTP/2 in regards to slow upload speeds and not realizing that they may need to also increase ` tune.h2.initial-window-size`. Thanks, -- Daniel
Re: Opinions desired on HTTP/2 config simplification
On Sat, Apr 15, 2023 at 05:50:53PM +0200, Pavlos Parissis wrote: > On Saturday, April 15, 2023 11:32:49 AM CEST Willy Tarreau wrote: > > Hi everyone, > > [...snip...] > > Even if I wouldn't share my feelings, some would consider that I'm > > trying to influence their opinion, so I'll share them anyway :-) I > > think that with the status change from "experimental-but-supported" to > > "production" for QUIC in 2.8, having to manually and explicitly deal > > with 3 HTTP versions in modern configs while the default (h1) only > > corresponds to 7% of what clients prefer is probably an indicator that > > it's the right moment to simplify these a little bit. But I'm open to > > any argument in any direction. > > > > It would be nice to be able to decide (and implement a change if needed) > > before next week's dev8, so that it leaves some time to collect feedback > > before end of May, so please voice in! > > > > I agree with enabling H2 by default and I also agree that 2.8 version is very > good candidate for > introducing that change. OK, thank you Pavlos! Willy
Re: Opinions desired on HTTP/2 config simplification
On Saturday, April 15, 2023 11:32:49 AM CEST Willy Tarreau wrote: > Hi everyone, [...snip...] > Even if I wouldn't share my feelings, some would consider that I'm > trying to influence their opinion, so I'll share them anyway :-) I > think that with the status change from "experimental-but-supported" to > "production" for QUIC in 2.8, having to manually and explicitly deal > with 3 HTTP versions in modern configs while the default (h1) only > corresponds to 7% of what clients prefer is probably an indicator that > it's the right moment to simplify these a little bit. But I'm open to > any argument in any direction. > > It would be nice to be able to decide (and implement a change if needed) > before next week's dev8, so that it leaves some time to collect feedback > before end of May, so please voice in! > I agree with enabling H2 by default and I also agree that 2.8 version is very good candidate for introducing that change. My 2 cents, Pavlos
Re: Opinions desired on HTTP/2 config simplification
Hi John, On Sat, Apr 15, 2023 at 10:20:22AM -0400, John Lauro wrote: > I agree defaulting to alpn h2,http/1.1 sooner (don't wait for 2.9), > and even 2.6 would be fine IMO. Wouldn't be a new feature for 2.6, > only a non breaking (AFAIK) default change... We won't change 2.6 now since it's already released and that this could require some config change for some users. > I would have concerns making QUIC default for 443 ssl (especially > prior to 2.8), but you are not suggesting that anyways. Don't get me wrong, you still need to have an explicit "bind" line for QUIC, it's just that right now it looks like this: bind quic4@:4443 ssl crt rsa+dh2048.pem alpn h3 allow-0rtt And without "alpn h3" it will just not work, so what I indend is to have this "alpn h3" by default for "quic" lines, but in no case enable QUIC by default when SSL is in use. Thanks! Willy
Re: Opinions desired on HTTP/2 config simplification
I agree defaulting to alpn h2,http/1.1 sooner (don't wait for 2.9), and even 2.6 would be fine IMO. Wouldn't be a new feature for 2.6, only a non breaking (AFAIK) default change... I would have concerns making QUIC default for 443 ssl (especially prior to 2.8), but you are not suggesting that anyways. On Sat, Apr 15, 2023 at 5:33 AM Willy Tarreau wrote: > > Hi everyone, > > I was discussing with Tristan a few hours ago about the widespread > deployment of H2 and H3, with Cloudflare showing that H1 only accounts > for less than 7% of their traffic and H3 getting close to 30% [1], > and the fact that on the opposite yesterday I heard someone say "we > still have not tried H2, so H3..." (!). > > Tristan said something along the lines of "if only proxies would enable > it by default by now", which resonated to me like when we decided to > switch some defaults on (keep-alive, http-reuse, threads, etc). > > And it's true that at the beginning there was not even a question about > enabling H2 by default on the edge, but nowadays it's as reliable as H1 > and used by virtually everyone, yet it still requires admins to know > about this TLS-specific extension called "ALPN" and the exact syntax of > its declaration, in order to enable H2 over TLS, while it's already on > by default for clear traffic. > > Thus you're seeing me coming with my question: does anyone have any > objection against turning "alpn h2,http/1.1" on by default for HTTP > frontends, and "alpn h3" by default for QUIC frontends, and have a new > "no-alpn" option to explicitly turn off ALPN negotiation on HTTP > frontends e.g. for debugging ? This would mean that it would no longer > be necessary to know the ALPN strings to configure these protocols. I > have not looked at the code but I think it should not be too difficult. > ALPN is always driven by the client anyway so the option states what we > do with it when it's presented, thus it will not make anything magically > fail. > > And if we change this default, do you prefer that we do it for 2.8 that > will be an LTS release and most likely to be shipped with next year's > LTS distros, or do you prefer that we skip this one and start with 2.9, > hence postpone to LTS distros of 2026 ? > > Even if I wouldn't share my feelings, some would consider that I'm > trying to influence their opinion, so I'll share them anyway :-) I > think that with the status change from "experimental-but-supported" to > "production" for QUIC in 2.8, having to manually and explicitly deal > with 3 HTTP versions in modern configs while the default (h1) only > corresponds to 7% of what clients prefer is probably an indicator that > it's the right moment to simplify these a little bit. But I'm open to > any argument in any direction. > > It would be nice to be able to decide (and implement a change if needed) > before next week's dev8, so that it leaves some time to collect feedback > before end of May, so please voice in! > > Thanks! > Willy > > [1] https://radar.cloudflare.com/adoption-and-usage >
Re: Opinions desired on HTTP/2 config simplification
Hi Lukas, On Sat, Apr 15, 2023 at 02:08:03PM +0200, Lukas Tribus wrote: > Hi, > > On Sat, 15 Apr 2023 at 11:32, Willy Tarreau wrote: > > Thus you're seeing me coming with my question: does anyone have any > > objection against turning "alpn h2,http/1.1" on by default for HTTP > > frontends, and "alpn h3" by default for QUIC frontends, and have a new > > "no-alpn" option to explicitly turn off ALPN negotiation on HTTP > > frontends e.g. for debugging ? > > For H2 I agree, and just to state which I think it's obvious: when > alpn is already configured the behaviour will remain as-is, the change > would only impact bind lines without alpn keyword. Absolutely, that's how I'm seeing it. > For QUIC frontends, they are explicit anyway, so yes, I think "alpn > h3" should be implicit in HTTP mode, unless we expect to negotiate > different alpn protocols on top of QUIC (but I guess in this case the > frontend would be in TCP mode). Yes indeed, and I still have no idea how long it will take before we see new protocols being transported over QUIC in a mainstream way (but I don't want to go into any direction making this impossible of course). > > And if we change this default, do you prefer that we do it for 2.8 that > > will be an LTS release and most likely to be shipped with next year's > > LTS distros, or do you prefer that we skip this one and start with 2.9, > > hence postpone to LTS distros of 2026 ? > > I agree this should be in 2.8 LTS. OK, thank you very much! Willy
Re: Opinions desired on HTTP/2 config simplification
Hi, On Sat, 15 Apr 2023 at 11:32, Willy Tarreau wrote: > Thus you're seeing me coming with my question: does anyone have any > objection against turning "alpn h2,http/1.1" on by default for HTTP > frontends, and "alpn h3" by default for QUIC frontends, and have a new > "no-alpn" option to explicitly turn off ALPN negotiation on HTTP > frontends e.g. for debugging ? For H2 I agree, and just to state which I think it's obvious: when alpn is already configured the behaviour will remain as-is, the change would only impact bind lines without alpn keyword. For QUIC frontends, they are explicit anyway, so yes, I think "alpn h3" should be implicit in HTTP mode, unless we expect to negotiate different alpn protocols on top of QUIC (but I guess in this case the frontend would be in TCP mode). > And if we change this default, do you prefer that we do it for 2.8 that > will be an LTS release and most likely to be shipped with next year's > LTS distros, or do you prefer that we skip this one and start with 2.9, > hence postpone to LTS distros of 2026 ? I agree this should be in 2.8 LTS. Lukas
Re: Opinions desired on HTTP/2 config simplification
Hi Ionel, On Sat, Apr 15, 2023 at 01:52:27PM +0200, Ionel GARDAIS wrote: > Hi Willy, > > Agree with that. > However, maybe a "common H2 troubleshooting guide" should be provided so > options like h2-workaround-bogus-websocket-clients will be highlighted if any > trouble arise. Good point, I even forgot about that one! Thank you for this feedback! Willy
Re: [*EXT*] Opinions desired on HTTP/2 config simplification
Hi Willy, Agree with that. However, maybe a "common H2 troubleshooting guide" should be provided so options like h2-workaround-bogus-websocket-clients will be highlighted if any trouble arise. -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Willy Tarreau" À: "haproxy" Envoyé: Samedi 15 Avril 2023 11:32:49 Objet: [*EXT*] Opinions desired on HTTP/2 config simplification Hi everyone, I was discussing with Tristan a few hours ago about the widespread deployment of H2 and H3, with Cloudflare showing that H1 only accounts for less than 7% of their traffic and H3 getting close to 30% [1], and the fact that on the opposite yesterday I heard someone say "we still have not tried H2, so H3..." (!). Tristan said something along the lines of "if only proxies would enable it by default by now", which resonated to me like when we decided to switch some defaults on (keep-alive, http-reuse, threads, etc). And it's true that at the beginning there was not even a question about enabling H2 by default on the edge, but nowadays it's as reliable as H1 and used by virtually everyone, yet it still requires admins to know about this TLS-specific extension called "ALPN" and the exact syntax of its declaration, in order to enable H2 over TLS, while it's already on by default for clear traffic. Thus you're seeing me coming with my question: does anyone have any objection against turning "alpn h2,http/1.1" on by default for HTTP frontends, and "alpn h3" by default for QUIC frontends, and have a new "no-alpn" option to explicitly turn off ALPN negotiation on HTTP frontends e.g. for debugging ? This would mean that it would no longer be necessary to know the ALPN strings to configure these protocols. I have not looked at the code but I think it should not be too difficult. ALPN is always driven by the client anyway so the option states what we do with it when it's presented, thus it will not make anything magically fail. And if we change this default, do you prefer that we do it for 2.8 that will be an LTS release and most likely to be shipped with next year's LTS distros, or do you prefer that we skip this one and start with 2.9, hence postpone to LTS distros of 2026 ? Even if I wouldn't share my feelings, some would consider that I'm trying to influence their opinion, so I'll share them anyway :-) I think that with the status change from "experimental-but-supported" to "production" for QUIC in 2.8, having to manually and explicitly deal with 3 HTTP versions in modern configs while the default (h1) only corresponds to 7% of what clients prefer is probably an indicator that it's the right moment to simplify these a little bit. But I'm open to any argument in any direction. It would be nice to be able to decide (and implement a change if needed) before next week's dev8, so that it leaves some time to collect feedback before end of May, so please voice in! Thanks! Willy [1] https://radar.cloudflare.com/adoption-and-usage -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Opinions desired on HTTP/2 config simplification
Hi everyone, I was discussing with Tristan a few hours ago about the widespread deployment of H2 and H3, with Cloudflare showing that H1 only accounts for less than 7% of their traffic and H3 getting close to 30% [1], and the fact that on the opposite yesterday I heard someone say "we still have not tried H2, so H3..." (!). Tristan said something along the lines of "if only proxies would enable it by default by now", which resonated to me like when we decided to switch some defaults on (keep-alive, http-reuse, threads, etc). And it's true that at the beginning there was not even a question about enabling H2 by default on the edge, but nowadays it's as reliable as H1 and used by virtually everyone, yet it still requires admins to know about this TLS-specific extension called "ALPN" and the exact syntax of its declaration, in order to enable H2 over TLS, while it's already on by default for clear traffic. Thus you're seeing me coming with my question: does anyone have any objection against turning "alpn h2,http/1.1" on by default for HTTP frontends, and "alpn h3" by default for QUIC frontends, and have a new "no-alpn" option to explicitly turn off ALPN negotiation on HTTP frontends e.g. for debugging ? This would mean that it would no longer be necessary to know the ALPN strings to configure these protocols. I have not looked at the code but I think it should not be too difficult. ALPN is always driven by the client anyway so the option states what we do with it when it's presented, thus it will not make anything magically fail. And if we change this default, do you prefer that we do it for 2.8 that will be an LTS release and most likely to be shipped with next year's LTS distros, or do you prefer that we skip this one and start with 2.9, hence postpone to LTS distros of 2026 ? Even if I wouldn't share my feelings, some would consider that I'm trying to influence their opinion, so I'll share them anyway :-) I think that with the status change from "experimental-but-supported" to "production" for QUIC in 2.8, having to manually and explicitly deal with 3 HTTP versions in modern configs while the default (h1) only corresponds to 7% of what clients prefer is probably an indicator that it's the right moment to simplify these a little bit. But I'm open to any argument in any direction. It would be nice to be able to decide (and implement a change if needed) before next week's dev8, so that it leaves some time to collect feedback before end of May, so please voice in! Thanks! Willy [1] https://radar.cloudflare.com/adoption-and-usage
