On Tue, Oct 10, 2023 at 07:23:25PM +0000, Tristan wrote: > Thanks for looking into it, Willy :-)
You're welcome. Actually last week I ran some tests *exactly* on this, thinking that it was the same principle as leaving orphaned sockets when you're limited on FDs. That's how I discovered the pool bug I mentioned the other day related to my recent change in -dev (that I still need to fix). I thought that it would be interesting to run a similar test on other stacks, but being busy on other more serious stuff and not being good at installing and testing software, I went back to my usual activities. I was aware that something was going to be announced without knowing what the problem was so I thought it was really not worth wasting more time on an issue if one was going to appear anyway, likely in the same area. I wouldn't have imagined it was exactly that one, though as it didn't look harmful enough (otherwise I would have fixed it last week and it could have looked like a leak for those working on theirs). But you can now understand better why I was particularly interested in having a deeper look into it now that we had the details ;-) > > On 10 Oct 2023, at 19:24, Willy Tarreau <w...@1wt.eu> wrote: > > [...] > > But for now if you site requires any of this, I can't see how it has not > > experienced weekly outages from standard attacks. > > Funny that you mention this; bit out of topic but we had enjoyed a relatively > peaceful 2023 for the most part, after some law enforcement operations last > December/January. [1] > > But things are picking up pace again since mid-summer, and this is indeed > just one of what feels like a new attack method a week that we hear of or > have the displeasure to receive... > Though most seem to be either L4 or non-HTTP L7. One important thing when you're sure you're dealing with targetted attacks is that you must not discuss them publicly, or you're giving a lot of info to your attackers about what you identify and how you're going to mitigate them. Sometimes you may even have to manually switch between an efficient config and a more robust one (the ".if" in the config sometimes help for this). I just hope you won't need that as it's quite annoying when it happens. > > As Tristan mentioned, lowering tune.h2.be.max-concurrent-streams may > > also slow them down but it will also slow down some sites with many > > objects (think shops with many images). For a long time the high > > parallelism of H2 was sold as a huge differentiator, I don't feel like > > starting to advertise lowering it now. > > For others that may be tempted to copypaste it and seeing errors, I think you > meant tune.h2.fe.max-... (emphasis on « fe ») Hehe indeed, I copy-pasted it from the doc but picked the wrong one! Thanks for correcting me! > > There are lots of other interesting attacks on the H2 protocol, that > > can be triggered just with a regular client with low timeouts, with low > > stream windows (use h2load -w 1 to have fun), zero-window during transfers, > > and even playing with one-byte continuation frames that may force some > > components to perform reallocations and copies. > > Are protections for these baked into the defaults/protocol or is there some > reading+tweaking one could consider for hardening purposes? As in even at the > cost of compatibility with « odd » clients. A bit of everything I would say. The implementation, the protocol, the timeouts, some possibilities with some rules when dealing with specific attacks, etc. The one about the small windows is more annoying. I've been thinking several times about whether we should implement a setting to reject connections with too small initial windows, but such approaches can also easily be worked around by the attacker accepting to download the initial data. What matters is always what it costs to the attacker to take you down. With small windows, the impact is only higher CPU usage and it still requires high bandwidth and low latency with the attacker. And when bandwidth and low latency are present, then attacks can switch to SSL keying which is quite cheap for a client. The H2 small windows attacks are not very different from L4 attacks advertising very small TCP MSS actually. Sometimes it's also when facing such corner cases that are difficult to address in one or another context that we decide that we need to add a new tunable. I just prefer to refrain from doing so until really necessary, in order to limit the complexity of counter-measures when they're needed! Cheers, Willy
