Re: Options for mitigating CVE-2023-44487 with HAProxy
On Tue, Oct 10, 2023 at 10:03:32PM +, Lukas Tribus wrote: > On Tue, 10 Oct 2023 at 20:22, Willy Tarreau wrote: > > > > So at this point I'm still failing to find any case where this attack > > hurts haproxy more than any of the benchmarks we're routinely inflicting > > it, given that it acts exactly like a client configured with a short > > timeout (e.g. if you configure haproxy with "timeout server 1" and > > have an h2 server, you will basically get the same traffic pattern). > > This is pretty much the situation with nginx as well: > > https://mailman.nginx.org/pipermail/nginx-devel/2023-October/S36Q5HBXR7CAIMPLLPRSSSYR4PCMWILK.html Thanks for the pointer Lukas. I'm not surprized at all, and Maxim has all my sympathy on this when he speaks about FUD. Stacks like haproxy and nginx which have been exposed for a long time to amazing amounts of garbage (and even to their own bugs sometimes) naturally had to be made more robust against this type of misbehavior. I'm noticing that they're counting resets and fixing some limits, but this can very easily be triggered by a legit client clicking stop a few times while loading many objects in parallel. Stefan Eissing said somewhere that in Apache they implement a "mood" on a connection indicating how apache feels about that connection and that as bad stuff happens its mood degrades and it progressively reduces the processing on that connection. I find this quite interesting because instead of taking many parameters into account, any event can say "that peer is not very nice with us". And we could count a ratio of number of streams that are reset before the response to the the total number of streams, as this is a normal behavior for a client clicking stop, but it should not represent high ratios. I was thinking that we could imagine sending settings with a reduced SETTINGS_MAX_CONCURRENT_STREAMS when bad stuff like this happens. I remember that we mentioned a while ago that it could be useful to implement a total stream limit per connection to force to reconnect, but my primary use case was to force h2load to disconnect/reconnect in tests. It could also increase the cost a little bit for the client here, though it would also increase the cost for the proxy by forcing new SSL connections more often. Amaury told me that H3/QUIC was not affected by this thanks to the way streams are flow-controlled, since the server must send a frame to allow the client to create new streams. And it turns out that Martin Thomson of the IETF HTTP WG proposed something very similar which could improve the situation: https://martinthomson.github.io/h2-stream-limits/draft-thomson-httpbis-h2-stream-limits.html So we can expect improvements in that area in the foreseable future. Willy
Re: Options for mitigating CVE-2023-44487 with HAProxy
On Tue, 10 Oct 2023 at 20:22, Willy Tarreau wrote: > > So at this point I'm still failing to find any case where this attack > hurts haproxy more than any of the benchmarks we're routinely inflicting > it, given that it acts exactly like a client configured with a short > timeout (e.g. if you configure haproxy with "timeout server 1" and > have an h2 server, you will basically get the same traffic pattern). This is pretty much the situation with nginx as well: https://mailman.nginx.org/pipermail/nginx-devel/2023-October/S36Q5HBXR7CAIMPLLPRSSSYR4PCMWILK.html Lukas
Re: Options for mitigating CVE-2023-44487 with HAProxy
On Tue, Oct 10, 2023 at 07:23:25PM +, Tristan wrote: > Thanks for looking into it, Willy :-) You're welcome. Actually last week I ran some tests *exactly* on this, thinking that it was the same principle as leaving orphaned sockets when you're limited on FDs. That's how I discovered the pool bug I mentioned the other day related to my recent change in -dev (that I still need to fix). I thought that it would be interesting to run a similar test on other stacks, but being busy on other more serious stuff and not being good at installing and testing software, I went back to my usual activities. I was aware that something was going to be announced without knowing what the problem was so I thought it was really not worth wasting more time on an issue if one was going to appear anyway, likely in the same area. I wouldn't have imagined it was exactly that one, though as it didn't look harmful enough (otherwise I would have fixed it last week and it could have looked like a leak for those working on theirs). But you can now understand better why I was particularly interested in having a deeper look into it now that we had the details ;-) > > On 10 Oct 2023, at 19:24, Willy Tarreau wrote: > > [...] > > But for now if you site requires any of this, I can't see how it has not > > experienced weekly outages from standard attacks. > > Funny that you mention this; bit out of topic but we had enjoyed a relatively > peaceful 2023 for the most part, after some law enforcement operations last > December/January. [1] > > But things are picking up pace again since mid-summer, and this is indeed > just one of what feels like a new attack method a week that we hear of or > have the displeasure to receive... > Though most seem to be either L4 or non-HTTP L7. One important thing when you're sure you're dealing with targetted attacks is that you must not discuss them publicly, or you're giving a lot of info to your attackers about what you identify and how you're going to mitigate them. Sometimes you may even have to manually switch between an efficient config and a more robust one (the ".if" in the config sometimes help for this). I just hope you won't need that as it's quite annoying when it happens. > > As Tristan mentioned, lowering tune.h2.be.max-concurrent-streams may > > also slow them down but it will also slow down some sites with many > > objects (think shops with many images). For a long time the high > > parallelism of H2 was sold as a huge differentiator, I don't feel like > > starting to advertise lowering it now. > > For others that may be tempted to copypaste it and seeing errors, I think you > meant tune.h2.fe.max-... (emphasis on « fe ») Hehe indeed, I copy-pasted it from the doc but picked the wrong one! Thanks for correcting me! > > There are lots of other interesting attacks on the H2 protocol, that > > can be triggered just with a regular client with low timeouts, with low > > stream windows (use h2load -w 1 to have fun), zero-window during transfers, > > and even playing with one-byte continuation frames that may force some > > components to perform reallocations and copies. > > Are protections for these baked into the defaults/protocol or is there some > reading+tweaking one could consider for hardening purposes? As in even at the > cost of compatibility with « odd » clients. A bit of everything I would say. The implementation, the protocol, the timeouts, some possibilities with some rules when dealing with specific attacks, etc. The one about the small windows is more annoying. I've been thinking several times about whether we should implement a setting to reject connections with too small initial windows, but such approaches can also easily be worked around by the attacker accepting to download the initial data. What matters is always what it costs to the attacker to take you down. With small windows, the impact is only higher CPU usage and it still requires high bandwidth and low latency with the attacker. And when bandwidth and low latency are present, then attacks can switch to SSL keying which is quite cheap for a client. The H2 small windows attacks are not very different from L4 attacks advertising very small TCP MSS actually. Sometimes it's also when facing such corner cases that are difficult to address in one or another context that we decide that we need to add a new tunable. I just prefer to refrain from doing so until really necessary, in order to limit the complexity of counter-measures when they're needed! Cheers, Willy
Re: Options for mitigating CVE-2023-44487 with HAProxy
Thanks for looking into it, Willy :-) > On 10 Oct 2023, at 19:24, Willy Tarreau wrote: > […] > But for now if you site requires any of this, I can't see how it has not > experienced weekly outages from standard attacks. Funny that you mention this; bit out of topic but we had enjoyed a relatively peaceful 2023 for the most part, after some law enforcement operations last December/January. [1] But things are picking up pace again since mid-summer, and this is indeed just one of what feels like a new attack method a week that we hear of or have the displeasure to receive… Though most seem to be either L4 or non-HTTP L7. > As Tristan mentioned, lowering tune.h2.be.max-concurrent-streams may > also slow them down but it will also slow down some sites with many > objects (think shops with many images). For a long time the high > parallelism of H2 was sold as a huge differentiator, I don't feel like > starting to advertise lowering it now. For others that may be tempted to copypaste it and seeing errors, I think you meant tune.h2.fe.max-… (emphasis on « fe ») > There are lots of other interesting attacks on the H2 protocol, that > can be triggered just with a regular client with low timeouts, with low > stream windows (use h2load -w 1 to have fun), zero-window during transfers, > and even playing with one-byte continuation frames that may force some > components to perform reallocations and copies. Are protections for these baked into the defaults/protocol or is there some reading+tweaking one could consider for hardening purposes? As in even at the cost of compatibility with « odd » clients. Regards, Tristan [1] https://krebsonsecurity.com/2022/12/six-charged-in-mass-takedown-of-ddos-for-hire-sites/
Re: Options for mitigating CVE-2023-44487 with HAProxy
On Tue, Oct 10, 2023 at 03:57:09PM +0200, Willy Tarreau wrote:
> On Tue, Oct 10, 2023 at 03:49:21PM +0200, Willy Tarreau wrote:
> > > Seems like a clever update to the "good old" h2 multiplexing abuse
> > > vectors:
> > > 1. client opens a lot of H2 streams on a connection
> > > 2. Spams some requests
> > > 3. immediately sends h2 RST frames for all of them
> > > 4. Go back to 1. repeat.
> >
> > Yes, precisely one of those I tested last week among the usual approchaes
> > consisting in creating tons of streams while staying within the protocol's
> > validity limits. The only thing it did was to detect the pool issue I
> > mentioned in the dev7 announce.
> >
> > > The idea being to cause resource exhaustion on the server/proxy at least
> > > when it allocates stream related buffers etc, and the underlying server
> > > too
> > > since it likely sees the requests before they get cancelled.
> > >
> > > Looking at HAProxy I'd like to know if someone's aware of a decent
> > > mitigation option?
> >
> > We first need to check if at all we're affected, since we keep a count
> > of the attached streams for precisely this case and we refrain from
> > processing headers frames when we have too many streams, so normally
> > the mux will pause waiting for the upper layers to close.
>
> So at first glance we indeed addressed this case in 2018 (1.9-dev)
> with this commit:
>
> f210191dc ("BUG/MEDIUM: h2: don't accept new streams if conn_streams are
> still in excess")
>
> It was incomplete by then an later refined, but the idea is there.
> But I'll try to stress that area again to see.
So I was bored by wasting my time trying to harm the process from scripts,
I finally wrote a small program that does the same but much faster. For
now it's boring as well. In short:
- the concurrency issue was addressed 5 years ago with the commit
above so all maintained version are immune to this. The principle
is that each H2 connection knows both the number of protocol-level
streams attached to them but also application-level streams, and
it's that one that enforces the limit, preventing from processing
more requests until the number of active streams is within the limit
again. In the worst case (i.e. if the attacker downloads and its
window updates cannot enter anymore), the streams will simply time
out then the connection, like on a single non-multiplexed connection
so nothing new here. It also means that no more than the configured
limit of streams per connection will reach the hosted application
at once.
- the risk of CPU usage that was also mentioned is not much relevant
either. These aborted requests actually cost less CPU than completed
ones, and on my laptop I found that I would reach up to 100-150k req/s
per core (depending on CPU thermal throttling) which is perfectly
within what we normally observe with a standard h2load injection.
With less streams I could even reach 1 million requests per second
total, because they were aborted before being turned into a regular
stream, so the load was essentially between haproxy and the client.
So at this point I'm still failing to find any case where this attack
hurts haproxy more than any of the benchmarks we're routinely inflicting
it, given that it acts exactly like a client configured with a short
timeout (e.g. if you configure haproxy with "timeout server 1" and
have an h2 server, you will basically get the same traffic pattern).
If you want to block some annoying testers who would fill your logs
in the next few days, I checked that the following works fine here
(set the limit to the max number of requests per 10 seconds you want
to accept, e.g. 1000 below, or 100/s to keep a large margin):
tcp-request connection track-sc0 src
http-request reject if { sc0_http_req_rate gt 1000 }
stick-table type ip size 1m store http_req_rate(10s)
It's even possible to play with "set-log-level silent" to avoid logging
them, and you may even block new connections at the TCP layer. But
for now if you site requires any of this, I can't see how it has not
experienced weekly outages from standard attacks. Note that when I'm
saying that it's not because your server can process 2 million req/s
that you have to use it at that speed on a single machine, it's exactly
to keep that type of comfortable margin.
I tested the client on an EPYC 74F3 server (24 cores at 3 GHz, the one
that I demoed at haproxyconf) and it handles 800k req/s at saturation,
spending most of its time in the weighted roundrobin lock (it reaches
850k with random so pretty much standard for this machine), and perf
top looks good:
Samples: 9M of event 'cycles', 4000 Hz, Event count (approx.): 864711451063
lost: 0/0 drop: 0/0
Overhead Shared Object Symbol
4.11% haproxy [.] process_stream
3.29% haproxy [.] srv_add_to_idle_list
3.02% haproxy [.
Re: Options for mitigating CVE-2023-44487 with HAProxy
On Tue, Oct 10, 2023 at 03:49:21PM +0200, Willy Tarreau wrote:
> > Seems like a clever update to the "good old" h2 multiplexing abuse vectors:
> > 1. client opens a lot of H2 streams on a connection
> > 2. Spams some requests
> > 3. immediately sends h2 RST frames for all of them
> > 4. Go back to 1. repeat.
>
> Yes, precisely one of those I tested last week among the usual approchaes
> consisting in creating tons of streams while staying within the protocol's
> validity limits. The only thing it did was to detect the pool issue I
> mentioned in the dev7 announce.
>
> > The idea being to cause resource exhaustion on the server/proxy at least
> > when it allocates stream related buffers etc, and the underlying server too
> > since it likely sees the requests before they get cancelled.
> >
> > Looking at HAProxy I'd like to know if someone's aware of a decent
> > mitigation option?
>
> We first need to check if at all we're affected, since we keep a count
> of the attached streams for precisely this case and we refrain from
> processing headers frames when we have too many streams, so normally
> the mux will pause waiting for the upper layers to close.
So at first glance we indeed addressed this case in 2018 (1.9-dev)
with this commit:
f210191dc ("BUG/MEDIUM: h2: don't accept new streams if conn_streams are
still in excess")
It was incomplete by then an later refined, but the idea is there.
But I'll try to stress that area again to see.
Willy
Re: Options for mitigating CVE-2023-44487 with HAProxy
Hi Tristan, On Tue, Oct 10, 2023 at 12:56:12PM +, Tristan wrote: > Hi all, > > This just got disclosed: > - > https://cloud.google.com/blog/products/identity-security/google-cloud-mitigated-largest-ddos-attack-peaking-above-398-million-rps/ > - > https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack Yes, finally it's public. > Seems like a clever update to the "good old" h2 multiplexing abuse vectors: > 1. client opens a lot of H2 streams on a connection > 2. Spams some requests > 3. immediately sends h2 RST frames for all of them > 4. Go back to 1. repeat. Yes, precisely one of those I tested last week among the usual approchaes consisting in creating tons of streams while staying within the protocol's validity limits. The only thing it did was to detect the pool issue I mentioned in the dev7 announce. > The idea being to cause resource exhaustion on the server/proxy at least > when it allocates stream related buffers etc, and the underlying server too > since it likely sees the requests before they get cancelled. > > Looking at HAProxy I'd like to know if someone's aware of a decent > mitigation option? We first need to check if at all we're affected, since we keep a count of the attached streams for precisely this case and we refrain from processing headers frames when we have too many streams, so normally the mux will pause waiting for the upper layers to close. > My current first guess was to adapt F5's recommendations here > https://my.f5.com/manage/s/article/K000137106, ie set > tune.h2.fe.max-concurrent-streams 10 I *tend* to think that just like for H1 with connect+GET+shut(), "option abortonclose" should be sufficient. But I'll run more tests. > But I'm not sure if there's a way to apply the suggested measure (by Google > this time) of catching misbehaving clients (opening more streams that > advertised, or sending a lot of reset frames). It's possible that storing http_fail_rate in a stick-table could be sufficient to flag them as abusing. I don't know yet. > Finally I'll note that I don't currently have access to a good way to test > how much it affects HAProxy. And maybe it doesn't due to some peculiar > implementation detail. But that would be good to know. I'll run some tests on this, I can forge sequences of frames like this. Thanks, Willy
