Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
Hi James, On Wed, Aug 18, 2021 at 04:53:09PM -0700, James Brown wrote: > Are there CVE numbers coming for these vulnerabilities? Yes, for what it's worth, Robert Frohl from SuSE got 3 assigned to this: - CVE-2021-39240: -> Domain parts in ":scheme" and ":path" - CVE-2021-39241: -> Spaces in the ":method" field - CVE-2021-39242: -> Mismatch between ":authority" and "Host" Regards, Willy
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
On Thursday, 19 August 2021, James Brown wrote: > Are there CVE numbers coming for these vulnerabilities? > > CVE-2021-39240: -> 2) Domain parts in ":scheme" and ":path" CVE-2021-39241: -> 1) Spaces in the ":method" field CVE-2021-39242: -> 3) Mismatch between ":authority" and "Host" Lukas
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
Are there CVE numbers coming for these vulnerabilities?
On Tue, Aug 17, 2021 at 8:14 AM Willy Tarreau wrote:
> Hi everyone,
>
> HAProxy is affected by 4 vulnerabilities in its HTTP/2 implementation in
> recent versions (starting with 2.0). Three of them are considered as having
> a moderate impact as they only affect the interpretation of the authority
> (Host header field) in H2->H2 communications in versions 2.2 and above.
> One only affects a risk of misinterpretation from lenient HTTP/1 backend
> servers, and affects version 2.0 and above, though at the time of writing
> we're not aware of any such vulnerable server among the mainstream ones
> that are commonly found behind HAProxy (Apache, NGINX, Varnish, etc).
>
> Background: on Aug 05 a research article was published about flaws
> affecting some HTTP/2 to HTTP/1 proxies or gateways:
>
> https://portswigger.net/research/http2
>
> A first analysis of the article compared to some selected pieces of code
> concluded that haproxy was safe. This was actually wrong as only older
> versions are safe (2.0 in "legacy" mode and 1.8). Tim Düsterhus conducted
> some additional tests and found some problems, which after digging for a
> few days, revealed to be significantly more embarrassing. In practice,
> version 2.0 in the default "HTX" mode and all later versions are affected
> to some degrees.
>
>
> 1) Spaces in the ":method" field
>
> The first problem is based on the ":method" field. By passing a space in
> the method, it is possible to build an invalid HTTP/1 request on the
> backend side, which some lenient servers might possibly interpret as valid,
> resulting in a different request between the one seen by haproxy and by the
> server. This might be abused to circumvent some switching rules for
> example,
> and get a request to be routed to a wrong server. Example:
>
>H2 request
> :method: "GET /admin? HTTP/1.1"
> :path: "/static/images"
>
> HAProxy would route all "/static" requests to the static server farm,
> but once the request is reassembled it would become this:
>
>GET /admin? HTTP/1.1 /static/images HTTP/1.1
>
> This is not valid but if a server fails to properly validate this input,
> it might be fooled into thinking this is a request for /admin.
>
> Please note that HTTP/2 backend servers are not affect as the request is
> sent as a new ":method" field there. Additionally, dangerous characters
> like CR, LF or NUL are always blocked on input so is is not possible to
> perform a request smuggling attack, and the risks are limited to HTTP/1
> servers which fail to properly parse the request line (note that all
> major server implementations are safe against this).
>
> A workaround for this issue for those having to rely on possibly unsafe
> servers is to reject invalid characters in the method by placing such a
> filtering rule on the request path either in the frontend or the backend:
>
>http-request reject if { method -m reg [^A-Z0-9] }
>
> A second workaround that may only be used on version 2.0 consists in
> disabling the HTX internal representation in the affected backends and
> the frontends that route to them:
>
>no option http-use-htx
>
> This will have for effect to transform the HTTP/2 requests to HTTP/1 that
> will then be submitted to the internal HTTP/1 parser which will reject
> the poorly formatted request. This older representation called "legacy"
> is not available any more in version 2.1 and above, and is not compatible
> with HTTP/2 nor FastCGI backend servers.
>
> This issue affects all versions from 2.0 and above, in HTX mode, with
> HTTP/1 on the server side.
>
>
> 2) Domain parts in ":scheme" and ":path"
>
> The ":scheme" HTTP/2 header field contains the scheme that prefixes the
> request URI, typically "http" or "https". ":path" contains the part that
> is local to the target host, and that usually starts with the "/". By
> appending a part of a request to ":scheme" or by prepending a part of a
> domain name to ":path", it is possible to make haproxy and a backend
> server see a different authority or URL prefix. Please note that this
> only affects HTTP/2 servers on versions 2.2 and above. These versions
> are indeed capable of passing an absolute URI from end to end, while
> earlier versions were limited to origin URIs. In addition, HTTP/2
> requests are always forwarded in origin form to HTTP/1 backend servers
> (i.e. they do not have a scheme nor authority parts). As such HTTP/1
> servers are safe and only HTTP/2 servers are exposed.
>
> What happens is that on the entry point, the :scheme, :authority and :path
> fields are concatenated to rebuild a full URI that is then passed along the
> chain, but the Host header is set from :authority before this concatenation
> is performed. As such, the Host header field used internally may not always
> match the authority part of the recomposed URI. Examples:
>
>H2 request
> :method: "GET"
> :scheme:
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
On Tue, Aug 17, 2021 at 06:57:28PM +0200, Tim Düsterhus wrote: > Hi Willy, Everyone, > > On 8/17/21 5:13 PM, Willy Tarreau wrote: > > 2) Domain parts in ":scheme" and ":path" > > > > [...] As such HTTP/1 servers are safe and only HTTP/2 servers are exposed. > > I'd like to clarify that the above statement is not true. The issue also > affects H2->HAProxy->H1 connections. It allows to forward a different 'host' > header than the one HAProxy sees to the backend. So to be more precise, based on the output of your test, both will see the same Host header field, however the server will receive a different authority than the Host header field. While this is not a valid request we know that some servers are more willing to accept that than the poorly formatted requests. However those which do most often only use the Host (which is why they don't check the authority). I'm not saying this to try to dismiss the problem, it's in order to help admins analyze strange logs that they may encounter before upgrading or deploying workarounds. > The 'http-request set-uri %[url]' workaround mentioned at the bottom of > Willy's email also fixes the issue for HTTP/1 backends. > > In any case I recommend to upgrade as soon as possible. That way you don't > have to think whether your setup requires a workaround or not. I agree on both points! Thanks for reporting that one. Willy
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
On Tue, Aug 17, 2021 at 05:56:15PM +0200, Tim Düsterhus wrote: > Vincent, > > On 8/17/21 5:49 PM, Vincent Bernat wrote: > > For users of haproxy.debian.net or Launchpad PPA, the vulnerabilities > > are fixed by patching the previous versions. Launchpad PPA builders are > > still running but it should be available in the next hour. I will upload > > the new versions later this week. > > As a consumer of your packages: A big thank you for your work. > > After I noticed Willy's announcement I checked whether any new 2.4.x > packages were available and indeed they were already there, saving me the > work from compiling a custom version / adjusting the config. I second that! And I'm well aware of the amount of work Vincent had to do behind the curtains to deal with this since the end of last week to provide us with all this just in time. Thanks a lot Vincent! Willy
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
Hi Willy, Everyone, On 8/17/21 5:13 PM, Willy Tarreau wrote: 2) Domain parts in ":scheme" and ":path" [...] As such HTTP/1 servers are safe and only HTTP/2 servers are exposed. I'd like to clarify that the above statement is not true. The issue also affects H2->HAProxy->H1 connections. It allows to forward a different 'host' header than the one HAProxy sees to the backend. The 'http-request set-uri %[url]' workaround mentioned at the bottom of Willy's email also fixes the issue for HTTP/1 backends. In any case I recommend to upgrade as soon as possible. That way you don't have to think whether your setup requires a workaround or not. Best regards Tim Düsterhus
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
Vincent, On 8/17/21 5:49 PM, Vincent Bernat wrote: For users of haproxy.debian.net or Launchpad PPA, the vulnerabilities are fixed by patching the previous versions. Launchpad PPA builders are still running but it should be available in the next hour. I will upload the new versions later this week. As a consumer of your packages: A big thank you for your work. After I noticed Willy's announcement I checked whether any new 2.4.x packages were available and indeed they were already there, saving me the work from compiling a custom version / adjusting the config. Best regards Tim Düsterhus
Re: [ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
❦ 17 August 2021 17:13 +02, Willy Tarreau: > HAProxy is affected by 4 vulnerabilities in its HTTP/2 implementation in > recent versions (starting with 2.0). Three of them are considered as having > a moderate impact as they only affect the interpretation of the authority > (Host header field) in H2->H2 communications in versions 2.2 and above. > One only affects a risk of misinterpretation from lenient HTTP/1 backend > servers, and affects version 2.0 and above, though at the time of writing > we're not aware of any such vulnerable server among the mainstream ones > that are commonly found behind HAProxy (Apache, NGINX, Varnish, etc). For users of haproxy.debian.net or Launchpad PPA, the vulnerabilities are fixed by patching the previous versions. Launchpad PPA builders are still running but it should be available in the next hour. I will upload the new versions later this week. Check the changelog (in /usr/share/doc/haproxy/changelog.Debian.gz) if you want to know if you are running a fixed version. The list of fixed versions are: haproxy_2.4.2-2~bpo10+1 haproxy_2.4.2-2~bpo11+1 haproxy_2.4.2-2ppa1~bionic haproxy_2.4.2-2ppa1~focal haproxy_2.2.9-2+deb11u1 (should be available from debian-security soon) haproxy_2.3.12-2~bpo10+1 haproxy_2.3.12-2ppa1~bionic haproxy_2.3.12-2ppa1~focal haproxy_2.2.15-3~bpo9+1 haproxy_2.2.15-3~bpo10+1 haproxy_2.2.15-3ppa1~bionic haproxy_2.2.15-3ppa1~focal haproxy_2.0.23-3~bpo9+1 haproxy_2.0.23-3~bpo10+1 haproxy_2.0.23-3ppa1~xenial haproxy_2.0.23-3ppa1~bionic haproxy_2.0.23-3ppa1~focal -- Make input easy to proofread. - The Elements of Programming Style (Kernighan & Plauger)
[ANNOUNCE] HTTP/2 vulnerabilities from 2.0 to 2.5-dev
Hi everyone,
HAProxy is affected by 4 vulnerabilities in its HTTP/2 implementation in
recent versions (starting with 2.0). Three of them are considered as having
a moderate impact as they only affect the interpretation of the authority
(Host header field) in H2->H2 communications in versions 2.2 and above.
One only affects a risk of misinterpretation from lenient HTTP/1 backend
servers, and affects version 2.0 and above, though at the time of writing
we're not aware of any such vulnerable server among the mainstream ones
that are commonly found behind HAProxy (Apache, NGINX, Varnish, etc).
Background: on Aug 05 a research article was published about flaws
affecting some HTTP/2 to HTTP/1 proxies or gateways:
https://portswigger.net/research/http2
A first analysis of the article compared to some selected pieces of code
concluded that haproxy was safe. This was actually wrong as only older
versions are safe (2.0 in "legacy" mode and 1.8). Tim Düsterhus conducted
some additional tests and found some problems, which after digging for a
few days, revealed to be significantly more embarrassing. In practice,
version 2.0 in the default "HTX" mode and all later versions are affected
to some degrees.
1) Spaces in the ":method" field
The first problem is based on the ":method" field. By passing a space in
the method, it is possible to build an invalid HTTP/1 request on the
backend side, which some lenient servers might possibly interpret as valid,
resulting in a different request between the one seen by haproxy and by the
server. This might be abused to circumvent some switching rules for example,
and get a request to be routed to a wrong server. Example:
H2 request
:method: "GET /admin? HTTP/1.1"
:path: "/static/images"
HAProxy would route all "/static" requests to the static server farm,
but once the request is reassembled it would become this:
GET /admin? HTTP/1.1 /static/images HTTP/1.1
This is not valid but if a server fails to properly validate this input,
it might be fooled into thinking this is a request for /admin.
Please note that HTTP/2 backend servers are not affect as the request is
sent as a new ":method" field there. Additionally, dangerous characters
like CR, LF or NUL are always blocked on input so is is not possible to
perform a request smuggling attack, and the risks are limited to HTTP/1
servers which fail to properly parse the request line (note that all
major server implementations are safe against this).
A workaround for this issue for those having to rely on possibly unsafe
servers is to reject invalid characters in the method by placing such a
filtering rule on the request path either in the frontend or the backend:
http-request reject if { method -m reg [^A-Z0-9] }
A second workaround that may only be used on version 2.0 consists in
disabling the HTX internal representation in the affected backends and
the frontends that route to them:
no option http-use-htx
This will have for effect to transform the HTTP/2 requests to HTTP/1 that
will then be submitted to the internal HTTP/1 parser which will reject
the poorly formatted request. This older representation called "legacy"
is not available any more in version 2.1 and above, and is not compatible
with HTTP/2 nor FastCGI backend servers.
This issue affects all versions from 2.0 and above, in HTX mode, with
HTTP/1 on the server side.
2) Domain parts in ":scheme" and ":path"
The ":scheme" HTTP/2 header field contains the scheme that prefixes the
request URI, typically "http" or "https". ":path" contains the part that
is local to the target host, and that usually starts with the "/". By
appending a part of a request to ":scheme" or by prepending a part of a
domain name to ":path", it is possible to make haproxy and a backend
server see a different authority or URL prefix. Please note that this
only affects HTTP/2 servers on versions 2.2 and above. These versions
are indeed capable of passing an absolute URI from end to end, while
earlier versions were limited to origin URIs. In addition, HTTP/2
requests are always forwarded in origin form to HTTP/1 backend servers
(i.e. they do not have a scheme nor authority parts). As such HTTP/1
servers are safe and only HTTP/2 servers are exposed.
What happens is that on the entry point, the :scheme, :authority and :path
fields are concatenated to rebuild a full URI that is then passed along the
chain, but the Host header is set from :authority before this concatenation
is performed. As such, the Host header field used internally may not always
match the authority part of the recomposed URI. Examples:
H2 request
:method: "GET"
:scheme: "http://localhost/?orig=;
:authority "example.org"
:path: "/"
or:
H2 request
:method: "GET"
:scheme: "http"
:authority "example.org"
:path: ".local/"
An internal Host header will be build with "example.org" then the complete
URI will become
