Re: http2 smuggling
On Fri, Sep 11, 2020 at 09:56:21AM +0200, Tim Düsterhus wrote:
> Willy,
>
> Am 11.09.20 um 09:42 schrieb Willy Tarreau:
> > On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
> >> According to the article performing a h2c upgrade via TLS is not valid
> >> according to the spec. HAProxy implements the H2 spec.
> >
> > "according to the article" :-) There's no such mention in the spec
> > itself from what I remember, it's just that it's usually pointless,
> > but there may be a lot of situations where it's considered better to
> > forward an upgradable connection over TLS to the next intermediary
> > because the intermediary network is not safe.
>
> I might misunderstand it, but I'd say that RFC 7540#3.3 specifically
> disallows h2c for TLS:
>
> >HTTP/2 over TLS uses the "h2" protocol identifier. The "h2c"
> >protocol identifier MUST NOT be sent by a client or selected by a
> >server; the "h2c" protocol identifier describes a protocol that does
> >not use TLS.
Ah OK, got it, this one is about the protocol indicated in ALPN, as
reflected by the preceeding sentence: "A client that makes a request
to an "https" URI uses TLS with the application-layer protocol
negotiation (ALPN) extension".
When this was designed, there were long discussions about whether h2
alone or both h2/h2c had to be registered as protocols, and about
whether h2c was allowed in ALPN and in this case what to do about it.
It was then concluded that the easiest solution was to state that h2c
is the cleartext upgrade version while h2 is the raw one over TLS.
> >> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
> >> TLS? I think it should. Basically the following rule should be applied
> >> automatically to my understanding.
> >>
> >> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
> >
> > No I disagree. Let's say you have an h2c client on datacenter 1 and
> > an h2c server on datacenter 2. This rule would prevent you from using
> > the local haproxy to secure the connection, while providing zero benefit.
>
> If I just want to secure the connection I would use 'mode tcp' where
> HAProxy is a dumb pipe and this rule would not apply.
You can't always. Let's say haproxy is your local side car and your
application sends everything to it. You'll route requests to various
places according to their Host and URI. Why is it that for a particular
reason one type of request is considered special enough to deserve being
broken ?
> > By the way, it's fun to see that a discussion started a few days ago
> > regarding the uselessness of h2c and its removal from the next H2 spec
> > because "nobody implemented it yet" :-) And actually the guy had to
> > implement its own server to find a complying one.
>
> I believe nginx can do h2c.
Maybe but then probably not by default, given that the guy used it in
his test to forward the Upgrade request. If it would support it by
default, it would have considered that request for itself instead. I'm
pretty sure that nginx does support cleartext h2 (without TLS) however.
By the way if you're interested in the todo list that starts to be
collected for an update of the H2 spec, it's here at the moment:
https://github.com/martinthomson/http2-spec/issues/
Cheers,
Willy
Re: http2 smuggling
Willy,
Am 11.09.20 um 09:42 schrieb Willy Tarreau:
> On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
>> According to the article performing a h2c upgrade via TLS is not valid
>> according to the spec. HAProxy implements the H2 spec.
>
> "according to the article" :-) There's no such mention in the spec
> itself from what I remember, it's just that it's usually pointless,
> but there may be a lot of situations where it's considered better to
> forward an upgradable connection over TLS to the next intermediary
> because the intermediary network is not safe.
I might misunderstand it, but I'd say that RFC 7540#3.3 specifically
disallows h2c for TLS:
>HTTP/2 over TLS uses the "h2" protocol identifier. The "h2c"
>protocol identifier MUST NOT be sent by a client or selected by a
>server; the "h2c" protocol identifier describes a protocol that does
>not use TLS.
-
>> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
>> TLS? I think it should. Basically the following rule should be applied
>> automatically to my understanding.
>>
>> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
>
> No I disagree. Let's say you have an h2c client on datacenter 1 and
> an h2c server on datacenter 2. This rule would prevent you from using
> the local haproxy to secure the connection, while providing zero benefit.
If I just want to secure the connection I would use 'mode tcp' where
HAProxy is a dumb pipe and this rule would not apply.
> By the way, it's fun to see that a discussion started a few days ago
> regarding the uselessness of h2c and its removal from the next H2 spec
> because "nobody implemented it yet" :-) And actually the guy had to
> implement its own server to find a complying one.
I believe nginx can do h2c.
Best regards
Tim Düsterhus
Re: http2 smuggling
On Fri, Sep 11, 2020 at 09:02:57AM +0200, Tim Düsterhus wrote:
> According to the article performing a h2c upgrade via TLS is not valid
> according to the spec. HAProxy implements the H2 spec.
"according to the article" :-) There's no such mention in the spec
itself from what I remember, it's just that it's usually pointless,
but there may be a lot of situations where it's considered better to
forward an upgradable connection over TLS to the next intermediary
because the intermediary network is not safe.
> Question 1: Should HAProxy reject requests that set Upgrade: h2c over
> TLS? I think it should. Basically the following rule should be applied
> automatically to my understanding.
>
> http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
No I disagree. Let's say you have an h2c client on datacenter 1 and
an h2c server on datacenter 2. This rule would prevent you from using
the local haproxy to secure the connection, while providing zero benefit.
By the way, it's fun to see that a discussion started a few days ago
regarding the uselessness of h2c and its removal from the next H2 spec
because "nobody implemented it yet" :-) And actually the guy had to
implement its own server to find a complying one.
> Further the article says that the HTTP2-Settings header is a hop by hop
> header. It should not be forwarded by a proxy. According to the article
> HAProxy *does* forward it.
>
> Question 2: Should HAProxy automatically strip the HTTP2-Settings header
> when forwarding requests?
Haproxy is not a proxy but a gateway. See it as a transparent cable
capable of staying synchronized with both ends and interacting the least
possible. There's nothing wrong with repeating hop-by-hop header fields
on gateways if you don't break the transport nor semantics, it's exactly
what happens when you use a raw TLS offloader for example, which doesn't
even understand HTTP. Technically speaking we're not "keeping" the
header, we're formulating a new request that places it again since it's
compatible with both sides' capabilities. In fact haproxy produces a
new connection header with all the tokens that it did not use, since
they are by definition for the next hop. "keep-alive" and "close" are
the only two that are of interest to us and that are terminated locally.
For example, the TE header is hop-by-hop. Haproxy doesn't need it but
maintains the transport, messaging and semantics end-to-end so it does
not need to eliminate it and disrupt end-to-end connectivity as long as
it stays synchronized. Really that's not different from a pure TCP proxy
(which is how haproxy started 20 years ago by the way).
Hoping this clarifies the point,
Willy
Re: http2 smuggling
On Fri, Sep 11, 2020 at 02:52:30AM -0400, John Lauro wrote: > I could be wrong, but I think he is stating that if you have that > allowed, it can be used to get a direct connection to the backend > bypassing any routing or acls you have in the load balancer, so if you > some endpoints are blocked, or internal only, they could potentially > be accessed this way. > For example, if you have something like: > acl is_restrict path_sub /.git/ > http-request deny if is_restrict !is_safe_ip > > The acl could be bypassed by using the method to connect directly to a > backend. > > That's not to say it's a security flaw in haproxy, but a potential > misconfiguration that could allow traffic you thought was blocked by > the proxy. We're talking about an upgrade agreement between the client and the server in order to use a protocol that the LB doesn't speak. This is typically used for websocket, I remember having seen one terminal server using this as well, maybe RDP or Citrix, I don't remember. This is exactly the same as CONNECT+200: both ends agree to upgrade the HTTP connection to another protocol till it ends. It's not HTTP that applies once the tunnel is set up, so there are no filtering rules nor whatever, exactly like after an accepted CONNECT request. Willy
Re: [*EXT*] Re: http2 smuggling
Hi Ionel, On Fri, Sep 11, 2020 at 08:35:58AM +0200, Ionel GARDAIS wrote: > Hi Willy, > > Being devil's advocate : isn't the point that even if this is a documented, > standardized and intended behavior, users relying on the reverse proxy for > security/sanity checks could by tricked by this feature inadvertently ? The security checks are properly performed during the request for upgrade, the reporter even mentioned it in his article where he blocks the upgrade by removing the Upgrade header. Note that the important point is that the server *must* accept the upgrade for this to work. So there's no real accident here. It's exactly what's being used for websocket. Willy
Re: http2 smuggling
Willy,
Am 11.09.20 um 08:07 schrieb Willy Tarreau:
> On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote:
>> Should we be worried?
>>
>> https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel
>
> But this stuff is total non-sense. Basically the guy is complaining
> that the products he tested work exactly as desired, designed and
> documented!
>
> The principle of the upgrade at the gateway level precisely is to say
> "OK both the client and the server want to speak another protocol you
> agreed upon, let me retract" and let them talk over a tunnel. That's
> exactly what is needed to support WebSocket for example. The simple
> fact that he found that many proxies/gateways work like this should
> ring a bell about the intended behavior!
>
I'm very well aware about your opinion regarding access control at the
edge by now, however I have 2 questions at this point.
I've read the official write-up of the findings of the researcher:
https://labs.bishopfox.com/tech-blog/h2c-smuggling-request-smuggling-via-http/2-cleartext-h2c.
It contains quite a few more details and I recommend to take a look at
it to answer my questions.
According to the article performing a h2c upgrade via TLS is not valid
according to the spec. HAProxy implements the H2 spec.
Question 1: Should HAProxy reject requests that set Upgrade: h2c over
TLS? I think it should. Basically the following rule should be applied
automatically to my understanding.
http-request deny deny_status 400 if { req.hdr(upgrade) h2c } { ssl_fc }
Further the article says that the HTTP2-Settings header is a hop by hop
header. It should not be forwarded by a proxy. According to the article
HAProxy *does* forward it.
Question 2: Should HAProxy automatically strip the HTTP2-Settings header
when forwarding requests?
Best regards
Tim Düsterhus
Re: http2 smuggling
I could be wrong, but I think he is stating that if you have that allowed, it can be used to get a direct connection to the backend bypassing any routing or acls you have in the load balancer, so if you some endpoints are blocked, or internal only, they could potentially be accessed this way. For example, if you have something like: acl is_restrict path_sub /.git/ http-request deny if is_restrict !is_safe_ip The acl could be bypassed by using the method to connect directly to a backend. That's not to say it's a security flaw in haproxy, but a potential misconfiguration that could allow traffic you thought was blocked by the proxy. On Fri, Sep 11, 2020 at 2:07 AM Willy Tarreau wrote: > > Hi Igor, > > On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote: > > Should we be worried? > > > > https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel > > But this stuff is total non-sense. Basically the guy is complaining > that the products he tested work exactly as desired, designed and > documented! > > The principle of the upgrade at the gateway level precisely is to say > "OK both the client and the server want to speak another protocol you > agreed upon, let me retract" and let them talk over a tunnel. That's > exactly what is needed to support WebSocket for example. The simple > fact that he found that many proxies/gateways work like this should > ring a bell about the intended behavior! > > In addition there is zero smuggling here as there is no desynchronisation. > It's just a tunnel between the client and the server, both agreeing to > do so. It does *exactly* the same as if the client had connected to the > server using a CONNECT method and the server had returned 200. So there > is absolutely no attack nor whatever here, just a connection that remains > dedicated to a client and a server till the end. > > Sadly, as usual after people discover protocols during the summer, some > journalists will surely want to make noise about this to put some bread > on their table... > > Thanks for the link anyway I had a partial laugh; partial only because > it makes useless noise. > > Cheers, > Willy >
Re: [*EXT*] Re: http2 smuggling
Hi Willy, Being devil's advocate : isn't the point that even if this is a documented, standardized and intended behavior, users relying on the reverse proxy for security/sanity checks could by tricked by this feature inadvertently ? -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Willy Tarreau" À: "Igor Cicimov" Cc: "haproxy" Envoyé: Vendredi 11 Septembre 2020 08:19:12 Objet: [*EXT*] Re: http2 smuggling On Fri, Sep 11, 2020 at 08:07:02AM +0200, Willy Tarreau wrote: > Sadly, as usual after people discover protocols during the summer, some > journalists will surely want to make noise about this to put some bread > on their table... > > Thanks for the link anyway I had a partial laugh; partial only because > it makes useless noise. And sadly, this one already started to make some noise there about his recent discovery of a 20-years old standard: https://twitter.com/theBumbleSec Had he asked if we supported 101, we could even have saved him time in his HTTP discover test by pointing him to the doc: http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/configuration.txt;h=c1f6f82;hb=HEAD#l332 Probably that next year he will discover that we also support CONNECT. It's not even funny, the world is really doomed... Willy -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: http2 smuggling
On Fri, Sep 11, 2020 at 08:07:02AM +0200, Willy Tarreau wrote: > Sadly, as usual after people discover protocols during the summer, some > journalists will surely want to make noise about this to put some bread > on their table... > > Thanks for the link anyway I had a partial laugh; partial only because > it makes useless noise. And sadly, this one already started to make some noise there about his recent discovery of a 20-years old standard: https://twitter.com/theBumbleSec Had he asked if we supported 101, we could even have saved him time in his HTTP discover test by pointing him to the doc: http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/configuration.txt;h=c1f6f82;hb=HEAD#l332 Probably that next year he will discover that we also support CONNECT. It's not even funny, the world is really doomed... Willy
Re: http2 smuggling
Hi Igor, On Fri, Sep 11, 2020 at 01:55:10PM +1000, Igor Cicimov wrote: > Should we be worried? > > https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel But this stuff is total non-sense. Basically the guy is complaining that the products he tested work exactly as desired, designed and documented! The principle of the upgrade at the gateway level precisely is to say "OK both the client and the server want to speak another protocol you agreed upon, let me retract" and let them talk over a tunnel. That's exactly what is needed to support WebSocket for example. The simple fact that he found that many proxies/gateways work like this should ring a bell about the intended behavior! In addition there is zero smuggling here as there is no desynchronisation. It's just a tunnel between the client and the server, both agreeing to do so. It does *exactly* the same as if the client had connected to the server using a CONNECT method and the server had returned 200. So there is absolutely no attack nor whatever here, just a connection that remains dedicated to a client and a server till the end. Sadly, as usual after people discover protocols during the summer, some journalists will surely want to make noise about this to put some bread on their table... Thanks for the link anyway I had a partial laugh; partial only because it makes useless noise. Cheers, Willy
http2 smuggling
Should we be worried? https://portswigger.net/daily-swig/http-request-smuggling-http-2-opens-a-new-attack-tunnel IC
