Sorry, for my long absence. Thank you, Lucas, for perfectly describing and
digging into the issue. I'll be here if there is any further assistance
required.
Did I get it right, according to the spec, the "Connection"-header is forbidden
("MUST NOT"), still, firefox does send it? This leads to the described issue.
Just checked it on https://http2.golang.org/.
Firefox sends "Connection: keep-alive" while Chrome does not.
>> I'd rather not fall into such idiocies, you see.
Thanks 😊 whereby, I'd rather prefer such idiocies instead installing plugins
without asking users (well, that's another topic, I guess..
https://www.theverge.com/2017/12/16/16784628/mozilla-mr-robot-arg-plugin-firefox-looking-glass
)
-----Ursprüngliche Nachricht-----
Von: Lucas Rolff [mailto:[email protected]]
Gesendet: Donnerstag, 28. Dezember 2017 11:27
An: Willy Tarreau <[email protected]>
Cc: [email protected]
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum
> It's normal then, as it's mandated by the HTTP/2 spec to reject
> requests containing any connection-specific header fields
In that case, haproxy should be consistent in it’s way of handling clients
sending connection-specific headers:
$ curl 'https://dashboard.domain.com/js/app.js?v=1' -H 'User-Agent: Mozilla/5.0
(Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0'
--compressed -H 'Connection: keep-alive' -o /dev/null -vvv
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying 178.63.183.40...
* TCP_NODELAY set
* Connected to dashboard.domain.com (178.63.183.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [93 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3000 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=Domain Control Validated; CN=*.domain.com
* start date: Jan 3 11:17:55 2017 GMT
* expire date: Jan 4 11:17:55 2018 GMT
* subjectAltName: host "dashboard.domain.com" matched cert's "*.domain.com"
* issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
* SSL certificate verify ok.
> GET /js/app.js?v=1 HTTP/1.1
> Host: dashboard.domain.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0)
> Gecko/20100101 Firefox/57.0
> Connection: keep-alive
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.5
< Date: Thu, 28 Dec 2017 10:11:34 GMT
< Content-Type: application/javascript; charset=utf-8 < Last-Modified: Sun, 25
Jun 2017 17:17:05 GMT < Transfer-Encoding: chunked < Vary: Accept-Encoding <
ETag: W/"594ff011-7b7"
< Content-Encoding: gzip
<
{ [695 bytes data]
100 683 0 683 0 0 3936 0 --:--:-- --:--:-- --:--:-- 3925
* Connection #0 to host dashboard.domain.com left intact
Making a GET request (and sending Connection: keep-alive) continues to work,
and haproxy doesn’t handle it as malformed according to the specification.
So not sure if POST/PUT is handled differently within haproxy when it comes to
the connection header compared to GET requests.
I agree that if the specification says it cannot be done, it shouldn’t, but
then at least it should be consistent.
Fyi, in that case – nginx isn’t compliant with the http2 specification.
I’m creating a bug report with Mozilla to see if they can change the behaviour
of their browser to not send a Connection header at all, maybe this will
resolve the issue.
> I still have no idea what this "quantum" is by the way ;-)
It’s the 2017 version of “We released a better version of Firefox that is
faster”.. Let’s call it Firefox Quantum.
Haproxy 2.0 could be named “haproxy Quantum 1.5” just for the giggles.
Best Regards,
Lucas R
On 28/12/2017, 10.27, "Willy Tarreau" <[email protected]> wrote:
Hi Lucas,
On Thu, Dec 28, 2017 at 08:38:52AM +0000, Lucas Rolff wrote:
> It worked as it should, so I started adding more and more headers, until I
> hit the culprit: -H "Connection: keep-alive" or -H "Connection: close" (or
> even "Connection: test")
(...)
It's normal then, as it's mandated by the HTTP/2 spec to reject requests
containing any connection-specific header fields (Connection being the
first one) :
8.1.2.2. Connection-Specific Header Fields
HTTP/2 does not use the Connection header field to indicate
connection-specific header fields; in this protocol, connection-
specific metadata is conveyed by other means. An endpoint MUST NOT
generate an HTTP/2 message containing connection-specific header
fields; any message containing connection-specific header fields MUST
be treated as malformed (Section 8.1.2.6).
> I tried to replicate the issue in haproxy version 1.8.1, 1.8.2 and latest
> commit from master - all with the same result, I also tried playing around
> with the options of forceclose, http-server-close etc on both the frontend
> and backend in haproxy, none of them seem to "fix" the issue.
That's normal, you don't even reach this step, as it dies while
decompressing
the request (right after hpack decoding just before conversion from H2 to
H1).
> However in 1.8.2 I have 100% chance of replicating it using post requests
in
> Firefox and nghttp, where in 1.8.1 the issue in the majority of the time
> works in Firefox and only have the few percentage failure rate.
There were so many state issues with 1.8.1 that it's not much surprizing,
it's
possible that some of them would fail differently.
> I haven't been able to replicate the issue in other than Firefox and
nghttp -
Clearly this means that this has to be reported to the Firefox team, as it's
expected to break basically everywhere (or to help detect non-compliant
servers). I still have no idea what this "quantum" is by the way ;-)
> Also, sorry for the lengthy email
Quite the opposite, it was extremely helpful in spotting the problem's
origin.
Thanks!
Willy
