> On 28 Oct 2019, at 11:51, Christopher Faulet <cfau...@haproxy.com> wrote: > > Le 26/10/2019 à 18:10, Ing. Andrea Vettori a écrit : >> Hello, >> I'm using haproxy 2.0.8 and ssl termination with h2 and http1.1 protocols. >> Since today we always used http1.1 on the backends. >> I’ve tried to use http2 on the development backend but I get truncated >> response (not always but very often). >> Trying to connect from the server running haproxy to the backend server >> using curl with http2 I never get a truncated response. >> Client-side I tried with two different browsers. >> Any hint on what can cause this ? >> Thank you! > > Hi, > > If it is on a development platform, is there any way to have a full network > capture ?
Thanks for your help. The server where haproxy is running is a production server while this specific front/backend is a development web site. I can arrange a network capture if needed. > BTW, is there anything strange in your logs ? No but I really don’t know how to configure haproxy logging (at the moment it just logs when background server goes down). I’ll try to read the documentation on logging later today. > > Finally, nghttp (https://nghttp2.org) is also a good tool to debug HTTP/2. > You may try it to know why the response is truncated. Thanks, I see that curl uses that library. Here are the test I did today as per your suggestions, with the results. All connections are done from a client on a different network (so like a real user). Note that the client is on OSX while haproxy and backend are on linux. This is the haproxy configuration I used for the tests frontend dev bind TESTIP:80 bind TESTIP:82 proto h2 bind TESTIP:443 ssl crt /etc/haproxy/ssl/wildcard1.pem crt /etc/haproxy/ssl/wildcard2.pem crt /etc/haproxy/ssl/wildcard2.pem alpn h2,http/1.1 acl local_hostname hdr(host) -m beg 10. http-request deny if local_hostname use_backend dev-ssl-servers if { ssl_fc } default_backend dev-servers backend dev-ssl-servers server webdev 10.2.2.50:8083 maxconn 750 proto h2 backend dev-servers server webdev 10.2.2.50:8080 maxconn 750 proto h2 This is an http2 connection with tls (third bind on the config file above) macbookpro:~ andreavettori$ curl --http2 -v https://g.testhost * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=IT; ST=Treviso; L=Cessalto; O=B2BIres s.r.l.; CN=*.testhost * start date: Oct 4 00:00:00 2018 GMT * expire date: Dec 2 12:00:00 2020 GMT * subjectAltName: host "g.testhost" matched cert's "*.testhost"testhost * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Thawte RSA CA 2018 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fe206005400) > GET / HTTP/2 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! < HTTP/2 200 < set-cookie: JSESSIONID=722E5F380244B0C9A9577CABFB103D7C; Path=/; Secure; HttpOnly < set-cookie: agentid=7RLX0GOL8GJC04N0P8K4EB5N77VTULZR7QJT3DD3; Max-Age=2147483647; Expires=Sat, 15-Nov-2087 16:45:45 GMT; Path=/ < set-cookie: mtt_id=7RLX0GOL8GJC04N0P8K4EB5N77VTULZR7QJT3DD3|; Max-Age=2147483647; Expires=Sat, 15-Nov-2087 16:45:45 GMT; Domain=testhost; Path=/ < cache-control: no-store, must-revalidate, max-age=0 < pragma: no-cache < expires: Sat, 26 Jul 1997 05:00:00 GMT < content-type: text/html;charset=UTF-8 < content-language: en-US < date: Mon, 28 Oct 2019 13:31:38 GMT < [partial content returned] * Connection #0 to host g.testhost left intact * Closing connection 0 This is a http1.1 client with tls (third bind on the config file above) macbookpro:~ andreavettori$ curl --http1.1 -v https://g.testhost * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 443 (#0) * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=IT; ST=Treviso; L=Cessalto; O=B2BIres s.r.l.; CN=*.testhost * start date: Oct 4 00:00:00 2018 GMT * expire date: Dec 2 12:00:00 2020 GMT * subjectAltName: host "g.testhost" matched cert's "*.testhost" * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Thawte RSA CA 2018 * SSL certificate verify ok. > GET / HTTP/1.1 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 200 < set-cookie: JSESSIONID=2C0205C76ED9FC1904126BF13DB2AB5D; Path=/; Secure; HttpOnly < set-cookie: agentid=86FMNZPQV9UYOS6CNRPGI3LEPLSBE9ZOMS1IOL15; Max-Age=2147483647; Expires=Sat, 15-Nov-2087 17:31:08 GMT; Path=/ < set-cookie: mtt_id=86FMNZPQV9UYOS6CNRPGI3LEPLSBE9ZOMS1IOL15|; Max-Age=2147483647; Expires=Sat, 15-Nov-2087 17:31:08 GMT; Domain=testhost; Path=/ < cache-control: no-store, must-revalidate, max-age=0 < pragma: no-cache < expires: Sat, 26 Jul 1997 05:00:00 GMT < content-type: text/html;charset=UTF-8 < content-language: en-US < date: Mon, 28 Oct 2019 14:17:01 GMT < transfer-encoding: chunked < [partial content returned] * TLSv1.2 (IN), TLS alert, close notify (256): * transfer closed with outstanding read data remaining * Closing connection 0 * TLSv1.2 (OUT), TLS alert, close notify (256): curl: (18) transfer closed with outstanding read data remaining Trying to connect with http1.1 and no tls I get the following (first bind on the configuration above). macbookpro:~ andreavettori$ curl --http1.1 -v http://g.testhost * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 80 (#0) > GET / HTTP/1.1 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > * Empty reply from server * Connection #0 to host g.testhost left intact curl: (52) Empty reply from server * Closing connection 0 Same response when optionally using http2 and no tls (first bind on the configuration above). macbookpro:~ andreavettori$ curl --http2 -v http://g.testhost * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 80 (#0) > GET / HTTP/1.1 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > Connection: Upgrade, HTTP2-Settings > Upgrade: h2c > HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA > * Empty reply from server * Connection #0 to host g.testhost left intact curl: (52) Empty reply from server * Closing connection 0 When trying to force the use of http2 and no tls (first bind on the configuration above) it enters a loop where it continually tries to create connections (the output continues but I just show the first three connections). macbookpro:~ andreavettori$ curl --http2-prior-knowledge -v http://g.testhost * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 80 (#0) * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fd18c005400) > GET / HTTP/2 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! * REFUSED_STREAM, retrying a fresh connect * Connection died, retrying a fresh connect * Closing connection 0 * Issue another request to this URL: 'http://g.testhost/' * Hostname g.testhost was found in DNS cache * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 80 (#1) * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fd18c005400) > GET / HTTP/2 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! * REFUSED_STREAM, retrying a fresh connect * Connection died, retrying a fresh connect * Closing connection 1 * Issue another request to this URL: 'http://g.testhost/' * Hostname g.testhost was found in DNS cache * Trying TESTIP... * TCP_NODELAY set * Connected to g.testhost (TESTIP) port 80 (#2) * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fd18c005400) > GET / HTTP/2 > Host: g.testhost > User-Agent: curl/7.64.1 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 100)! * REFUSED_STREAM, retrying a fresh connect * Connection died, retrying a fresh connect * Closing connection 2 Finally trying on the third bind (the one where h2 protocol is specified and no ssl option) gives the same results as the last two tests. I’m not able to fully understand these results and I hope they have some meaning to others. Also note that when doing the plaintext call from the haproxy server to the backend server, either http2 or http1.1 they work with no errors and return full content. I’ll ty to do more tests later. Thank you
