RE: HAProxy 1.8.x not serving errorfiles with H2
Thank you! That fixed the issue. Might I suggest adding the below snippet into the "example" section of the errorfile documentation at [1]? The current example only shows the configuration itself but not what the file format should be - since it's this specific, the description given is just simply not enough for a casual proxy admin like myself to successfully configure it. -Original Message- From: Tim Düsterhus [mailto:t...@bastelstu.be] Sent: Tuesday, June 12, 2018 2:22 PM To: J. Casalino ; Cyril Bonté Cc: haproxy@formilux.org Subject: Re: HAProxy 1.8.x not serving errorfiles with H2 Hi Am 12.06.2018 um 17:43 schrieb J. Casalino: > Thank you so much for your response. [1] is the simplest example of our 503 > errorfile. Based on what you're saying it sounds like we need to add some > additional information into the file. Is this documented somewhere? > it is [1]: > The files are returned verbatim on the TCP socket. This allows any > trick such as redirections to another URL or site, as well as tricks > to clean cookies, force enable or disable caching, etc... The package > provides default error files returning the same contents as default errors. You need to prefix HTTP response headers. Something like this should work [2]: > HTTP/1.0 503 Service Unavailable > Cache-Control: no-cache > Connection: close > Content-Type: text/html Best regards Tim Düsterhus [1] https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-errorfile [2] http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/errorfiles/503.http;h=48fde5881e680857462c32cc305f9179af19091f;hb=HEAD
Re: HAProxy 1.8.x not serving errorfiles with H2
Hi Am 12.06.2018 um 17:43 schrieb J. Casalino: > Thank you so much for your response. [1] is the simplest example of our 503 > errorfile. Based on what you're saying it sounds like we need to add some > additional information into the file. Is this documented somewhere? > it is [1]: > The files are returned verbatim on the TCP socket. This allows any trick such > as redirections to another URL or site, as well as tricks to clean cookies, > force enable or disable caching, etc... The package provides default error > files returning the same contents as default errors. You need to prefix HTTP response headers. Something like this should work [2]: > HTTP/1.0 503 Service Unavailable > Cache-Control: no-cache > Connection: close > Content-Type: text/html Best regards Tim Düsterhus [1] https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-errorfile [2] http://git.haproxy.org/?p=haproxy.git;a=blob;f=examples/errorfiles/503.http;h=48fde5881e680857462c32cc305f9179af19091f;hb=HEAD
RE: HAProxy 1.8.x not serving errorfiles with H2
Hi Cyril, Thank you so much for your response. [1] is the simplest example of our 503 errorfile. Based on what you're saying it sounds like we need to add some additional information into the file. Is this documented somewhere? [1] HTTP 503 Author is unavailable Either the instance is temporarily unavailable, down or being restarted. If this issue persists, please file a ticket W W W WW W W '. W .-""-._ \ \.--| / "-..__) .-' | _ / \'-.__, .__.,' `''._\--' V -Original Message- From: Cyril Bonté [mailto:cyril.bo...@free.fr] Sent: Monday, June 11, 2018 4:37 PM To: J. Casalino Cc: haproxy@formilux.org Subject: Re: HAProxy 1.8.x not serving errorfiles with H2 Hi, Le 11/06/2018 à 16:39, J. Casalino a écrit : > Trying again - has anyone else seen this? > > *From:* J. Casalino [mailto:casal...@adobe.com] > *Sent:* Tuesday, June 5, 2018 12:27 PM > *To:* haproxy@formilux.org > *Subject:* HAProxy 1.8.x not serving errorfiles with H2 > > We are in the process of testing HAProxy 1.8.x with ALPN and H2 on > some of our servers. We have default 502 and 503 errorfiles defined (ex. > errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that > these errorfiles are not served to the user's browser when the error > occurs (for instance, if the backend is down, a user should get the > 503 errorfile). > > Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) > HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and > Firefox shows "The connection to was interrupted while > the page was loading." > > With debug logging turned on, I can see that HAProxy is recognizing a > 503 if the back-end server is down [2], but it doesn't seem to pass > that error through to the client browser. If the backend is up and a > 502 is generated, users do not receive the errorfile either. If we > turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed > properly (though via HTTP/0.9) If it looks like HTTP/0.9, I tend to think that your errorfile is not properly set. Such files must contain the status line, the headers and the response body. And indeed, from a quick test, if I remove the status line, I get the same error with a HTTP/2 request. Once the file is correctly defined, everything is OK. Can you provide the content of your errorfile(s) ? > This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon > Linux, using openssl-1.0.2k-12.109.amzn1.x86_64. > > Thanks in advance for any thoughts you might have - > > [1] > > Curl verbose (curl -I) output: > > * Trying ... > > * TCP_NODELAY set > > * Connected to () port 443 (#0) > > * ALPN, offering h2 > > * ALPN, offering http/1.1 > > * Cipher selection: > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH > > * successfully set certificate verify locations: > > * CAfile: /etc/ssl/cert.pem > > CApath: none > > * TLSv1.2 (OUT), TLS handshake, Client hello (1): > > * TLSv1.2 (IN), TLS handshake, Server hello (2): > > * TLSv1.2 (IN), TLS handshake, Certificate (11): > > * TLSv1.2 (IN), TLS handshake, Server key exchange (12): > > * TLSv1.2 (IN), TLS handshake, Server finished (14): > > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): > > * TLSv1.2 (OUT), TLS change cipher, Client hello (1): > > * TLSv1.2 (OUT), TLS handshake, Finished (20): > > * TLSv1.2 (IN), TLS change cipher, Client hello (1): > > * TLSv1.2 (IN), TLS handshake, Finished (20): > > * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 > > * ALPN, server accepted to use h2 > > * Server certificate: > > * subject: [removed] > > * start date: Mar 20 00:00:00 2017 GMT > > * expire date: Mar 24 12:00:00 2020 GMT > > * subjectAltName: host "" matched cert's "" > > * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA > > * SSL certificate verify ok. > > * Using HTTP2, server supports multi-use > > * Connection state changed (HTTP/2 confirmed) > > * Copying HTTP/2 data in stream buffer to connection buffer after > upgrade: len=0 > > * Using Stream ID: 1 (easy handle 0x7fbded005400) > > > HEAD /libs/cq/core/content/welcome.html HTTP/2 > > > Host: > > > User-Agent: curl/7.54.0 > > > Accept: */* > > > > > * Connection state changed (MAX_CONCURRENT_STREAMS updated)! > > * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) > > * Closing connection 0 > > * TLSv1.2 (OUT), TLS alert, Client hello (1): > > curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err > 2) > > [2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] > incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - > - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1" > -- Cyril Bonté
Re: HAProxy 1.8.x not serving errorfiles with H2
Hi Cyril, On Mon, Jun 11, 2018 at 10:36:43PM +0200, Cyril Bonté wrote: > If it looks like HTTP/0.9, I tend to think that your errorfile is not > properly set. Such files must contain the status line, the headers and the > response body. > > And indeed, from a quick test, if I remove the status line, I get the same > error with a HTTP/2 request. Once the file is correctly defined, everything > is OK. > > Can you provide the content of your errorfile(s) ? That's indeed very possible. The H2 to H1 gateway requires that the server side respects HTTP/1.1 messaging and semantics, otherwise it will not be able to parse the response. It's obviously the same for error messages since these ones are converted back to H2 by the same gateway. Willy
Re: HAProxy 1.8.x not serving errorfiles with H2
Hi, Le 11/06/2018 à 16:39, J. Casalino a écrit : Trying again – has anyone else seen this? *From:* J. Casalino [mailto:casal...@adobe.com] *Sent:* Tuesday, June 5, 2018 12:27 PM *To:* haproxy@formilux.org *Subject:* HAProxy 1.8.x not serving errorfiles with H2 We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we’ve noticed that these errorfiles are not served to the user’s browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile). Chrome returns "ERR_SPDY_PROTOCOL_ERROR”, Curl [1] returns “curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)”, and Firefox shows “The connection to was interrupted while the page was loading.” With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn’t seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9) If it looks like HTTP/0.9, I tend to think that your errorfile is not properly set. Such files must contain the status line, the headers and the response body. And indeed, from a quick test, if I remove the status line, I get the same error with a HTTP/2 request. Once the file is correctly defined, everything is OK. Can you provide the content of your errorfile(s) ? This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64. Thanks in advance for any thoughts you might have – [1] Curl verbose (curl -I) output: * Trying ... * TCP_NODELAY set * Connected to () port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: [removed] * start date: Mar 20 00:00:00 2017 GMT * expire date: Mar 24 12:00:00 2020 GMT * subjectAltName: host "" matched cert's "" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fbded005400) > HEAD /libs/cq/core/content/welcome.html HTTP/2 > Host: > User-Agent: curl/7.54.0 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS updated)! * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) [2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1" -- Cyril Bonté
RE: HAProxy 1.8.x not serving errorfiles with H2
Trying again - has anyone else seen this? From: J. Casalino [mailto:casal...@adobe.com] Sent: Tuesday, June 5, 2018 12:27 PM To: haproxy@formilux.org Subject: HAProxy 1.8.x not serving errorfiles with H2 We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that these errorfiles are not served to the user's browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile). Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and Firefox shows "The connection to was interrupted while the page was loading." With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn't seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9) This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64. Thanks in advance for any thoughts you might have - [1] Curl verbose (curl -I) output: * Trying ... * TCP_NODELAY set * Connected to () port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: [removed] * start date: Mar 20 00:00:00 2017 GMT * expire date: Mar 24 12:00:00 2020 GMT * subjectAltName: host "" matched cert's "" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fbded005400) > HEAD /libs/cq/core/content/welcome.html HTTP/2 > Host: > User-Agent: curl/7.54.0 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS updated)! * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) [2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
HAProxy 1.8.x not serving errorfiles with H2
We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that these errorfiles are not served to the user's browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile). Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and Firefox shows "The connection to was interrupted while the page was loading." With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn't seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9) This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64. Thanks in advance for any thoughts you might have - [1] Curl verbose (curl -I) output: * Trying ... * TCP_NODELAY set * Connected to () port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: [removed] * start date: Mar 20 00:00:00 2017 GMT * expire date: Mar 24 12:00:00 2020 GMT * subjectAltName: host "" matched cert's "" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fbded005400) > HEAD /libs/cq/core/content/welcome.html HTTP/2 > Host: > User-Agent: curl/7.54.0 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS updated)! * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) [2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
