Re: Problems using custom error files with HTTP/2
Hi Alex,
On 2.6, with H2 enabled, there are certain error pages that HAProxy
can't serve over HTTP/2.
For example, if all the back ends are unavailable it should return an
error page, but it just returns an invalid H2 response.
To reproduce, just set up a minimal config with an H2 frontend and a
single backend which can't be reached.
Earlier in this thread back in April, Willy said:
On 15/04/2023 22:08, Willy Tarreau wrote:
And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which are silent
about this. We overlooked this specificity, it would need to be rethought a
little bit I guess.
Willy
Hope that helps - if there's anything else I can provide/issues I can
raise just let me know.
Regards,
Nick
On 07/08/2023 21:18, Aleksandar Lazic wrote:
Hi.
On 2023-08-07 (Mo.) 18:35, Nick Wood wrote:
Hello all,
I'm not sure if anything further happened with this, but after
upgrading from 2.6 to 2.8.1, custom pages are now broken by default
over HTTP/2.
Please can you specific more deeper what you mean with "broken by
default".
What does not work anymore?
what's your config?
Is the custom page also broken when you activate H2 on 2.6?
Has HTTP/2 support been enabled by default? If so how would one turn
it off so we don't have to downgrade back to v2.6?
In the Announcement of 1.8 is described how to deactivate the H2.
https://www.mail-archive.com/haproxy@formilux.org/msg43600.html
```
- HTTP/2 is advertised by default in ALPN on TLS listeners. It was about
time, 5 years have passed since it was introduced, it's been enabled by
default in clear text as an HTTP/1 upgrade for 4 years, yet some users
do not know how to enable it. From now on, ALPN defaults to
"h2,http/1.1"
on TCP and "h3" on QUIC so that these protocol versions work by
default.
It's still possible to set/reset the ALPN to disable them of course.
The
old concern some users were having about window sizes was addressed by
having a setting for each side (front vs back).
```
That the doc link to the alpn keyword.
http://docs.haproxy.org/2.8/configuration.html#5.1-alpn
Thanks,
Nick
Regards
Alex
On 17/04/2023 15:09, Aleksandar Lazic wrote:
On 17.04.23 15:08, Willy Tarreau wrote:
On Mon, Apr 17, 2023 at 03:04:05PM +0200, Lukas Tribus wrote:
On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
Hi Nick,
On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
And here is my configuration - I've slimmed it down to the
absolute minimum
to reproduce the problem:
If the back end is down, the custom 503.http page should be
served.
This works on HTTP/1.1 but not over HTTP/2:
Very useful, thank you. In fact it's irrelevant to the errorfile
but
it's the 503 that is not produced in this case. I suspect that it's
interpreted on the server side as only a retryable connection error
and that if the HTTP/1 client had faced it on its second request it
would have been the same (in H1 there's a special case for the
first
request on a connection, that is not automatically retryable, but
after the first one we have the luxry of closing silently to force
the client to retry, something that H2 supports natively).
I'm still trying to figure when this problem appeared, and it looks
like even 2.4.0 did behave like this. I'm still digging.
And indeed, this issue appeared with this commit in 1.9-dev10 4
years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which
are silent
about this. We overlooked this specificity, it would need to be
rethought a
little bit I guess.
Even though we had this issue for a long time and nobody noticed, we
should probably not enable H2 on a massive scale with new 2.8
defaults
before this is fixed to avoid silently breaking this error condition.
I totally agree ;-)
Well, I would prefer to keep on the line so that such bugs could be
found much earlier :-).
Jm2c
Willy
Re: Problems using custom error files with HTTP/2
Hi.
On 2023-08-07 (Mo.) 18:35, Nick Wood wrote:
Hello all,
I'm not sure if anything further happened with this, but after upgrading
from 2.6 to 2.8.1, custom pages are now broken by default over HTTP/2.
Please can you specific more deeper what you mean with "broken by default".
What does not work anymore?
what's your config?
Is the custom page also broken when you activate H2 on 2.6?
Has HTTP/2 support been enabled by default? If so how would one turn it
off so we don't have to downgrade back to v2.6?
In the Announcement of 1.8 is described how to deactivate the H2.
https://www.mail-archive.com/haproxy@formilux.org/msg43600.html
```
- HTTP/2 is advertised by default in ALPN on TLS listeners. It was about
time, 5 years have passed since it was introduced, it's been enabled by
default in clear text as an HTTP/1 upgrade for 4 years, yet some users
do not know how to enable it. From now on, ALPN defaults to "h2,http/1.1"
on TCP and "h3" on QUIC so that these protocol versions work by default.
It's still possible to set/reset the ALPN to disable them of course. The
old concern some users were having about window sizes was addressed by
having a setting for each side (front vs back).
```
That the doc link to the alpn keyword.
http://docs.haproxy.org/2.8/configuration.html#5.1-alpn
Thanks,
Nick
Regards
Alex
On 17/04/2023 15:09, Aleksandar Lazic wrote:
On 17.04.23 15:08, Willy Tarreau wrote:
On Mon, Apr 17, 2023 at 03:04:05PM +0200, Lukas Tribus wrote:
On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
Hi Nick,
On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
And here is my configuration - I've slimmed it down to the
absolute minimum
to reproduce the problem:
If the back end is down, the custom 503.http page should be served.
This works on HTTP/1.1 but not over HTTP/2:
Very useful, thank you. In fact it's irrelevant to the errorfile but
it's the 503 that is not produced in this case. I suspect that it's
interpreted on the server side as only a retryable connection error
and that if the HTTP/1 client had faced it on its second request it
would have been the same (in H1 there's a special case for the first
request on a connection, that is not automatically retryable, but
after the first one we have the luxry of closing silently to force
the client to retry, something that H2 supports natively).
I'm still trying to figure when this problem appeared, and it looks
like even 2.4.0 did behave like this. I'm still digging.
And indeed, this issue appeared with this commit in 1.9-dev10 4
years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which
are silent
about this. We overlooked this specificity, it would need to be
rethought a
little bit I guess.
Even though we had this issue for a long time and nobody noticed, we
should probably not enable H2 on a massive scale with new 2.8 defaults
before this is fixed to avoid silently breaking this error condition.
I totally agree ;-)
Well, I would prefer to keep on the line so that such bugs could be
found much earlier :-).
Jm2c
Willy
Re: Problems using custom error files with HTTP/2
Hello all,
I'm not sure if anything further happened with this, but after upgrading
from 2.6 to 2.8.1, custom pages are now broken by default over HTTP/2.
Has HTTP/2 support been enabled by default? If so how would one turn it
off so we don't have to downgrade back to v2.6?
Thanks,
Nick
On 17/04/2023 15:09, Aleksandar Lazic wrote:
On 17.04.23 15:08, Willy Tarreau wrote:
On Mon, Apr 17, 2023 at 03:04:05PM +0200, Lukas Tribus wrote:
On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
Hi Nick,
On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
And here is my configuration - I've slimmed it down to the
absolute minimum
to reproduce the problem:
If the back end is down, the custom 503.http page should be served.
This works on HTTP/1.1 but not over HTTP/2:
Very useful, thank you. In fact it's irrelevant to the errorfile but
it's the 503 that is not produced in this case. I suspect that it's
interpreted on the server side as only a retryable connection error
and that if the HTTP/1 client had faced it on its second request it
would have been the same (in H1 there's a special case for the first
request on a connection, that is not automatically retryable, but
after the first one we have the luxry of closing silently to force
the client to retry, something that H2 supports natively).
I'm still trying to figure when this problem appeared, and it looks
like even 2.4.0 did behave like this. I'm still digging.
And indeed, this issue appeared with this commit in 1.9-dev10 4
years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which
are silent
about this. We overlooked this specificity, it would need to be
rethought a
little bit I guess.
Even though we had this issue for a long time and nobody noticed, we
should probably not enable H2 on a massive scale with new 2.8 defaults
before this is fixed to avoid silently breaking this error condition.
I totally agree ;-)
Well, I would prefer to keep on the line so that such bugs could be
found much earlier :-).
Jm2c
Willy
Re: Problems using custom error files with HTTP/2
On 17.04.23 15:08, Willy Tarreau wrote:
On Mon, Apr 17, 2023 at 03:04:05PM +0200, Lukas Tribus wrote:
On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
Hi Nick,
On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
And here is my configuration - I've slimmed it down to the absolute minimum
to reproduce the problem:
If the back end is down, the custom 503.http page should be served.
This works on HTTP/1.1 but not over HTTP/2:
Very useful, thank you. In fact it's irrelevant to the errorfile but
it's the 503 that is not produced in this case. I suspect that it's
interpreted on the server side as only a retryable connection error
and that if the HTTP/1 client had faced it on its second request it
would have been the same (in H1 there's a special case for the first
request on a connection, that is not automatically retryable, but
after the first one we have the luxry of closing silently to force
the client to retry, something that H2 supports natively).
I'm still trying to figure when this problem appeared, and it looks
like even 2.4.0 did behave like this. I'm still digging.
And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which are silent
about this. We overlooked this specificity, it would need to be rethought a
little bit I guess.
Even though we had this issue for a long time and nobody noticed, we
should probably not enable H2 on a massive scale with new 2.8 defaults
before this is fixed to avoid silently breaking this error condition.
I totally agree ;-)
Well, I would prefer to keep on the line so that such bugs could be
found much earlier :-).
Jm2c
Willy
Re: Problems using custom error files with HTTP/2
On Mon, Apr 17, 2023 at 03:04:05PM +0200, Lukas Tribus wrote:
> On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
> >
> > On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
> > > Hi Nick,
> > >
> > > On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
> > > > And here is my configuration - I've slimmed it down to the absolute
> > > > minimum
> > > > to reproduce the problem:
> > > >
> > > > If the back end is down, the custom 503.http page should be served.
> > > >
> > > > This works on HTTP/1.1 but not over HTTP/2:
> > >
> > > Very useful, thank you. In fact it's irrelevant to the errorfile but
> > > it's the 503 that is not produced in this case. I suspect that it's
> > > interpreted on the server side as only a retryable connection error
> > > and that if the HTTP/1 client had faced it on its second request it
> > > would have been the same (in H1 there's a special case for the first
> > > request on a connection, that is not automatically retryable, but
> > > after the first one we have the luxry of closing silently to force
> > > the client to retry, something that H2 supports natively).
> > >
> > > I'm still trying to figure when this problem appeared, and it looks
> > > like even 2.4.0 did behave like this. I'm still digging.
> >
> > And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
> >
> > 746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
> > conn_streams.")
> >
> > So it makes h2 behave like the second and more H1 requests which are silent
> > about this. We overlooked this specificity, it would need to be rethought a
> > little bit I guess.
>
> Even though we had this issue for a long time and nobody noticed, we
> should probably not enable H2 on a massive scale with new 2.8 defaults
> before this is fixed to avoid silently breaking this error condition.
I totally agree ;-)
Willy
Re: Problems using custom error files with HTTP/2
On Sat, 15 Apr 2023 at 23:08, Willy Tarreau wrote:
>
> On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
> > Hi Nick,
> >
> > On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
> > > And here is my configuration - I've slimmed it down to the absolute
> > > minimum
> > > to reproduce the problem:
> > >
> > > If the back end is down, the custom 503.http page should be served.
> > >
> > > This works on HTTP/1.1 but not over HTTP/2:
> >
> > Very useful, thank you. In fact it's irrelevant to the errorfile but
> > it's the 503 that is not produced in this case. I suspect that it's
> > interpreted on the server side as only a retryable connection error
> > and that if the HTTP/1 client had faced it on its second request it
> > would have been the same (in H1 there's a special case for the first
> > request on a connection, that is not automatically retryable, but
> > after the first one we have the luxry of closing silently to force
> > the client to retry, something that H2 supports natively).
> >
> > I'm still trying to figure when this problem appeared, and it looks
> > like even 2.4.0 did behave like this. I'm still digging.
>
> And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
>
> 746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
> conn_streams.")
>
> So it makes h2 behave like the second and more H1 requests which are silent
> about this. We overlooked this specificity, it would need to be rethought a
> little bit I guess.
Even though we had this issue for a long time and nobody noticed, we
should probably not enable H2 on a massive scale with new 2.8 defaults
before this is fixed to avoid silently breaking this error condition.
Lukas
Re: Problems using custom error files with HTTP/2
Hi Willy,
On 15/04/2023 22:08, Willy Tarreau wrote:
And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new
conn_streams.")
So it makes h2 behave like the second and more H1 requests which are silent
about this. We overlooked this specificity, it would need to be rethought a
little bit I guess.
Willy
Thanks for looking into this - at least I know its not an issue with our
config.
Just to give you some background: This backend is a development/test
server so its not always powered on.
That's why we return a friendly 503 error so that people know the reason
why the backend is not available.
I did look at trying to workaround this using a backup server to serve
my 503 if we can't connect to the primary.
However I don't want to introduce a check as this server is down quite a
lot of the time by design and I don't want HAProxy reporting it as a
problem when it isn't.
For now we're just going to stick with HTTP/1.1, but if there's anything
I can do to assist in fixing this please let me know. Unfortunately I'm
not up to speed on the internals of HTTP/2 at all.
Regards,
Nick
Re: Problems using custom error files with HTTP/2
On Sat, Apr 15, 2023 at 10:59:42PM +0200, Willy Tarreau wrote:
> Hi Nick,
>
> On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote:
> > And here is my configuration - I've slimmed it down to the absolute minimum
> > to reproduce the problem:
> >
> > If the back end is down, the custom 503.http page should be served.
> >
> > This works on HTTP/1.1 but not over HTTP/2:
>
> Very useful, thank you. In fact it's irrelevant to the errorfile but
> it's the 503 that is not produced in this case. I suspect that it's
> interpreted on the server side as only a retryable connection error
> and that if the HTTP/1 client had faced it on its second request it
> would have been the same (in H1 there's a special case for the first
> request on a connection, that is not automatically retryable, but
> after the first one we have the luxry of closing silently to force
> the client to retry, something that H2 supports natively).
>
> I'm still trying to figure when this problem appeared, and it looks
> like even 2.4.0 did behave like this. I'm still digging.
And indeed, this issue appeared with this commit in 1.9-dev10 4 years ago:
746fb772f ("MEDIUM: mux_h2: Always set CS_FL_NOT_FIRST for new conn_streams.")
So it makes h2 behave like the second and more H1 requests which are silent
about this. We overlooked this specificity, it would need to be rethought a
little bit I guess.
Willy
Re: Problems using custom error files with HTTP/2
Hi Nick, On Sat, Apr 15, 2023 at 09:44:32PM +0100, Nick Wood wrote: > And here is my configuration - I've slimmed it down to the absolute minimum > to reproduce the problem: > > If the back end is down, the custom 503.http page should be served. > > This works on HTTP/1.1 but not over HTTP/2: Very useful, thank you. In fact it's irrelevant to the errorfile but it's the 503 that is not produced in this case. I suspect that it's interpreted on the server side as only a retryable connection error and that if the HTTP/1 client had faced it on its second request it would have been the same (in H1 there's a special case for the first request on a connection, that is not automatically retryable, but after the first one we have the luxry of closing silently to force the client to retry, something that H2 supports natively). I'm still trying to figure when this problem appeared, and it looks like even 2.4.0 did behave like this. I'm still digging. Thanks! Willy
Re: Problems using custom error files with HTTP/2
Hi Alex,
On 15/04/2023 20:32, Aleksandar Lazic wrote:
Please can you share the haproxy version `haproxy -vv`.
What is your configuration? Include as much configuration as possible,
including global and default sections. Replace confidential data like
domain names and IP addresses.
Output of haproxy -vv:
HAProxy version 2.6.12-f588462 2023/03/28 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.12.html
Running on: Linux 5.15.0-1033-aws #37~20.04.1-Ubuntu SMP Fri Mar 17
11:39:30 UTC 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement
-Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2
-Wduplicated-cond -Wnull-dereference -fwrapv
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
+LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY -LUA -MEMORY_PROFILING +NETFILTER
+NS -OBSOLETE_LINKER +OPENSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT
+POLL +PRCTL -PROCCTL -PROMEX -QUIC +RT -SLZ -STATIC_PCRE -STATIC_PCRE2
+SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
: mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
: mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
And here is my configuration - I've slimmed it down to the absolute
minimum to reproduce the problem:
If the back end is down, the custom 503.http page should be served.
This works on HTTP/1.1 but not over HTTP/2:
global
maxconn 4096
tune.ssl.default-dh-param 2048
ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
crt-base /etc/ssl/private
log /dev/log local2
defaults
mode http
timeout connect 1000ms
timeout client 3ms
timeout http-request 5s
timeout server 65000ms
option dontlognull
option redispatch
log global
errorfile 503 /etc/haproxy/errorpages/503.http
frontend https-in
bind *:443,:::443 v6only ssl crt /etc/haproxy/certs alpn h2,http/1.1
default_backend singleserver
option forwardfor
option httplog
capture request header Host len 48
backend singleserver
server app1 172.31.131.199:80 maxconn 64
retries 1
Thanks,
Nick
Re: Problems using custom error files with HTTP/2
Hi Nic, On 15.04.23 19:35, Nick Wood wrote: Hello all, I have recently enabled HTTP/2 on our HAProxy server by adding the following to the bind line: alpn h2,http/1.1 Everything appears to be working fine, apart from our custom error pages. Rather than serving the custom page as before, browsers just report an error. In Chrome its ERR_HTTP2_SERVER_REFUSED_STREAM. In Firefox its a more generic response about the data being invalid. Here is the content of /etc/haproxy/errorpages/503.http: [snipp] I've searched the archives but not found anyone else with this issue - apart from someone who didn't have the correct HTTP headers defined at the top of their error file - but mine look OK. I've tried using HTTP/1.1 instead of HTTP/1.0 and also removing the Connection: close header, but nothing makes a difference. Any clues as to what I'm doing wrong would be much appreciated. Please can you share the haproxy version `haproxy -vv`. What is your configuration? Include as much configuration as possible, including global and default sections. Replace confidential data like domain names and IP addresses. Thanks, Nick Best regards Alex
