Hello everybody:
I'm experimenting a bit with plaintext HTTP/2 backends (for gRPC 😢) and
encountering some weird issues. My goal is to be able to support etcd,
which multiplexes a normal good API on HTTP/1.1 with a gRPC API on HTTP/2
on the same plaintext port. I'm doing my testing on HAproxy 2.2.0 so that I
can get the PRI upgrade support.
In order to pass through HTTP/2 to the backend, it seems like I need "proto
h2" on the `server` line? Otherwise I see that the frontend connection is
HTTP/2 and the backend connection is HTTP/1.1 (verified with tcpdump on the
load balancer and the backend), which does not work for the gRPC use case.
Is this expected? One would think that the h2-upgrade functionality would
upgrade both the frontend *and* the backend connection. And, of course,
when "proto h2" is on the server line, HTTP/1.1 requests fail entirely.
As an aside: when "proto h2" is on the server line, is there any way to do
a healthcheck with HTTP/1.1? It is... very annoying to construct a gRPC
healthcheck message using the http-check send language, but the -vvv output
indicates that there's no way to pass a non-h2 proto.
I have no inclination to provision thousands of TLS certificates for all
these internal use-cases and would love to just get HTTP/2 working over
plaintext, but it kind of seems like even under HAproxy 2.2 this is still
not really functional.
Initial config looks something like the following:
frontend sys-etcd_fe
mode http
option http-buffer-request
option httplog
option log-separate-errors
timeout client 9s
timeout client-fin 12s
bind ::1:26462
default_backend sys-etcd_be
backend sys-etcd_be
mode http
option forwardfor
option redispatch
retries 3
timeout connect 4s
timeout server 90s
timeout tunnel 3600s
server host1 10.132.15.129:26462 weight 16
This works fine for HTTP/1.1 but cannot be used to transit gRPC traffic
because it doesn't support plaintext HTTP/2. Even if the client connects
with HTTP/2 (e.g., with --http2 in curl, or with a gRPC client), that only
affects the frontend connection, not the backend connection.
The following config passes HTTP/2 but does not work for HTTP/1.1 at all:
frontend sys-etcd_fe
mode http
option http-buffer-request
option httplog
option log-separate-errors
timeout client 9s
timeout client-fin 12s
bind ::1:26462
default_backend sys-etcd_be
backend sys-etcd_be
mode http
option forwardfor
option redispatch
retries 3
http-check connect proto h2
http-check send meth POST uri /etcdserverpb.Maintenance/Status body
"00000" ver HTTP/2 hdr Host etcd hdr Content-Type application/grpc
http-check expect status 200
timeout connect 4s
timeout server 90s
timeout tunnel 3600s
server host1 10.132.15.129:26462 weight 16 proto h2 check
This is the best config I've got so far, which feels like it shouldn't
work, but does:
frontend sys-etcd_fe
mode http
option http-buffer-request
option httplog
option log-separate-errors
timeout client 9s
timeout client-fin 12s
bind ::1:26462
use_backend sys-etcd_be_h2 if { fc_http_major eq 2 }
default_backend sys-etcd_be_h1
backend sys-etc_be_h1
mode http
option forwardfor
option redispatch
option httpchk /health
retries 3
timeout connect 4s
timeout server 90s
timeout tunnel 3600s
server host1 10.132.15.129:26462 weight 16 check
backend sys-etcd_be_h2
mode http
option forwardfor
option redispatch
retries 3
http-check connect proto h2
http-check send meth POST uri /etcdserverpb.Maintenance/Status body
"00000" ver HTTP/2 hdr Host etcd hdr Content-Type application/grpc
http-check expect status 200
timeout connect 4s
timeout server 90s
timeout tunnel 3600s
server host1 10.132.15.129:26462 weight 16 proto h2 check
HAproxy version output:
$ /usr/sbin/haproxy -vvv
HA-Proxy version 2.2.0 2020/07/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.0.html
Running on: Linux 5.4.15-1ep.el6.x86_64 #1 SMP Mon Jan 27 17:13:12 UTC 2020
x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -Wall -Wextra
-Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wtype-limits
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=
Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT
+POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
+GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO
-NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL
+THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019
Running on OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with zlib version : 1.2.3
Running on zlib version : 1.2.3
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 4.4.7 20120313 (Red Hat 4.4.7-23)
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
fcgi : mode=HTTP side=BE mux=FCGI
<default> : mode=HTTP side=FE|BE mux=H1
h2 : mode=HTTP side=FE|BE mux=H2
<default> : mode=TCP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
[CACHE] cache
[FCGI] fcgi-app
--
James Brown
Engineer
