Re: H/2 via Unix Sockets fails
Hi Jarno,
On 2019-06-04 12:44, Jarno Huuskonen wrote:
Hi Christian,
On Thu, Apr 25, Christian Ruppert wrote:
listen genlisten_10320-cust1.tls-tcp
acl REQ_TLS_HAS_ECC req.ssl_ec_ext eq 1
tcp-request content accept if { req_ssl_hello_type 1 } # Match
Client SSL Hello
use-server socket-10320-rsa if !REQ_TLS_HAS_ECC
server socket-10320-rsa unix@/run/haproxy-10320-rsa.sock
send-proxy-v2
use-server socket-10320-ecc if REQ_TLS_HAS_ECC
server socket-10320-ecc unix@/run/haproxy-10320-ecc.sock
send-proxy-v2
Do you need this tcp frontend for just serving both rsa/ecc
certificates ?
If so I think haproxy can do this(with openssl >= 1.0.2) with crt
keyword:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.1-crt
-Jarno
listen genlisten_10320-cust1.tls
bind unix@/run/haproxy-10320-rsa.sock accept-proxy user haproxy
group root mode 600 ssl crt /etc/haproxy/test-rsa.pem alpn
h2,http/1.1 process 3
bind unix@/run/haproxy-10320-ecc.sock accept-proxy user haproxy
group root mode 600 ssl crt /etc/haproxy/test-ecc.pem alpn
h2,http/1.1 process 4-8
Yeah, I think we'll still need that construct. What we want to achieve
with this kind of setup is:
One process/core for pure connections (that TCP stuff), one for HTTP,
*one* for RSA and all the rest for ECC. RSA costs so much that it's
really easy to (D)DoS that process which would otherwise affect all
other processes as well. So we just want to have all that separated,
http from https and RSA from ECC.
--
Regards,
Christian Ruppert
Re: H/2 via Unix Sockets fails
Hi Christian,
On Thu, Apr 25, Christian Ruppert wrote:
>
> listen genlisten_10320-cust1.tls-tcp
> acl REQ_TLS_HAS_ECC req.ssl_ec_ext eq 1
> tcp-request content accept if { req_ssl_hello_type 1 } # Match
> Client SSL Hello
>
> use-server socket-10320-rsa if !REQ_TLS_HAS_ECC
> server socket-10320-rsa unix@/run/haproxy-10320-rsa.sock send-proxy-v2
>
> use-server socket-10320-ecc if REQ_TLS_HAS_ECC
> server socket-10320-ecc unix@/run/haproxy-10320-ecc.sock send-proxy-v2
Do you need this tcp frontend for just serving both rsa/ecc
certificates ?
If so I think haproxy can do this(with openssl >= 1.0.2) with crt keyword:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.1-crt
-Jarno
> listen genlisten_10320-cust1.tls
>
> bind unix@/run/haproxy-10320-rsa.sock accept-proxy user haproxy
> group root mode 600 ssl crt /etc/haproxy/test-rsa.pem alpn
> h2,http/1.1 process 3
> bind unix@/run/haproxy-10320-ecc.sock accept-proxy user haproxy
> group root mode 600 ssl crt /etc/haproxy/test-ecc.pem alpn
> h2,http/1.1 process 4-8
--
Jarno Huuskonen
Re: H/2 via Unix Sockets fails
Hi Jarno,
thanks, your propsal seems to work. Here's a working test config based
on one of our production configs:
curl -kvs -o /dev/null https://127.0.0.1:10320 --http1.1
Apr 25 15:32:51 localhost haproxy[2847]: 127.0.0.1:36880
[25/Apr/2019:15:32:51.554] genfrontend_10310-cust1
genfrontend_10310-cust1/ -1/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0
0/0 "GET / HTTP/1.1"
Apr 25 15:32:51 localhost haproxy[2846]: 127.0.0.1:36880
[25/Apr/2019:15:32:51.553] genlisten_10320-cust1.tls~
genlisten_10320-cust1.tls/socket-10310 1/0/1 212 -- 1/1/0/0/0 0/0
Apr 25 15:32:51 localhost haproxy[2841]: 127.0.0.1:36880
[25/Apr/2019:15:32:51.549] genlisten_10320-cust1.tls-tcp
genlisten_10320-cust1.tls-tcp/socket-10320-ecc 4/0/5 995 -- 1/1/0/0/0
0/0
curl -kvs -o /dev/null https://127.0.0.1:10320 --http2
Apr 25 15:32:59 localhost haproxy[2847]: 127.0.0.1:36882
[25/Apr/2019:15:32:59.246] genfrontend_10310-cust1
genfrontend_10310-cust1/ -1/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0
0/0 "GET / HTTP/1.1"
Apr 25 15:32:59 localhost haproxy[2845]: 127.0.0.1:36882
[25/Apr/2019:15:32:59.243] genlisten_10320-cust1.tls~
genlisten_10320-cust1.tls/socket-10310-h2 3/0/3 184 -- 1/1/0/0/0 0/0
Apr 25 15:32:59 localhost haproxy[2841]: 127.0.0.1:36882
[25/Apr/2019:15:32:59.228] genlisten_10320-cust1.tls-tcp
genlisten_10320-cust1.tls-tcp/socket-10320-ecc 16/0/19 990 CD 1/1/0/0/0
0/0
global
nbproc 8
# ...
listen genlisten_10320-cust1.tls-tcp
mode tcp
bind-process 2
bind :10320
log global
option tcplog
# ...
tcp-request inspect-delay 7s
acl REQ_TLS_HAS_ECC req.ssl_ec_ext eq 1
tcp-request content accept if { req_ssl_hello_type 1 } # Match Client
SSL Hello
use-server socket-10320-rsa if !REQ_TLS_HAS_ECC
server socket-10320-rsa unix@/run/haproxy-10320-rsa.sock send-proxy-v2
use-server socket-10320-ecc if REQ_TLS_HAS_ECC
server socket-10320-ecc unix@/run/haproxy-10320-ecc.sock send-proxy-v2
listen genlisten_10320-cust1.tls
mode tcp
log global
option tcplog
bind-process 3-8
bind unix@/run/haproxy-10320-rsa.sock accept-proxy user haproxy group
root mode 600 ssl crt /etc/haproxy/test-rsa.pem alpn h2,http/1.1 process
3
bind unix@/run/haproxy-10320-ecc.sock accept-proxy user haproxy group
root mode 600 ssl crt /etc/haproxy/test-ecc.pem alpn h2,http/1.1 process
4-8
use-server socket-10310-h2 if { ssl_fc_alpn h2 }
server socket-10310-h2 unix@/run/haproxy-10310-h2.sock send-proxy-v2
use-server socket-10310 if !{ ssl_fc_alpn h2 }
server socket-10310 unix@/run/haproxy-10310.sock send-proxy-v2
frontend genfrontend_10310-cust1
bind :10310
bind unix@/run/haproxy-10310-h2.sock id 210312 accept-proxy user
haproxy group root mode 600 proto h2 # TLS uplink H2
bind unix@/run/haproxy-10310.sock id 210310 accept-proxy user haproxy
group root mode 600 # TLS uplink
mode http
option httplog
log global
# ...
So it would be cool if both were possible, H2 as well as H1 via that
socket, using "alpn h2,http/1.1"
--
Regards,
Christian Ruppert
Re: H/2 via Unix Sockets fails
Hi,
On Tue, Apr 23, Christian Ruppert wrote:
> we have an older setup using nbproc >1 and having a listener for the
> initial tcp connection and one for the actual SSL/TLS, also using
> tcp mode which then goes to the actual frontend using http mode.
> Each being bound to different processes.
> So here's the test config I've used:
(Your config seems quite similar to what I tested in this thread:
https://www.mail-archive.com/haproxy@formilux.org/msg32255.html)
It kind of works if you add a second bind (with proto h2) to some_frontend
and from h2test_tcp.tls use server h2 if { ssl_fc_alpn h2 }
(BUT client can (at least in theory) choose alpn h2 and speak http/1.1).
So using mode http on h2test_tcp.tls is probably safer choice.
> listen h2test_tcp
> mode tcp
> bind :444
> option tcplog
> log global
> server socket-444-h2test unix@/run/haproxy-444-h2test.sock
> send-proxy-v2
>
> listen h2test_tcp.tls
> mode tcp
> option tcplog
> log global
> bind unix@/run/haproxy-444-h2test.sock accept-proxy user haproxy
> group haproxy mode 600 ssl crt /etc/haproxy/ssl/h2test.pem alpn
> h2,http/1.1
> server socket-444_2 unix@/run/haproxy-444_2-h2test.sock
> send-proxy-v2
>
> frontend some_frontend
> mode http
> log global
> bind unix@/run/haproxy-444_2-h2test.sock id 444 accept-proxy
> user haproxy group haproxy mode 600
> bind :80
>
> ...
[...]
> curl says:
> # curl -k4vs
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F127.0.0.1%3A444%2Fdata=02%7C01%7C%7Ca512378ae52d4457158908d6c7f5fb34%7C87879f2e73044bf2baf263e7f83f3c34%7C0%7C1%7C636916256693248226sdata=Fw%2F21TnhE6yuJJq4dDDZf%2BdJfIVX3b6yR4nVjoD%2BkhA%3Dreserved=0
> --http2
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to 127.0.0.1 (127.0.0.1) port 444 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * Cipher selection:
[...]
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after
> upgrade: len=0
> * Using Stream ID: 1 (easy handle 0x56087e29b770)
> >GET / HTTP/2
> >Host: 127.0.0.1:444
> >User-Agent: curl/7.64.1
> >Accept: */*
> >
> * http2 error: Remote peer returned unexpected data while we
> expected SETTINGS frame. Perhaps, peer does not support HTTP/2
> properly.
> * Connection #0 to host 127.0.0.1 left intact
> * Closing connection 0
>
> Can anybody else confirm that? Tested with HAProxy 1.9.6.
> Any ideas what might be the reason? Right now, I'd guess that's a
> Problem with H/2 and those sockets on the HAProxy side.
I think the problem is that "bind unix@/run/haproxy-444_2-h2test.sock"
expects/speaks http/1.1.
-Jarno
--
Jarno Huuskonen
Re: H/2 via Unix Sockets fails
Hi Willy, that doesn't seem to work either, only HTTP/1.1 We have several hundret listener/frontends/backends and we're using the old nbproc > 1 process model. We have the initial TCP listener that's bound to one core. It checks wether it's ECC capable or not and then it goes to the second listener that does the actual SSL termination with RSA/ECC on multiple cores and from there it goes to the actual frontend, which is on a different core. We plan to test and migrate to the threading model if it performs as good as the current one or even better. But actually that was meant for much later that year or even 2020 :( I'm not sure if that would solve the actual problem, since may still need sockets for RSA/ECC I guess. The inital plan was to just make it also support HTTP2 by adding "alpn h2,http/1.1" to the unix bind in the "h2test_tcp.tls" On 2019-04-24 15:06, Willy Tarreau wrote: Hi Christian, On Wed, Apr 24, 2019 at 02:29:40PM +0200, Christian Ruppert wrote: Hi, so I did some more tests and it seems to be an issue between h2test_tcp.tls and the frontend, using the UNIX sockets. Adding a TCP bind to that listener also doesn't work. Am I doing it wrong or is it a bug somewhere with H/2 and UNIX sockets? I also disabled the PROXY protocol - doesn't help. I currently have no idea about this one. There should be no reason for H2 to depend on the underlying socket type. Hmm wait a minute. It might not be related to the UNIX sockets at all. In fact what's happening is that your first proxy is not advertising H2 in the ALPN connection, so the second one doesn't receive it and negociates H1. You could try to add "alpn h2" at the end of your server line below : listen h2test_tcp mode tcp bind :444 option tcplog log global server socket-444-h2test unix@/run/haproxy-444-h2test.sock send-proxy-v2 ^ listen h2test_tcp.tls mode tcp option tcplog log global bind unix@/run/haproxy-444-h2test.sock accept-proxy user haproxy group haproxy mode 600 ssl crt /etc/haproxy/ssl/h2test.pem alpn h2,http/1.1 server socket-444_2 unix@/run/haproxy-444_2-h2test.sock send-proxy-v2 ^ And on this one as well. However it will break your H1. What are you trying to do exactly ? Maybe there is a simpler solution. Willy -- Regards, Christian Ruppert
Re: H/2 via Unix Sockets fails
Hi Christian, On Wed, Apr 24, 2019 at 02:29:40PM +0200, Christian Ruppert wrote: > Hi, > > so I did some more tests and it seems to be an issue between h2test_tcp.tls > and the frontend, using the UNIX sockets. Adding a TCP bind to that listener > also doesn't work. Am I doing it wrong or is it a bug somewhere with H/2 and > UNIX sockets? > I also disabled the PROXY protocol - doesn't help. I currently have no idea about this one. There should be no reason for H2 to depend on the underlying socket type. Hmm wait a minute. It might not be related to the UNIX sockets at all. In fact what's happening is that your first proxy is not advertising H2 in the ALPN connection, so the second one doesn't receive it and negociates H1. You could try to add "alpn h2" at the end of your server line below : > listen h2test_tcp > mode tcp > bind :444 > option tcplog > log global > server socket-444-h2test unix@/run/haproxy-444-h2test.sock send-proxy-v2 ^ > listen h2test_tcp.tls > mode tcp > option tcplog > log global > bind unix@/run/haproxy-444-h2test.sock accept-proxy user haproxy > group haproxy mode 600 ssl crt /etc/haproxy/ssl/h2test.pem alpn h2,http/1.1 > server socket-444_2 unix@/run/haproxy-444_2-h2test.sock send-proxy-v2 ^ And on this one as well. However it will break your H1. What are you trying to do exactly ? Maybe there is a simpler solution. Willy
Re: H/2 via Unix Sockets fails
Hi,
so I did some more tests and it seems to be an issue between
h2test_tcp.tls and the frontend, using the UNIX sockets. Adding a TCP
bind to that listener also doesn't work. Am I doing it wrong or is it a
bug somewhere with H/2 and UNIX sockets?
I also disabled the PROXY protocol - doesn't help.
On 2019-04-23 15:57, Christian Ruppert wrote:
Hey,
we have an older setup using nbproc >1 and having a listener for the
initial tcp connection and one for the actual SSL/TLS, also using tcp
mode which then goes to the actual frontend using http mode. Each
being bound to different processes.
So here's the test config I've used:
listen h2test_tcp
mode tcp
bind :444
option tcplog
log global
server socket-444-h2test unix@/run/haproxy-444-h2test.sock
send-proxy-v2
listen h2test_tcp.tls
mode tcp
option tcplog
log global
bind unix@/run/haproxy-444-h2test.sock accept-proxy user haproxy
group haproxy mode 600 ssl crt /etc/haproxy/ssl/h2test.pem alpn
h2,http/1.1
server socket-444_2 unix@/run/haproxy-444_2-h2test.sock
send-proxy-v2
frontend some_frontend
mode http
log global
bind unix@/run/haproxy-444_2-h2test.sock id 444 accept-proxy user
haproxy group haproxy mode 600
bind :80
...
So what I'm doing is:
curl -k4vs https://127.0.0.1:444/~idl0r/ --http1.1
curl -k4vs https://127.0.0.1:444/~idl0r/ --http2
So with HTTP/1.1 I get:
public_http backend_qasl_de/qasl1 0/0/0/0/0 200 510 - - 3/1/0/0/0
0/0 {127.0.0.1:444|curl/7.64.1|} "GET / HTTP/1.1"
h2test_tcp.tls~ h2test_tcp.tls/socket-444_2 5/1/6 605 -- 2/1/0/0/0 0/0
h2test_tcp h2test_tcp/socket-444-h2test 1/0/6 3335 CD 1/1/0/0/0 0/0
With H/2:
public_http public_http/ -1/-1/-1/-1/0 400 187 - - PR--
3/1/0/0/0 0/0 {||} ""
h2test_tcp.tls~ h2test_tcp.tls/socket-444_2 6/0/5 187 SD 2/1/0/0/0 0/0
h2test_tcp h2test_tcp/socket-444-h2test 1/0/5 2911 SD 1/1/0/0/0 0/0
curl says:
# curl -k4vs https://127.0.0.1:444/ --http2
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=...
* start date: Mar 1 18:00:17 2019 GMT
* expire date: May 30 18:00:17 2019 GMT
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56087e29b770)
GET / HTTP/2
Host: 127.0.0.1:444
User-Agent: curl/7.64.1
Accept: */*
* http2 error: Remote peer returned unexpected data while we expected
SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0
Can anybody else confirm that? Tested with HAProxy 1.9.6.
Any ideas what might be the reason? Right now, I'd guess that's a
Problem with H/2 and those sockets on the HAProxy side.
--
Regards,
Christian Ruppert
H/2 via Unix Sockets fails
Hey,
we have an older setup using nbproc >1 and having a listener for the
initial tcp connection and one for the actual SSL/TLS, also using tcp
mode which then goes to the actual frontend using http mode. Each being
bound to different processes.
So here's the test config I've used:
listen h2test_tcp
mode tcp
bind :444
option tcplog
log global
server socket-444-h2test unix@/run/haproxy-444-h2test.sock
send-proxy-v2
listen h2test_tcp.tls
mode tcp
option tcplog
log global
bind unix@/run/haproxy-444-h2test.sock accept-proxy user haproxy
group haproxy mode 600 ssl crt /etc/haproxy/ssl/h2test.pem alpn
h2,http/1.1
server socket-444_2 unix@/run/haproxy-444_2-h2test.sock
send-proxy-v2
frontend some_frontend
mode http
log global
bind unix@/run/haproxy-444_2-h2test.sock id 444 accept-proxy user
haproxy group haproxy mode 600
bind :80
...
So what I'm doing is:
curl -k4vs https://127.0.0.1:444/~idl0r/ --http1.1
curl -k4vs https://127.0.0.1:444/~idl0r/ --http2
So with HTTP/1.1 I get:
public_http backend_qasl_de/qasl1 0/0/0/0/0 200 510 - - 3/1/0/0/0
0/0 {127.0.0.1:444|curl/7.64.1|} "GET / HTTP/1.1"
h2test_tcp.tls~ h2test_tcp.tls/socket-444_2 5/1/6 605 -- 2/1/0/0/0 0/0
h2test_tcp h2test_tcp/socket-444-h2test 1/0/6 3335 CD 1/1/0/0/0 0/0
With H/2:
public_http public_http/ -1/-1/-1/-1/0 400 187 - - PR-- 3/1/0/0/0
0/0 {||} ""
h2test_tcp.tls~ h2test_tcp.tls/socket-444_2 6/0/5 187 SD 2/1/0/0/0 0/0
h2test_tcp h2test_tcp/socket-444-h2test 1/0/5 2911 SD 1/1/0/0/0 0/0
curl says:
# curl -k4vs https://127.0.0.1:444/ --http2
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=...
* start date: Mar 1 18:00:17 2019 GMT
* expire date: May 30 18:00:17 2019 GMT
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56087e29b770)
GET / HTTP/2
Host: 127.0.0.1:444
User-Agent: curl/7.64.1
Accept: */*
* http2 error: Remote peer returned unexpected data while we expected
SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0
Can anybody else confirm that? Tested with HAProxy 1.9.6.
Any ideas what might be the reason? Right now, I'd guess that's a
Problem with H/2 and those sockets on the HAProxy side.
--
Regards,
Christian Ruppert
