HAProxy Stats and SSL Problems
Hello,
I've been trying to diagnose an odd issue with HAProxy (1.5.x)
statistics and SSL. I'm seeing clients having problems with the SSL
negotiation. When digging with openssl, there seems to be a clear text
http 1.x response which causes the negotiation to fail:
$ openssl s_client -debug -connect lb.com:44300
CONNECTED(0003)
write to 0x7f96a3504c70 [0x7f96a3804200] (130 bytes = 130 (0x82))
- 80 80 01 03 01 00 57 00-00 00 20 00 00 39 00 00 ..W... ..9..
0010 - 38 00 00 35 00 00 16 00-00 13 00 00 0a 07 00 c0 8..5
0020 - 00 00 33 00 00 32 00 00-2f 00 00 9a 00 00 99 00 ..3..2../...
0030 - 00 96 03 00 80 00 00 05-00 00 04 01 00 80 00 00
0040 - 15 00 00 12 00 00 09 06-00 40 00 00 14 00 00 11 .@..
0050 - 00 00 08 00 00 06 04 00-80 00 00 03 02 00 80 00
0060 - 00 ff 79 2a 0a d7 d8 37-c8 50 b6 f7 c3 8e ce 96 ..y*...7.P..
0070 - cf 2b d9 b8 92 c5 6f 1f-74 7f c0 d1 22 46 71 7a .+o.t...Fqz
0080 - e2 b4 ..
read from 0x7f96a3504c70 [0x7f96a3809800] (7 bytes = 7 (0x7))
- 48 54 54 50 2f 31 2e HTTP/1.
1371:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol:/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/ssl/s23_clnt.c:618:
$ telnet lb.com 44300
Trying X.X.X.X...
Connected to X.X.X.X.
Escape character is '^]'.
GET /
HTTP/1.0 403 Forbidden
Cache-Control: no-cache
Connection: close
Content-Type: text/html
htmlbodyh1403 Forbidden/h1
Request forbidden by administrative rules.
/body/html
The proxy log doesn't have anything that helps me understand what's
going on:
Jun 15 16:47:44 lb.com haproxy[430]: X.X.X.X:55877
[15/Jun/2015:16:47:44.967] stats stats/NOSRV -1/-1/-1/-1/0 400 187 - -
PR-- 0/0/0/0/3 0/0 BADREQ
The pertinent configuration sections are:
global
log 127.0.0.1local1 info
maxconn 10240
chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
# local stats sockets for read access - change operator to
admin for r/w
stats socket /var/run/haproxy/haproxy.sock mode 0600 level operator
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
# Set global SSL bind options
ssl-default-bind-options no-sslv3 no-tls-tickets
tune.ssl.default-dh-param 2048
ssl-server-verify none
defaults
log global
mode http
optionhttplog
optiondontlognull
retries 3
optionredispatch
maxconn 10240
# Mime types from here:
#
http://blogs.alfresco.com/wp/developer/2013/11/13/haproxy-for-alfresco/
# and here
# http://serverfault.com/questions/575744/nginx-mime-types-and-gzip
compression algo gzip
compression type text/plain text/html text/html;charset=utf-8
text/css text/javascript application/json
listen stats :44300
bind *:44300 ssl crt /etc/ssl/private/the.pem.withkey.pem
mode http
http-request deny if !{ ssl_fc }
stats enable
stats refresh 5s
stats uri /stats
stats realm proxies
stats show-node
stats show-legends
option httplog
option contstats
acl auth_ok_stats http_auth(users_stats)
http-request auth if !auth_ok_stats
Does anyone have any insight?
Thank you in advance,
Matt
Re: HAProxy Stats and SSL Problems
Matthew Cox schreef op 15-6-2015 om 20:05:
Hello,
I've been trying to diagnose an odd issue with HAProxy (1.5.x)
statistics and SSL. I'm seeing clients having problems with the SSL
negotiation. When digging with openssl, there seems to be a clear text
http 1.x response which causes the negotiation to fail:
$ openssl s_client -debug -connect lb.com:44300
CONNECTED(0003)
write to 0x7f96a3504c70 [0x7f96a3804200] (130 bytes = 130 (0x82))
- 80 80 01 03 01 00 57 00-00 00 20 00 00 39 00 00 ..W... ..9..
0010 - 38 00 00 35 00 00 16 00-00 13 00 00 0a 07 00 c0 8..5
0020 - 00 00 33 00 00 32 00 00-2f 00 00 9a 00 00 99 00 ..3..2../...
0030 - 00 96 03 00 80 00 00 05-00 00 04 01 00 80 00 00
0040 - 15 00 00 12 00 00 09 06-00 40 00 00 14 00 00 11 .@..
0050 - 00 00 08 00 00 06 04 00-80 00 00 03 02 00 80 00
0060 - 00 ff 79 2a 0a d7 d8 37-c8 50 b6 f7 c3 8e ce 96 ..y*...7.P..
0070 - cf 2b d9 b8 92 c5 6f 1f-74 7f c0 d1 22 46 71 7a .+o.t...Fqz
0080 - e2 b4 ..
read from 0x7f96a3504c70 [0x7f96a3809800] (7 bytes = 7 (0x7))
- 48 54 54 50 2f 31 2e HTTP/1.
1371:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol:/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/ssl/s23_clnt.c:618:
$ telnet lb.com 44300
Trying X.X.X.X...
Connected to X.X.X.X.
Escape character is '^]'.
GET /
HTTP/1.0 403 Forbidden
Cache-Control: no-cache
Connection: close
Content-Type: text/html
htmlbodyh1403 Forbidden/h1
Request forbidden by administrative rules.
/body/html
The proxy log doesn't have anything that helps me understand what's
going on:
Jun 15 16:47:44 lb.com haproxy[430]: X.X.X.X:55877
[15/Jun/2015:16:47:44.967] stats stats/NOSRV -1/-1/-1/-1/0 400 187 -
- PR-- 0/0/0/0/3 0/0 BADREQ
The pertinent configuration sections are:
global
log 127.0.0.1 local1 info
maxconn 10240
chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
# local stats sockets for read access - change operator to
admin for r/w
stats socket /var/run/haproxy/haproxy.sock mode 0600 level operator
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
# Set global SSL bind options
ssl-default-bind-options no-sslv3 no-tls-tickets
tune.ssl.default-dh-param 2048
ssl-server-verify none
defaults
log global
mode http
optionhttplog
optiondontlognull
retries 3
optionredispatch
maxconn 10240
# Mime types from here:
#
http://blogs.alfresco.com/wp/developer/2013/11/13/haproxy-for-alfresco/
# and here
# http://serverfault.com/questions/575744/nginx-mime-types-and-gzip
compression algo gzip
compression type text/plain text/html text/html;charset=utf-8
text/css text/javascript application/json
listen stats :44300
Remove the port like:
listen stats
bind *:44300 ssl crt /etc/ssl/private/the.pem.withkey.pem
mode http
http-request deny if !{ ssl_fc }
stats enable
stats refresh 5s
stats uri /stats
stats realm proxies
stats show-node
stats show-legends
option httplog
option contstats
acl auth_ok_stats http_auth(users_stats)
http-request auth if !auth_ok_stats
Does anyone have any insight?
Thank you in advance,
Matt
Re: HAProxy Stats and SSL Problems
As stated by Piba-nl, your error is here: listen stats :44300 bind *:44300 ssl crt /etc/ssl/private/the.pem.withkey.pem When you declare your listen section like this, it is equivalent to: listen stats bind :44300 bind *:44300 ssl crt /etc/ssl/private/the.pem.withkey.pem Which means that 2 listening sockets will get the traffic, one deciphering the traffic, and the other one not... Simply remove the ':44300' from your listen section definition. Baptiste
