Hello,
On 28/09/2023 00:30, Shawn Heisey wrote:
The haproxy -vv output is at the end of this message.
I got the built-in OCSP udpating mechanism working. Works beautifully.
Today I discovered that once an hour when the OCSP gets updated,
haproxy stops all its proxies and starts them back up. syslog:
That's really strange, the OCSP update mechanism does not have anything
to do with proxies. Are you sure you did not have a crash and
autorestart of your haproxy ?
Sep 27 15:00:01 - haproxy[3520801] Proxy web80 stopped (cumulated
conns: FE: 42, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy web stopped (cumulated conns:
FE: 1403, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_deny stopped (cumulated
conns: FE: 0, BE: 122).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi1_81 stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi2_81 stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi3_81 stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_smeagol_81 stopped
(cumulated conns: FE: 0, BE: 700).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_plex_32400_tls stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab_8881 stopped
(cumulated conns: FE: 0, BE: 235).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab2_8881 stopped
(cumulated conns: FE: 0, BE: 180).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_artifactory_8082 stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_zabbix_81 stopped
(cumulated conns: FE: 0, BE: 969).
Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.668]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED_org.wildcards.combined
.pem 1 "Update successful" 0 1
Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.795]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p
em 1 "Update successful" 0 1
Sep 27 15:00:01 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.944]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED_org.wildcards.combined
.pem 1 "Update successful" 0 2
Sep 27 15:00:02 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.998]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p
em 1 "Update successful" 0 2
The really irritating effect is that once an hour, my Zabbix server
records an event saying haproxy has been restarted:
https://imgur.com/a/WPkKoFa
(imgur will claim the image has mature content. it doesn't.)
It looks like the only thing that resets back to zero on the stats
page is the uptime in the "status" column for each backend. That's
good news, but I would hope for none of the data to be reset.
I have one big concern, which may be unfounded: I'm worried that the
proxies going down will mean that in-flight connections will be
terminated. I'm guessing that the work for seamless reloads will
ensure that doesn't happen, I just want to be sure.
Not knowing a lot about how haproxy is architected, I do not know if
there is some reason that the backends have to be cycled. Seems like
only frontends that listen with TLS would need that. I would hope it
would be possible to even avoid that ... maybe have OCSP data be
copied from a certain memory location every time a frontend needs it,
and when OCSP gets updated, overwrite the data in that memory location
in a thread-safe way. I know a fair amount about thread safety in
Java, but nothing about it in C.
That's what's supposed to happen, no listener should have to be restarted :)
That's why it looks more like a crash to me.
Final questions for today:
1) Can the OCSP update interval be changed? I don't recall exactly
what the validity for a LetsEncrypt OCSP response is, but I know it
was at least 24 hours, and I think it might have even been as long as
a week. I would like to increase the interval to 8-12 hours if I can.
The "tune.ssl.ocsp-update.maxdelay" and "tune.ssl.ocsp-update.mindelay"
global options can change the default update interval. The idea behind
the update every hour is that we can't just rely on the OCSP response's
expiration time to determine when to perform update since it might be
updated before its expiration date. We went for a 1 hour interval pretty
arbitrarily but the mentioned options allow to adapt it to your use case.
Just note that those options will concern all the OCSP updates, we do
not have a per-certificate configuration mean yet.
2) There are two certs being used in my setup, and haproxy logs
updates for both of them twice. I would have hoped for that to only
happen once. I'm a bit mystified by the fact that it is done twice.
I would have expected either one time or four times ... I have one
frontend that listens with TLS, with four bind lines all using exactly
the same certificate list. (one TCP, and three UDP)
Not sure why that's the case yet. You could use some CLI commands to
check whether the OCSP update tree is built correctly (that should be
done at build time unless you perform some runtime modifications of your
conf). You could use the "show ssl ocsp-updates" command to dump the
list of updated responses.
A way to check for a possible crash in the OCSP update code would be to
use the "update ssl ocsp-response <certfile>" from the CLI as well. It
would use most of the OCSP update code so if a crash were to happen you
might see it this way.
Thanks
Rémi LB
