Re: OCSP update restarts all proxies
On 10/4/23 09:18, William Lallemand wrote: Nothing in haproxy initiate a service reload, are sure you don't have an external process which is doing it? The systemd support within HAProxy is only meant to provide a status to systemd, it does not send it actions. I found the issue. I am not surprised to learn that it was a PEBCAK problem. :) I have a certs webapp I wrote in PHP for Lets Encrypt certificate generation and management. One of the things it does is update a whitelist and reload haproxy, and it has an hourly cronjob to make sure that the whitelist is always current. I have updated the code for this so that it actually checks to see whether the whitelist has changed, and only issue a reload when there is a change. This will eliminate the hourly reloads. I actually developed this webapp for $DAYJOB and deployed it on my own server as well. A cow-orker noticed frequent alerts from zabbix about haproxy restarting on that system and asked me about it. I have never employed my OCSP updating script on that system ... the only thing the two systems have in common is my certs webapp. I had forgotten about the hourly cronjob. I actually don't need the cronjob on my own server. My personal haproxy doesn't use that whitelist. It's in place for $DAYJOB so that only certain public IP addresses can reach my webapp. Thank you for helping me work out that it was not haproxy's OCSP update that caused the anomaly, that just happened to be occuring at the same time. Shawn
Re: OCSP update restarts all proxies
On Wed, Oct 04, 2023 at 08:52:39AM -0600, Shawn Heisey wrote: > On 10/4/23 05:34, Remi Tricot-Le Breton wrote: > > You just have to run the following commands : > > > > $ echo "update ssl ocsp-response " | socat > > /path_to_socket/haproxy.sock - > > When I do this, the update is successful and shows in the logfile > created by rsyslogd ... but unlike when haproxy does the automatic > hourly update, there is no service reload, so the proxies are not stopped. > > When my old ocsp updating script sent an ocsp response to the stats > socket, there was no service reload either. > > I couldn't follow what's in the src/ssl_ocsp.c file. It has been a > REALLY long time since I actually wrote C code myself. I was hoping to > find out whether or not that code was initiating a service reload when > systemd support is enabled. > > I have tried to find something external to haproxy that might be > initiating the reload, but I haven't found anything. > > Thanks, > Shawn Hello Shawn, Nothing in haproxy initiate a service reload, are sure you don't have an external process which is doing it? The systemd support within HAProxy is only meant to provide a status to systemd, it does not send it actions. HAProxy only sends a "READY=1" status to systemd, so if you are seeing a "Reloaded" message from systemd, it was sent by something external to HAProxy. And it would be a `systemctl reload` action, not a manual kill -USR2 signal. Regards, -- William Lallemand
Re: OCSP update restarts all proxies
On 10/4/23 05:34, Remi Tricot-Le Breton wrote: You just have to run the following commands : $ echo "update ssl ocsp-response " | socat /path_to_socket/haproxy.sock - When I do this, the update is successful and shows in the logfile created by rsyslogd ... but unlike when haproxy does the automatic hourly update, there is no service reload, so the proxies are not stopped. When my old ocsp updating script sent an ocsp response to the stats socket, there was no service reload either. I couldn't follow what's in the src/ssl_ocsp.c file. It has been a REALLY long time since I actually wrote C code myself. I was hoping to find out whether or not that code was initiating a service reload when systemd support is enabled. I have tried to find something external to haproxy that might be initiating the reload, but I haven't found anything. Thanks, Shawn
Re: OCSP update restarts all proxies
On 04/10/2023 03:05, Shawn Heisey wrote: On 10/3/23 01:33, Remi Tricot-Le Breton wrote: This command relies on the same task that performs the automatic update. What it does is basically add the certificate at the top of the task's update list and wakes it up. The update is asynchronous so we can't return a status to the CLI command. In order to check if the update was successful you can display the contents of the updated OCSP response via the "show ssl ocsp-response" command. If the response you updated is also set to be updated automatically, you can also use the "show ssl ocsp-updates" command that gives the update success and failure numbers as well as the last update status for all the responses registered in the auto update list. I have no idea how to get an interactive session going on the stats socket so that I can see whatever response a command generates. The only command I know for the socket is for the old-style OCSP update where the OCSP response is obtained with openssl, converted to base64, and sent to the socket. No response comes back when using socat in this way. You just have to run the following commands : $ echo "update ssl ocsp-response " | socat /path_to_socket/haproxy.sock - This command does not return anything (which is normal because the update will be performed asynchronously). You could see some log lines being emitted by your haproxy (depending on your log configuration) such as the following : <134>Oct 4 13:28:49 haproxy[14955]: -:- [04/Oct/2023:13:28:49.910] /path_to_cert/server_ocsp.pem.rsa 1 "Update successful" 0 1 You can then either check the contents of your OCSP response : $ echo "show ssl ocsp-response " | socat /path_to_socket/haproxy.sock - or, if your tested OCSP response had the 'ocsp-update' option set to 'on', you could dump the contents of the update tree thanks to the "echo ssl ocsp-updates" command: $ echo "show ssl ocsp-updates" | socat /tmp/haproxy.sock - OCSP Certid | Path | Next Update | Last Update | Successes | Failures | Last Update Status | Last Update Status (str) 303b300906052b0e03021a050004148a83e0060faff709ca7e9b95522a2e81635fda0a0414f652b0e435d5ea923851508f0adbe92d85de007a02021015 | /path_to_cert/server_ocsp.pem.rsa | 04/Oct/2023:14:28:49 +0200 | 04/Oct/2023:13:28:49 +0200 | 1 | 0 | 1 | Update successful The "Successes" and "Failures" counters should change when you call the "update ssl ocsp-response" CLI command. Here is my old script for OCSP updates, which I stopped using once I learned how to set up haproxy to do it automatically: https://paste.elyograg.org/view/5e88c914 Rémi LB
Re: OCSP update restarts all proxies
On 10/3/23 01:33, Remi Tricot-Le Breton wrote: This command relies on the same task that performs the automatic update. What it does is basically add the certificate at the top of the task's update list and wakes it up. The update is asynchronous so we can't return a status to the CLI command. In order to check if the update was successful you can display the contents of the updated OCSP response via the "show ssl ocsp-response" command. If the response you updated is also set to be updated automatically, you can also use the "show ssl ocsp-updates" command that gives the update success and failure numbers as well as the last update status for all the responses registered in the auto update list. I have no idea how to get an interactive session going on the stats socket so that I can see whatever response a command generates. The only command I know for the socket is for the old-style OCSP update where the OCSP response is obtained with openssl, converted to base64, and sent to the socket. No response comes back when using socat in this way. Here is my old script for OCSP updates, which I stopped using once I learned how to set up haproxy to do it automatically: https://paste.elyograg.org/view/5e88c914 (seems that I removed the final \ that made the blank lines necessary. oops!) Thanks, Shawn
Re: OCSP update restarts all proxies
On 30/09/2023 09:20, Shawn Heisey wrote: On 9/28/23 02:29, Remi Tricot-Le Breton wrote: That's really strange, the OCSP update mechanism does not have anything to do with proxies. Are you sure you did not have a crash and autorestart of your haproxy ? I did not think that I had autorestart for haproxy, but it turns out that the service file created by the systemd stuff in the source repo DOES have "Restart=always". After I changed that to never and did systemctl daemon-reload, I discovered that at the top of the hour, something caused systemd to reload the service. From systemctl status haproxy: Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_gitlab_8881 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_gitlab2_8881 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_artifactory_8082 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_zabbix_81 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : New worker (236124) forked Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : Loading success. Sep 30 01:00:02 smeagol systemd[1]: Reloaded HAProxy Load Balancer. Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : haproxy version is 2.8.3-0499db-3 Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : path to executable is /usr/local/sbin/haproxy Sep 30 01:00:02 smeagol haproxy[234279]: [WARNING] (234279) : Former worker (234282) exited with code 0 (Exit) There are no relevant systemd timers, nothing in user crontabs, nothing in the various cron.* directories that could cause this. I did compile haproxy with systemd support ... can haproxy itself ask systemd for a reload? A way to check for a possible crash in the OCSP update code would be to use the "update ssl ocsp-response " from the CLI as well. It would use most of the OCSP update code so if a crash were to happen you might see it this way. Can you explain to me how to do this and see any output? I tried piping the command to socat talking to the stats proxy socket, and got no response. I think I do not know how to use socat correctly for this. This command relies on the same task that performs the automatic update. What it does is basically add the certificate at the top of the task's update list and wakes it up. The update is asynchronous so we can't return a status to the CLI command. In order to check if the update was successful you can display the contents of the updated OCSP response via the "show ssl ocsp-response" command. If the response you updated is also set to be updated automatically, you can also use the "show ssl ocsp-updates" command that gives the update success and failure numbers as well as the last update status for all the responses registered in the auto update list. Rémi LB
Re: OCSP update restarts all proxies
On 9/28/23 02:29, Remi Tricot-Le Breton wrote: That's really strange, the OCSP update mechanism does not have anything to do with proxies. Are you sure you did not have a crash and autorestart of your haproxy ? I did not think that I had autorestart for haproxy, but it turns out that the service file created by the systemd stuff in the source repo DOES have "Restart=always". After I changed that to never and did systemctl daemon-reload, I discovered that at the top of the hour, something caused systemd to reload the service. From systemctl status haproxy: Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_gitlab_8881 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_gitlab2_8881 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_artifactory_8082 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234282]: [WARNING] (234282) : Proxy be_zabbix_81 stopped (cumulated conns: FE: 0, BE: 0). Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : New worker (236124) forked Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : Loading success. Sep 30 01:00:02 smeagol systemd[1]: Reloaded HAProxy Load Balancer. Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : haproxy version is 2.8.3-0499db-3 Sep 30 01:00:02 smeagol haproxy[234279]: [NOTICE] (234279) : path to executable is /usr/local/sbin/haproxy Sep 30 01:00:02 smeagol haproxy[234279]: [WARNING] (234279) : Former worker (234282) exited with code 0 (Exit) There are no relevant systemd timers, nothing in user crontabs, nothing in the various cron.* directories that could cause this. I did compile haproxy with systemd support ... can haproxy itself ask systemd for a reload? A way to check for a possible crash in the OCSP update code would be to use the "update ssl ocsp-response " from the CLI as well. It would use most of the OCSP update code so if a crash were to happen you might see it this way. Can you explain to me how to do this and see any output? I tried piping the command to socat talking to the stats proxy socket, and got no response. I think I do not know how to use socat correctly for this. Thanks, Shawn
Re: OCSP update restarts all proxies
Hello, On 28/09/2023 00:30, Shawn Heisey wrote: The haproxy -vv output is at the end of this message. I got the built-in OCSP udpating mechanism working. Works beautifully. Today I discovered that once an hour when the OCSP gets updated, haproxy stops all its proxies and starts them back up. syslog: That's really strange, the OCSP update mechanism does not have anything to do with proxies. Are you sure you did not have a crash and autorestart of your haproxy ? Sep 27 15:00:01 - haproxy[3520801] Proxy web80 stopped (cumulated conns: FE: 42, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy web stopped (cumulated conns: FE: 1403, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_deny stopped (cumulated conns: FE: 0, BE: 122). Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi1_81 stopped (cumulated conns: FE: 0, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi2_81 stopped (cumulated conns: FE: 0, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi3_81 stopped (cumulated conns: FE: 0, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_smeagol_81 stopped (cumulated conns: FE: 0, BE: 700). Sep 27 15:00:01 - haproxy[3520801] Proxy be_plex_32400_tls stopped (cumulated conns: FE: 0, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab_8881 stopped (cumulated conns: FE: 0, BE: 235). Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab2_8881 stopped (cumulated conns: FE: 0, BE: 180). Sep 27 15:00:01 - haproxy[3520801] Proxy be_artifactory_8082 stopped (cumulated conns: FE: 0, BE: 0). Sep 27 15:00:01 - haproxy[3520801] Proxy be_zabbix_81 stopped (cumulated conns: FE: 0, BE: 969). Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.668] /etc/ssl/certs/local/REDACTED_org.wildcards.combined .pem 1 "Update successful" 0 1 Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.795] /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p em 1 "Update successful" 0 1 Sep 27 15:00:01 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.944] /etc/ssl/certs/local/REDACTED_org.wildcards.combined .pem 1 "Update successful" 0 2 Sep 27 15:00:02 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.998] /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p em 1 "Update successful" 0 2 The really irritating effect is that once an hour, my Zabbix server records an event saying haproxy has been restarted: https://imgur.com/a/WPkKoFa (imgur will claim the image has mature content. it doesn't.) It looks like the only thing that resets back to zero on the stats page is the uptime in the "status" column for each backend. That's good news, but I would hope for none of the data to be reset. I have one big concern, which may be unfounded: I'm worried that the proxies going down will mean that in-flight connections will be terminated. I'm guessing that the work for seamless reloads will ensure that doesn't happen, I just want to be sure. Not knowing a lot about how haproxy is architected, I do not know if there is some reason that the backends have to be cycled. Seems like only frontends that listen with TLS would need that. I would hope it would be possible to even avoid that ... maybe have OCSP data be copied from a certain memory location every time a frontend needs it, and when OCSP gets updated, overwrite the data in that memory location in a thread-safe way. I know a fair amount about thread safety in Java, but nothing about it in C. That's what's supposed to happen, no listener should have to be restarted :) That's why it looks more like a crash to me. Final questions for today: 1) Can the OCSP update interval be changed? I don't recall exactly what the validity for a LetsEncrypt OCSP response is, but I know it was at least 24 hours, and I think it might have even been as long as a week. I would like to increase the interval to 8-12 hours if I can. The "tune.ssl.ocsp-update.maxdelay" and "tune.ssl.ocsp-update.mindelay" global options can change the default update interval. The idea behind the update every hour is that we can't just rely on the OCSP response's expiration time to determine when to perform update since it might be updated before its expiration date. We went for a 1 hour interval pretty arbitrarily but the mentioned options allow to adapt it to your use case. Just note that those options will concern all the OCSP updates, we do not have a per-certificate configuration mean yet. 2) There are two certs being used in my setup, and haproxy logs updates for both of them twice. I would have hoped for that to only happen once. I'm a bit mystified by the fact that it is done twice. I would have expected either one time or four times ... I have one frontend that listens with TLS, with four bind lines all using exactly the same certificate list. (one TCP, and three UDP) Not sure why that's the case yet. You could use some CLI commands to check whether the OCSP update tree is built correctly (that should be done at
