Re: OCSP renewal with 2.8
Am 05.06.2023 um 10:08 schrieb William Lallemand: As I explained in my previous mail, the option was not set on the bind lines because of architectural problems, but you could expect to have a way to do it globally in future versions. thanks a lot for this information. I will wait then to have a more simple, less verbose configuration (ideally just a single global parameter to enable ocsp renewal on bind line) The scripts I use to renew OCSP are working for years without any problems, no need to rush here ;) Gruß Matthias -- "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning." -- Rich Cook
Re: OCSP renewal with 2.8
On Sat, 3 Jun 2023 at 14:30, William Lallemand wrote: > That's what we've done in the first place, but I decided to remove it > because I was not happy with the architecture. And once you have > something like this, you have to keep the configuration compatibility > for the next versions and then you are stuck with something awful. > > My concern here, is that the ocsp-update option was never a "bind" > option, it's a feature which applies on the internal storage part, which > is not directly exposed in the configuration. So for example if you use > the same certificate on multiple bind lines, setting "ocsp-update on" on > one line and "ocsp-update off" on the other doesn't make sense. I understand, I just think that those are tradeoffs that need to be made. We could document it well, and trigger configuration warnings or alerts (depending on severity) for conflicts. Not providing bind lines configuration support to avoid conflicting configurations in a small number of cases, while not supporting the most commonly used configuration does not seem like a good tradeoff. Note that I'm not saying conflicting configuration warnings for this are trivial to implement or anything like that. I don't actually know; I'm just saying this sounds like in this case the cure may be worse than the disease. > We are well aware on the current limitations of this model, and we are > working on it, that's why it landed in the crt-list for now, but that > will evolve! Great, thank you! Lukas
Re: OCSP renewal with 2.8
Hello, On Sat, Jun 03, 2023 at 04:28:30PM -0600, Shawn Heisey wrote: > On 6/3/23 15:37, Shawn Heisey wrote: > > On 6/3/23 15:28, Shawn Heisey wrote: > >> So maybe a completely separate global option makes sense. The > >> crt-list requirement is not really a burden for me, but for someone > >> who uses a LOT of certificates that change frequently, it probably > >> would become a burden. > > > > Unless it is possible to have a directory as an entry in the crt-list > > file like it is for the crt option. The crt-list doc does not say that > > this is possible, and I have not tested it. > > Using a directory as an entry in the crt-list file causes `haproxy -c > -f` to hang. Which I think means that crt-list doesn't support directories. > Hm that's weird, that shouldn't load at all! I'll take a look. > How hard would it be to add that support? I would hope that most of the > code needed is already present in the part that parses crt options. > We won't support this, directories are represented internally as a crt-list, this would add a huge complexity in the code. crt-list were made as an cleaner alternative to directories so you could specify each file you need manually and set options to them. As I explained in my previous mail, the option was not set on the bind lines because of architectural problems, but you could expect to have a way to do it globally in future versions. -- William Lallemand
Re: OCSP renewal with 2.8
On 6/3/23 15:37, Shawn Heisey wrote: On 6/3/23 15:28, Shawn Heisey wrote: So maybe a completely separate global option makes sense. The crt-list requirement is not really a burden for me, but for someone who uses a LOT of certificates that change frequently, it probably would become a burden. Unless it is possible to have a directory as an entry in the crt-list file like it is for the crt option. The crt-list doc does not say that this is possible, and I have not tested it. Using a directory as an entry in the crt-list file causes `haproxy -c -f` to hang. Which I think means that crt-list doesn't support directories. How hard would it be to add that support? I would hope that most of the code needed is already present in the part that parses crt options. Thanks, Shawn
Re: OCSP renewal with 2.8
On 6/3/23 15:28, Shawn Heisey wrote: So maybe a completely separate global option makes sense. The crt-list requirement is not really a burden for me, but for someone who uses a LOT of certificates that change frequently, it probably would become a burden. Unless it is possible to have a directory as an entry in the crt-list file like it is for the crt option. The crt-list doc does not say that this is possible, and I have not tested it. Thanks, Shawn
Re: OCSP renewal with 2.8
On 6/2/23 14:42, Lukas Tribus wrote:
I suggest we make it configurable on the bind line like other ssl
options, so it will work for the common use cases that don't involve
crt-lists, like a simple crt statement pointing to a certificate or a
directory.
It could also be a global option *as well*, but imho it does need to
be a bind line configuration option, just like strict-sni, alpn and
ciphers, so we can enable it specifically (per frontend, per bind
line) without requiring crt-list.
One of the places I tried to add it (which of course did not work) was
ssl-default-bind-options.
It might make sense to have it configurable there. Though that would
imply of course that it is also an option on each bind line, which was
the other place I tried to configure it.
So maybe a completely separate global option makes sense. The crt-list
requirement is not really a burden for me, but for someone who uses a
LOT of certificates that change frequently, it probably would become a
burden.
A question arises on where to log failures in getting OCSP data. I have
haproxy using two different syslog targets, but the way this config
evolved is lost to time.
TL;DR:
In global, I have:
log 127.0.0.1 len 65535 format rfc5424 local0
log 127.0.0.1 len 65535 format rfc5424 local1 notice
tune.http.logurilen 49152
In defaults I have:
log global
option httplog
option dontlognull
In each backend, I have:
no log
log 127.0.0.1 len 65535 format rfc5424 local0 notice err
In /etc/rsyslog.d/99-haproxy.conf I have:
local0.info /var/log/debug-haproxy
local1.*/var/log/haproxy
In /etc/rsyslog.d/0001-remote.conf I have:
module(load="imudp")
input(type="imudp" port="514")
$MaxMessageSize 64k
$template BindLog,"/var/log/rsyslog/bind/log"
$template CudoLog,"/var/log/rsyslog/cudo/log"
$template UFWLog,"/var/log/rsyslog/ufw/log"
$template RemoteLogs,"/var/log/rsyslog/%HOSTNAME%/log"
$template RemoteHostFileFormat,"%TIMESTAMP% %fromhost%
%syslogfacility-text%
%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
if $msg contains 'UFW' then {
*.* -?UFWLog;RemoteHostFileFormat
stop
}
if $syslogtag contains 'cudo' then {
*.* -?CudoLog;RemoteHostFileFormat
stop
}
if $syslogtag contains 'named' then {
*.* -?BindLog;RemoteHostFileFormat
stop
}
if $inputname == 'imudp' then {
if $fromhost-ip != '127.0.0.1' then {
if $fromhost != '-' then {
*.* -?RemoteLogs;RemoteHostFileFormat
stop
}
}
}
The effective result of all this is that all log messages are logged to
/var/log/debug-haproxy and anything more severe than a request is also
logged to /var/log/haproxy. This makes it so that I do not need to wade
through megabytes of request logs to see other problems, though I do
have the option of seeing the problem inline with requests in the other
logfile.
I came up with this config back in the 1.4 to 1.5 days, and I cannot
remember how it evolved. There was some valid reason why I needed to do
the "no log" followed by "log" in the backend, but I cannot remember
what that reason was.
Thanks,
Shawn
Re: OCSP renewal with 2.8
> On 2023-06-02 (Fr.) 22:42, Lukas Tribus wrote: > > I suggest we make it configurable on the bind line like other ssl > > options, so it will work for the common use cases that don't involve > > crt-lists, like a simple crt statement pointing to a certificate or a > > directory. > > That's what we've done in the first place, but I decided to remove it because I was not happy with the architecture. And once you have something like this, you have to keep the configuration compatibility for the next versions and then you are stuck with something awful. My concern here, is that the ocsp-update option was never a "bind" option, it's a feature which applies on the internal storage part, which is not directly exposed in the configuration. So for example if you use the same certificate on multiple bind lines, setting "ocsp-update on" on one line and "ocsp-update off" on the other doesn't make sense. It is the same problem we have with the "key" option, which applies on the storage. We are well aware on the current limitations of this model, and we are working on it, that's why it landed in the crt-list for now, but that will evolve! > > It could also be a global option *as well*, but imho it does need to > > be a bind line configuration option, just like strict-sni, alpn and > > ciphers, so we can enable it specifically (per frontend, per bind > > line) without requiring crt-list. Sure, that what considered for the evolution of the feature ! -- William Lallemand
Re: OCSP renewal with 2.8
On Sat, Jun 03, 2023 at 01:50:48PM +0200, William Lallemand wrote: > On Thu, Jun 01, 2023 at 11:42:34PM +0200, Willy Tarreau wrote: > > So this means that the doc is still not clear enough and we need to > > improve this. And indeed, I'm myself confused because William told me > > a few days ago that "ocsp-update" was for crt-list lines only and it's > > found in the "bind line options" section. And of course, when there are > > examples, they're not the ones you're looking for, that's classical! > > Yep, that's exactly what I said to you, that it's in the documentation > of the bind lines because we don't have specific crt-list > documentation, and it's specified in the documentation that it's only > for crt-list. I see now. What makes this very confusing is that "crt-list" is described at the same level, and that the first paragraph of "bind options" does not mention crt-list. At least (for stable versions) we should mention in this first paragraph something like: Most keywords here are valid on "bind" lines. Some are only valid when SSL is compiled in, and among them a few are also usable in crt-list files, and a few are only valid in crt-list files. In any case if the scope differs from "bind" line only, it will be explicitly mentioned in the keyword description. A quick check at the various keywords to make sure it's always mentioned when they're OK for crt-list would help, and such a small change could be backported to stable versions without impact. Thanks, Willy
Re: OCSP renewal with 2.8
On Fri, Jun 02, 2023 at 09:55:25PM +0200, Willy Tarreau wrote: > On Fri, Jun 02, 2023 at 01:29:31PM +0300, Matthias Fechner wrote: > > Am 02.06.2023 um 04:13 schrieb Shawn Heisey: > > > @Matthias I have no idea whether crt-list can load all certs in a > > > directory like crt can. If it can't, then you will probably need a > > > script for starting/restarting haproxy that generates the cert list > > > file. If you wantthat script to be automatically run whenever someone > > > does `systemctl restart haproxy`, you could use the ExecStartPre and > > > ExecReloadPre options in a systemd service file to run your script. > > > > > > My certificate files contain the server cert, the issuer cert, the > > > private key, and DH PARAMETERS that are unique to that cert. > > > > maybe adding a global configuration parameter to enable ocsp retrieval for > > all certificates? > > Initially during the design phase we thought about having 3 states: > "off", "on", "auto", with the last one only enabling updates for certs > that already had a .ocsp file. But along discussions with some users > we were told that it was not going to be that convenient (I don't > remember why, but I think that Rémi and/or William probably remember > the reason), and it ended up dropping "auto". > Because at the time it was an option not to activate the ocsp-update, but to activate the ocsp, so it could be activated by letting a .ocsp file, or by doing it automatically by retrieving it, but that was super confusing so we decided to just have an on/off option to activate it. > Alternately maybe instead of enabling for all certs, what would be > useful would be to just change the default, because if you have 100k > certs, it's likely that 99.9k work one way and the other ones the other > way, and what you want is to indicate the default and only mention the > exception for those concerned. Indeed that could be a way to do it. -- William Lallemand
Re: OCSP renewal with 2.8
On Thu, Jun 01, 2023 at 11:42:34PM +0200, Willy Tarreau wrote: > So this means that the doc is still not clear enough and we need to > improve this. And indeed, I'm myself confused because William told me > a few days ago that "ocsp-update" was for crt-list lines only and it's > found in the "bind line options" section. And of course, when there are > examples, they're not the ones you're looking for, that's classical! Yep, that's exactly what I said to you, that it's in the documentation of the bind lines because we don't have specific crt-list documentation, and it's specified in the documentation that it's only for crt-list. -- William Lallemand
Re: OCSP renewal with 2.8
Hi. On 2023-06-02 (Fr.) 22:42, Lukas Tribus wrote: On Fri, 2 Jun 2023 at 21:55, Willy Tarreau wrote: Initially during the design phase we thought about having 3 states: "off", "on", "auto", with the last one only enabling updates for certs that already had a .ocsp file. But along discussions with some users we were told that it was not going to be that convenient (I don't remember why, but I think that Rémi and/or William probably remember the reason), and it ended up dropping "auto". Alternately maybe instead of enabling for all certs, what would be useful would be to just change the default, because if you have 100k certs, it's likely that 99.9k work one way and the other ones the other way, and what you want is to indicate the default and only mention the exception for those concerned. I suggest we make it configurable on the bind line like other ssl options, so it will work for the common use cases that don't involve crt-lists, like a simple crt statement pointing to a certificate or a directory. It could also be a global option *as well*, but imho it does need to be a bind line configuration option, just like strict-sni, alpn and ciphers, so we can enable it specifically (per frontend, per bind line) without requiring crt-list. +1 to this suggestion. Lukas
Re: OCSP renewal with 2.8
On Fri, 2 Jun 2023 at 21:55, Willy Tarreau wrote: > Initially during the design phase we thought about having 3 states: > "off", "on", "auto", with the last one only enabling updates for certs > that already had a .ocsp file. But along discussions with some users > we were told that it was not going to be that convenient (I don't > remember why, but I think that Rémi and/or William probably remember > the reason), and it ended up dropping "auto". > > Alternately maybe instead of enabling for all certs, what would be > useful would be to just change the default, because if you have 100k > certs, it's likely that 99.9k work one way and the other ones the other > way, and what you want is to indicate the default and only mention the > exception for those concerned. I suggest we make it configurable on the bind line like other ssl options, so it will work for the common use cases that don't involve crt-lists, like a simple crt statement pointing to a certificate or a directory. It could also be a global option *as well*, but imho it does need to be a bind line configuration option, just like strict-sni, alpn and ciphers, so we can enable it specifically (per frontend, per bind line) without requiring crt-list. Lukas
Re: OCSP renewal with 2.8
On Fri, Jun 02, 2023 at 01:29:31PM +0300, Matthias Fechner wrote: > Am 02.06.2023 um 04:13 schrieb Shawn Heisey: > > @Matthias I have no idea whether crt-list can load all certs in a > > directory like crt can. If it can't, then you will probably need a > > script for starting/restarting haproxy that generates the cert list > > file. If you wantthat script to be automatically run whenever someone > > does `systemctl restart haproxy`, you could use the ExecStartPre and > > ExecReloadPre options in a systemd service file to run your script. > > > > My certificate files contain the server cert, the issuer cert, the > > private key, and DH PARAMETERS that are unique to that cert. > > maybe adding a global configuration parameter to enable ocsp retrieval for > all certificates? Initially during the design phase we thought about having 3 states: "off", "on", "auto", with the last one only enabling updates for certs that already had a .ocsp file. But along discussions with some users we were told that it was not going to be that convenient (I don't remember why, but I think that Rémi and/or William probably remember the reason), and it ended up dropping "auto". Alternately maybe instead of enabling for all certs, what would be useful would be to just change the default, because if you have 100k certs, it's likely that 99.9k work one way and the other ones the other way, and what you want is to indicate the default and only mention the exception for those concerned. Willy
Re: OCSP renewal with 2.8
Am 02.06.2023 um 04:13 schrieb Shawn Heisey: @Matthias I have no idea whether crt-list can load all certs in a directory like crt can. If it can't, then you will probably need a script for starting/restarting haproxy that generates the cert list file. If you wantthat script to be automatically run whenever someone does `systemctl restart haproxy`, you could use the ExecStartPre and ExecReloadPre options in a systemd service file to run your script. My certificate files contain the server cert, the issuer cert, the private key, and DH PARAMETERS that are unique to that cert. maybe adding a global configuration parameter to enable ocsp retrieval for all certificates? Adding an additional script does not make sense to me. If this would be required a keep with the current setup that proved to work fine. Gruß Matthias -- "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning." -- Rich Cook
Re: OCSP renewal with 2.8
On 6/1/23 16:19, Shawn Heisey wrote: I asked ChatGPT for help, and with that info, I was able to work out what to do. - elyograg@smeagol:/etc/haproxy$ cat crt-list.txt /etc/ssl/certs/local/REDACTED1.combined.pem [ocsp-update on] /etc/ssl/certs/local/REDACTED2.combined.pem [ocsp-update on] - Instead of two "crt" options, I now have "crt-list /etc/haproxy/crt-list.txt" on each bind line. Haproxy handles getting and updating the OCSP response for stapling. It's beautiful. @Matthias I have no idea whether crt-list can load all certs in a directory like crt can. If it can't, then you will probably need a script for starting/restarting haproxy that generates the cert list file. If you wantthat script to be automatically run whenever someone does `systemctl restart haproxy`, you could use the ExecStartPre and ExecReloadPre options in a systemd service file to run your script. My certificate files contain the server cert, the issuer cert, the private key, and DH PARAMETERS that are unique to that cert. Thanks, Shawn
Re: OCSP renewal with 2.8
On 6/1/23 15:42, Willy Tarreau wrote: So this means that the doc is still not clear enough and we need to improve this. And indeed, I'm myself confused because William told me a few days ago that "ocsp-update" was for crt-list lines only and it's found in the "bind line options" section. And of course, when there are examples, they're not the ones you're looking for, that's classical! I looked at the 2.8.0 documentation for crt-list and it was not very clear what to actually put in the config to use it. I asked ChatGPT for help, and with that info, I was able to work out what to do. - elyograg@smeagol:/etc/haproxy$ cat crt-list.txt /etc/ssl/certs/local/REDACTED1.combined.pem [ocsp-update on] /etc/ssl/certs/local/REDACTED2.combined.pem [ocsp-update on] - I commented the crontab entry that was handling ocsp renewal, deleted the *.ocsp files from the certificate location, restarted haproxy, and did a fresh Qualys SSL test. That test indicated that it is still stapling OCSP. Awesome new feature! Thanks, Shawn
Re: OCSP renewal with 2.8
On Thu, Jun 01, 2023 at 03:30:36PM -0600, Shawn Heisey wrote: > On 5/31/23 23:25, Matthias Fechner wrote: > > I just saw in the release notes for 2.8 that an automatic OCSP renewal > > is now included and I would like to get rid of my manual scripts that > > are currently injecting the OCSP information. > > > > I checked a little bit the documentation here: > > https://docs.haproxy.org/2.8/configuration.html#ocsp-update > > https://docs.haproxy.org/2.8/configuration.html#5.1-crt-list > > I can't figure out where to put the option. I've tried several different > places and the config check fails every time. So this means that the doc is still not clear enough and we need to improve this. And indeed, I'm myself confused because William told me a few days ago that "ocsp-update" was for crt-list lines only and it's found in the "bind line options" section. And of course, when there are examples, they're not the ones you're looking for, that's classical! So normally in order to enable ocsp updates you have to use "crt-list" instead of "crt", and reference your certificate(s) there, and in front of each of them you can place some options. One of them is "ocsp-update on", which will enable automatic renewal. > Upgraded from dev13 to 2.8.0 and that didn't help. > > It will be very cool for haproxy to handle ocsp renewal itself so I can > retire my script. That's exactly the goal, that for those whose load balancer has direct access to the internet (i.e. not on an internal network nor in an inbound DMZ), the renewal can be done automatically. > The doc said that it would need the issuer cert, which is included in the > file referenced by the crt option. Is that enough? At this point I don't know :-/ William and/or Rémi will know better, let's wait for their complementary info if you can't sort it out with the info above. I intend to enable them soon on haproxy.org so I'm interested in configuring it as well :-) Regards, Willy
Re: OCSP renewal with 2.8
On 5/31/23 23:25, Matthias Fechner wrote: I just saw in the release notes for 2.8 that an automatic OCSP renewal is now included and I would like to get rid of my manual scripts that are currently injecting the OCSP information. I checked a little bit the documentation here: https://docs.haproxy.org/2.8/configuration.html#ocsp-update https://docs.haproxy.org/2.8/configuration.html#5.1-crt-list I can't figure out where to put the option. I've tried several different places and the config check fails every time. Upgraded from dev13 to 2.8.0 and that didn't help. It will be very cool for haproxy to handle ocsp renewal itself so I can retire my script. The doc said that it would need the issuer cert, which is included in the file referenced by the crt option. Is that enough? Thanks, Shawn
OCSP renewal with 2.8
Dear all, I just saw in the release notes for 2.8 that an automatic OCSP renewal is now included and I would like to get rid of my manual scripts that are currently injecting the OCSP information. I checked a little bit the documentation here: https://docs.haproxy.org/2.8/configuration.html#ocsp-update https://docs.haproxy.org/2.8/configuration.html#5.1-crt-list and if I understood it correctly it only works if used together with a crt-list line. I currently use the crt definition on a bind line like: frontend www-https mode tcp option tcplog bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs/ alpn h2,http/1.1 bind :::443 ssl crt /usr/local/etc/haproxy/certs/ alpn h2,http/1.1 Could you please help me, how I need to configure haproxy to use ocsp renewal. It is not my intent to list all certificates in the haproxy configuration as that would make it unnecessarily complicated. Thanks a lot. Gruß Matthias -- "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning." -- Rich Cook
