Re: OCSP update mechanism startup
On 07/07/2023 18:24, Willy Tarreau wrote: On Fri, Jul 07, 2023 at 03:42:58PM +, Tristan wrote: Also personally I have never understood the point of default server certs... besides getting unwanted attention from censys/shodan/etc... I remember some users who were hosting many applications from internal subsidiaries wanted to make sure that unhandled names were always served on the corporate site, at least to say something along "this site is no longer here" or just to direct users to the list of available sites. In the past it was common as well on hosting providers that any unhandled site was handled by a default page hosted by that provider. However it seems it's less common nowadays with SSL (though I could be wrong). Hmmm just testing right now seems to indicate it's still the case for some: $ openssl s_client -servername blah.com -connect www.cloudflare.com:443 /dev/null| grep -m1 CN 0 s:CN = ssl383624.cloudflaressl.com $ openssl s_client -servername blah.com -connect www.fastly.com:443 /dev/null | grep -m1 CN 0 s:CN = www.fastly.com So it seems to indicate that such infrastructures which are designed to host hundreds of thousands of domains still prefer to be able to deliver something if the user is willing to click on "accept the risk". Hmm, I understand why it was decided to go that route, and indeed that is probably not too too hard to do... Though I can't help wonder if an opt-in mechanism similar to the server state file would make sense for it? I seem to remember that it's possible to issue a "show ocsp" on the CLI and direct it to a file, but I could be wrong. That's the same way the server-state works by the way. I'll take a look at it, though the main advantage would be to keep it all in HAProxy and to avoid dirty cron jobs. But I appreciate that's me asking for HAProxy to do everything and more beyond its job... :) For the server state, it was performed at reload time, no need for a cron job. I don't see why it would have to be different here. It might be a bit more complicated than for the server state file since we could have OCSP responses that don't correspond to any file on the filesystem. We can't just assume that the foo.pem.ocsp file does not exist and create a file at this location since it might exist but not have been loaded by haproxy (through a ssl-load-extra-files none for instance). And updating only OCSP responses that were loaded at build time and for which we have an actual path would not be that satisfying. It would still require an extra step for new responses. We could add an extra global parameter that defines a directory in which those responses are written but reusing them for a new haproxy process would require that those responses are moved alongside their corresponding certificate, which could be a pain as well. But I fully agree that it would help to be able to perform this operation from haproxy directly. I just don't see a simple solution yet. Rémi
Re: OCSP update mechanism startup
On Fri, Jul 07, 2023 at 03:42:58PM +, Tristan wrote: > Also personally I have never understood the point of default server certs... > besides getting unwanted attention from censys/shodan/etc... I remember some users who were hosting many applications from internal subsidiaries wanted to make sure that unhandled names were always served on the corporate site, at least to say something along "this site is no longer here" or just to direct users to the list of available sites. In the past it was common as well on hosting providers that any unhandled site was handled by a default page hosted by that provider. However it seems it's less common nowadays with SSL (though I could be wrong). Hmmm just testing right now seems to indicate it's still the case for some: $ openssl s_client -servername blah.com -connect www.cloudflare.com:443 /dev/null| grep -m1 CN 0 s:CN = ssl383624.cloudflaressl.com $ openssl s_client -servername blah.com -connect www.fastly.com:443 /dev/null | grep -m1 CN 0 s:CN = www.fastly.com So it seems to indicate that such infrastructures which are designed to host hundreds of thousands of domains still prefer to be able to deliver something if the user is willing to click on "accept the risk". > > > Hmm, I understand why it was decided to go that route, and indeed that is > > > probably not too too hard to do... Though I can't help wonder if an opt-in > > > mechanism similar to the server state file would make sense for it? > > > > I seem to remember that it's possible to issue a "show ocsp" on the CLI > > and direct it to a file, but I could be wrong. That's the same way the > > server-state works by the way. > > I'll take a look at it, though the main advantage would be to keep it all in > HAProxy and to avoid dirty cron jobs. But I appreciate that's me asking for > HAProxy to do everything and more beyond its job... :) For the server state, it was performed at reload time, no need for a cron job. I don't see why it would have to be different here. Willy
Re: OCSP update mechanism startup
On 07/07/2023 16:34, Willy Tarreau wrote: On Fri, Jul 07, 2023 at 03:06:52PM +, Tristan wrote: The ocsp-update option should be between brackets /etc/haproxy/ssl/mangadex.dev.pem [ocsp-update on] mangadex.dev *.mangadex.dev Oh that makes more sense indeed; should have guessed so since other crt-list bind params used those indeed... I think that we should at least emit a warning when domain names not containing any dot are found, because IIRC these are not valid for SNI anyway thus there should be no valid reason that an option name is accepted as a server name. As an aside, I found the doc relating to preserving bind's strict-sni behaviour to be a little bit unclear. Indeed, I first tried just keeping strict-sni on the bind line, with no explicit SNIs filters specified on certificate lines, and that caused SSL failures. And it wasn't serving a default cert but rather just failing the SSL conns. My uneducated guess is that crt-list just doesn't take into account the bind line's strict-sni in such cases, or doesn't extract CNs/SANs. But the doc explicitly says that CNs/SANs are taken into account, so I assume the former. That's just my uneducated guess anyway. After a bit of trial-and-error I found out that a working combination was: 1. strict-sni on the bind line 2. explicit sni filters on cert lines in the crt-list Tho I'd bet that in this case only 2. does anything, and 1. isn't really doing anything anymore. Also personally I have never understood the point of default server certs... besides getting unwanted attention from censys/shodan/etc... Hmm, I understand why it was decided to go that route, and indeed that is probably not too too hard to do... Though I can't help wonder if an opt-in mechanism similar to the server state file would make sense for it? I seem to remember that it's possible to issue a "show ocsp" on the CLI and direct it to a file, but I could be wrong. That's the same way the server-state works by the way. I'll take a look at it, though the main advantage would be to keep it all in HAProxy and to avoid dirty cron jobs. But I appreciate that's me asking for HAProxy to do everything and more beyond its job... :) Tristan
Re: OCSP update mechanism startup
On Fri, Jul 07, 2023 at 03:06:52PM +, Tristan wrote: > > The ocsp-update option should be between brackets > > /etc/haproxy/ssl/mangadex.dev.pem [ocsp-update on] mangadex.dev > > *.mangadex.dev > > Oh that makes more sense indeed; should have guessed so since other crt-list > bind params used those indeed... I think that we should at least emit a warning when domain names not containing any dot are found, because IIRC these are not valid for SNI anyway thus there should be no valid reason that an option name is accepted as a server name. > > > - if not, what happens if upon restart/reload of HAProxy the .ocsp > > > file is outdated? will HAProxy aggressively try to get an up-to-date > > > response before starting its listeners or will I be risking ssl > > > issues by enabling OCSP must-staple with it? > > > > The OCSP update mechanism will not block anything, it runs alongside all > > the "regular" HAProxy tasks. > > If I remember correctly, you cannot load outdated OCSP responses so you > > should not face this particular problem. But if you have many > > certificates for which OCSP update was activated and no OCSP response > > was provided, fetching all the missing responses will indeed take some > > time and OCSP stapling will temporarily fail for the given server > > certificates. > > Hmm, I understand why it was decided to go that route, and indeed that is > probably not too too hard to do... Though I can't help wonder if an opt-in > mechanism similar to the server state file would make sense for it? I seem to remember that it's possible to issue a "show ocsp" on the CLI and direct it to a file, but I could be wrong. That's the same way the server-state works by the way. > Either way, it all worked flawlessly within these constraints after putting > the brackets, so that's great. Thanks. Great, thanks for the feedback! Willy
Re: OCSP update mechanism startup
The ocsp-update option should be between brackets /etc/haproxy/ssl/mangadex.dev.pem [ocsp-update on] mangadex.dev *.mangadex.dev Oh that makes more sense indeed; should have guessed so since other crt-list bind params used those indeed... - does the OCSP update mechanism update the files on-disk? No we never write anything on disk. [...] - if not, what happens if upon restart/reload of HAProxy the .ocsp file is outdated? will HAProxy aggressively try to get an up-to-date response before starting its listeners or will I be risking ssl issues by enabling OCSP must-staple with it? The OCSP update mechanism will not block anything, it runs alongside all the "regular" HAProxy tasks. If I remember correctly, you cannot load outdated OCSP responses so you should not face this particular problem. But if you have many certificates for which OCSP update was activated and no OCSP response was provided, fetching all the missing responses will indeed take some time and OCSP stapling will temporarily fail for the given server certificates. Hmm, I understand why it was decided to go that route, and indeed that is probably not too too hard to do... Though I can't help wonder if an opt-in mechanism similar to the server state file would make sense for it? Either way, it all worked flawlessly within these constraints after putting the brackets, so that's great. Thanks. Tristan
Re: OCSP update mechanism startup
Hello Tristan, On 06/07/2023 13:24, Tristan wrote: Hello, I'm trying to make use of the new ocsp-update mechanism, and finding no success (yet). I've migrated my crt bind arguments to a crt-list argument (+ relevant file) and that loads in and gets used fine, but despite having "ocsp-update on" on the lines, I'm not seeing any ocsp update being triggered (and naturally there are no ocsp responses being returned). Relevant outputs: $ echo "show ssl crt-list" | socat - /var/run/haproxy/admin.sock /etc/haproxy/ssl/crtlist-mdex-public-direct.conf ... $ echo "show ssl crt-list /etc/haproxy/ssl/crtlist-mdex-public-direct.conf" | socat - /var/run/haproxy/admin.sock /etc/haproxy/ssl/mangadex.dev.pem ocsp-update on mangadex.dev *.mangadex.dev The ocsp-update option should be between brackets /etc/haproxy/ssl/mangadex.dev.pem [ocsp-update on] mangadex.dev *.mangadex.dev $ echo "show ssl ocsp-updates" | socat - /var/run/haproxy/admin.sock OCSP Certid | Path | Next Update | Last Update | Successes | Failures | Last Update Status | Last Update Status (str) One potential hint I find is when using "update ssl ocsp-response ": $ echo "update ssl ocsp-response /etc/haproxy/ssl/mangadex.dev.pem" | socat - /var/run/haproxy/admin.sock 'update ssl ocsp-response' only works on certificates that already have a known OCSP response. Can't send ocsp request for /etc/haproxy/ssl/mangadex.dev.pem! And indeed, the (cli command) docs mention: > This command will only work for certificates that already had a stored OCSP response, either because it was provided during init or if it was previously set through the "set ssl cert" or "set ssl ocsp-response" commands So then: - does the ocsp-update feature also require an initial OCSP response to start at all, like the cli command? No, it will try to get the missing OCSP update responses right after init. - if so, why? (and we should get that documented on the ocsp-update flag) - does the OCSP update mechanism update the files on-disk? No we never write anything on disk. - if not, what happens if upon restart/reload of HAProxy the .ocsp file is outdated? will HAProxy aggressively try to get an up-to-date response before starting its listeners or will I be risking ssl issues by enabling OCSP must-staple with it? The OCSP update mechanism will not block anything, it runs alongside all the "regular" HAProxy tasks. If I remember correctly, you cannot load outdated OCSP responses so you should not face this particular problem. But if you have many certificates for which OCSP update was activated and no OCSP response was provided, fetching all the missing responses will indeed take some time and OCSP stapling will temporarily fail for the given server certificates. Please note that the 'show ssl ocsp-response' can now dump an OCSP response in base64 so that you can write the update OCSP responses to your filesystem by yourself. (Probably not super relevant, but this is with HAProxy 2.8.1 and QuicTLS 1.1.1t) Thanks in advance, Tristan Rémi
OCSP update mechanism startup
Hello, I'm trying to make use of the new ocsp-update mechanism, and finding no success (yet). I've migrated my crt bind arguments to a crt-list argument (+ relevant file) and that loads in and gets used fine, but despite having "ocsp-update on" on the lines, I'm not seeing any ocsp update being triggered (and naturally there are no ocsp responses being returned). Relevant outputs: $ echo "show ssl crt-list" | socat - /var/run/haproxy/admin.sock /etc/haproxy/ssl/crtlist-mdex-public-direct.conf ... $ echo "show ssl crt-list /etc/haproxy/ssl/crtlist-mdex-public-direct.conf" | socat - /var/run/haproxy/admin.sock /etc/haproxy/ssl/mangadex.dev.pem ocsp-update on mangadex.dev *.mangadex.dev $ echo "show ssl ocsp-updates" | socat - /var/run/haproxy/admin.sock OCSP Certid | Path | Next Update | Last Update | Successes | Failures | Last Update Status | Last Update Status (str) One potential hint I find is when using "update ssl ocsp-response ": $ echo "update ssl ocsp-response /etc/haproxy/ssl/mangadex.dev.pem" | socat - /var/run/haproxy/admin.sock 'update ssl ocsp-response' only works on certificates that already have a known OCSP response. Can't send ocsp request for /etc/haproxy/ssl/mangadex.dev.pem! And indeed, the (cli command) docs mention: > This command will only work for certificates that already had a stored OCSP response, either because it was provided during init or if it was previously set through the "set ssl cert" or "set ssl ocsp-response" commands So then: - does the ocsp-update feature also require an initial OCSP response to start at all, like the cli command? - if so, why? (and we should get that documented on the ocsp-update flag) - does the OCSP update mechanism update the files on-disk? - if not, what happens if upon restart/reload of HAProxy the .ocsp file is outdated? will HAProxy aggressively try to get an up-to-date response before starting its listeners or will I be risking ssl issues by enabling OCSP must-staple with it? (Probably not super relevant, but this is with HAProxy 2.8.1 and QuicTLS 1.1.1t) Thanks in advance, Tristan
