Re: dynamic ssl certificate updates with changed intermediate
On Fri, Dec 11, 2020 at 02:53:13PM +0100, Björn Jacke wrote: > Hi William, > > On 11.12.20 12:29, William Lallemand wrote: > > If we want the "set ssl ocsp-response" command to work in this particular > > case, > > I think we need to change the key, but the problem is that the OCSP response > > only contains an OCSP_CERTID for helping us finding where we should apply > > the > > certificate, and the serialNumber alone is not enough to index the response. > > thanks to your description I understand the technical background but I > think it's a usability issue for people running haproxy. If people > follow setting up hitless certificate updates as described at > https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/ > then it will not be clear how to set this up correct if you use ocsp > stapling also. Finding the underlying problem is not quite easy. Most > people who run haproxy with dynamic updates and ocsp stapling enabled, > will probably run into the same problem. Now that the letsencrypt is > about to issue certs with a different intermediate cert (and soon will > change again the intermediate for ecdsa certs), this problem might pop > up for more people. > In my opinion the problem is that there is no warning during the "set ssl cert" and that it allowed you to commit silently. In this particular case we should have a warning which states that the ocsp response must be updated before the commit. -- William Lallemand
Re: dynamic ssl certificate updates with changed intermediate
Hi William, On 11.12.20 12:29, William Lallemand wrote: > If we want the "set ssl ocsp-response" command to work in this particular > case, > I think we need to change the key, but the problem is that the OCSP response > only contains an OCSP_CERTID for helping us finding where we should apply the > certificate, and the serialNumber alone is not enough to index the response. thanks to your description I understand the technical background but I think it's a usability issue for people running haproxy. If people follow setting up hitless certificate updates as described at https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/ then it will not be clear how to set this up correct if you use ocsp stapling also. Finding the underlying problem is not quite easy. Most people who run haproxy with dynamic updates and ocsp stapling enabled, will probably run into the same problem. Now that the letsencrypt is about to issue certs with a different intermediate cert (and soon will change again the intermediate for ecdsa certs), this problem might pop up for more people. Björn
Re: dynamic ssl certificate updates with changed intermediate
Hello Björn,
On Thu, Dec 10, 2020 at 08:14:35PM +0100, Björn Jacke wrote:
>
> What I'm finally wondering: The need for running a "set ssl cert
> fullchain.pem.ocsp" is not intended but instead the matching ocsp
> response *should* be loaded again automatically, if a certificate (with
> or without intermediate cert changes) was replaced right? If you want I
> can file an issue to track this.
The OCSP is activated by storing the configured response in a tree
during the loading of the configuration (or the set ssl cert).
This response is indexed with a key which is a OCSP_CERTID structure,
which contains the issuer name hash and the issuer key hash.
/*- CertID ::= SEQUENCE {
* hashAlgorithmAlgorithmIdentifier,
* issuerNameHash OCTET STRING, -- Hash of Issuer's DN
* issuerKeyHash OCTET STRING, -- Hash of Issuers public key
(excluding the tag & length fields)
* serialNumber CertificateSerialNumber }
*/
struct ocsp_cert_id_st {
X509_ALGOR hashAlgorithm;
ASN1_OCTET_STRING issuerNameHash;
ASN1_OCTET_STRING issuerKeyHash;
ASN1_INTEGER serialNumber;
};
A pointer to this node is then stored in the SSL context so it can be
used by the OCSP callback.
During an update with the "set ssl ocsp-response", an OCSP_CERTID is
extracted from the new response, and a lookup is done in the tree to
replace the previous one. If we didn't find the corresponding
OCSP_CERTID in the tree the message "OCSP single response: Certificate
ID does not match any certificate or issuer" is issued. This is the
problem you have because the issuer hashes changed.
If we want the "set ssl ocsp-response" command to work in this particular case,
I think we need to change the key, but the problem is that the OCSP response
only contains an OCSP_CERTID for helping us finding where we should apply the
certificate, and the serialNumber alone is not enough to index the response.
--
William Lallemand
Re: dynamic ssl certificate updates with changed intermediate
Hello William, On 10.12.20 15:55, William Lallemand wrote: >> As far as I can see the "set ssl cert fullchain.pem.ocsp" method is >> *generally* suitable to update ocsp responses and can be used as a drop >> in replacement for the "set ssl ocsp-response" method, which is not >> working correctly in the case, where the intermediate cert changed? >> > The "set ssl cert" method generates the new SSL context the same way it > is done with a reload. So it's a little bit heavier than just updating > the OCSP response. > > If you commit the certificate without the .ocsp, it's like you > reloaded haproxy with the previous .ocsp file. thank you for the clarification! What I'm finally wondering: The need for running a "set ssl cert fullchain.pem.ocsp" is not intended but instead the matching ocsp response *should* be loaded again automatically, if a certificate (with or without intermediate cert changes) was replaced right? If you want I can file an issue to track this. Björn
Re: dynamic ssl certificate updates with changed intermediate
On Thu, Dec 10, 2020 at 03:24:39PM +0100, Björn Jacke wrote: > Hi William, > > On 09.12.20 09:27, William Lallemand wrote: > > $ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat > > server2.fullchain.pem)\n\n" | socat - /tmp/master.socket > > $ echo -e "@1 set ssl cert server1.fullchain.pem.ocsp <<\n$(base64 -w > > 1 server2.fullchain.ocsp)\n" | socat - /tmp/master.socket > > $ echo "@1 commit ssl cert server1.fullchain.pem" | socat - > > /tmp/master.socket > > > > It should activate the OCSP with the new SSL context. > > thanks, yes, using "set ssl cert fullchain.pem.ocsp" instead of "set ssl > ocsp-response ..." makes it succeed. > > As far as I can see the "set ssl cert fullchain.pem.ocsp" method is > *generally* suitable to update ocsp responses and can be used as a drop > in replacement for the "set ssl ocsp-response" method, which is not > working correctly in the case, where the intermediate cert changed? > The "set ssl cert" method generates the new SSL context the same way it is done with a reload. So it's a little bit heavier than just updating the OCSP response. If you commit the certificate without the .ocsp, it's like you reloaded haproxy with the previous .ocsp file. -- William Lallemand
Re: dynamic ssl certificate updates with changed intermediate
Hi William, On 09.12.20 09:27, William Lallemand wrote: > $ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat > server2.fullchain.pem)\n\n" | socat - /tmp/master.socket > $ echo -e "@1 set ssl cert server1.fullchain.pem.ocsp <<\n$(base64 -w > 1 server2.fullchain.ocsp)\n" | socat - /tmp/master.socket > $ echo "@1 commit ssl cert server1.fullchain.pem" | socat - > /tmp/master.socket > > It should activate the OCSP with the new SSL context. thanks, yes, using "set ssl cert fullchain.pem.ocsp" instead of "set ssl ocsp-response ..." makes it succeed. As far as I can see the "set ssl cert fullchain.pem.ocsp" method is *generally* suitable to update ocsp responses and can be used as a drop in replacement for the "set ssl ocsp-response" method, which is not working correctly in the case, where the intermediate cert changed? Thanks Björn
Re: dynamic ssl certificate updates with changed intermediate
On Tue, Dec 08, 2020 at 06:42:13PM +0100, Björn Jacke wrote:
> Hi William,
>
> On 08.12.20 15:13, William Lallemand wrote:> I then updated the
> certificate this way:
> >
> > $ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat
> > server2.fullchain.pem)\n\n" | socat - /tmp/master.socket
> > Transaction created for certificate server1.fullchain.pem!
> >
> > $ echo "@1 commit ssl cert server1.fullchain.pem" | socat -
> > /tmp/master.socket
> > Committing server1.fullchain.pem.
> > Success!
> >
> > And checked that the certificate is correctly updated:
>
> true, what fail though is the dynamic ocsp-response update after that,
> sorry for the unprecise problem description before. This happens after a
> dynamic cert update that *includes* an intermediate cert update if you
> then also try make a dynamic ocsp-response update:
>
> # echo "set ssl ocsp-response $(base64 -w 1 ${DIRNAME}/ocsp.der)" |
> socat ...
>
> OCSP single response: Certificate ID does not match any certificate or
> issuer.
>
Hello,
Okay thanks for confirming.
Could you try this method?
$ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat
server2.fullchain.pem)\n\n" | socat - /tmp/master.socket
$ echo -e "@1 set ssl cert server1.fullchain.pem.ocsp <<\n$(base64 -w
1 server2.fullchain.ocsp)\n" | socat - /tmp/master.socket
$ echo "@1 commit ssl cert server1.fullchain.pem" | socat -
/tmp/master.socket
It should activate the OCSP with the new SSL context.
--
William Lallemand
Re: dynamic ssl certificate updates with changed intermediate
Hi William,
On 08.12.20 15:13, William Lallemand wrote:> I then updated the
certificate this way:
>
> $ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat
> server2.fullchain.pem)\n\n" | socat - /tmp/master.socket
> Transaction created for certificate server1.fullchain.pem!
>
> $ echo "@1 commit ssl cert server1.fullchain.pem" | socat -
> /tmp/master.socket
> Committing server1.fullchain.pem.
> Success!
>
> And checked that the certificate is correctly updated:
true, what fail though is the dynamic ocsp-response update after that,
sorry for the unprecise problem description before. This happens after a
dynamic cert update that *includes* an intermediate cert update if you
then also try make a dynamic ocsp-response update:
# echo "set ssl ocsp-response $(base64 -w 1 ${DIRNAME}/ocsp.der)" |
socat ...
OCSP single response: Certificate ID does not match any certificate or
issuer.
Björn
Re: dynamic ssl certificate updates with changed intermediate
On Tue, Dec 08, 2020 at 11:48:41AM +0100, William Lallemand wrote:
> On Sat, Dec 05, 2020 at 02:57:03AM +0100, Björn Jacke wrote:
> > Hi,
> >
> > I ran into an issue with haproxy 2.2.6, where I'm not sure if this is
> > working as intended or not. I have a frontend, which has a ssl cert
> > configured in a combined pam file, containing the private, public and
> > intermediate certificate. The bind line looks like this:
> >
> > bind 203.0.113.1 ssl crt /certs/host.example.org/combined.pem.rsa ...
> >
> > If I renew the certificate, it works as also shown in
> >
> > https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/
> >
> > via
> >
> > echo "set ssl cert ${DIRNAME}/combined.pem.rsa" | socat ...
> >
> > Everything worked fine since quite a while ...
> >
> > until now the issuing intermediate certificate changed. I would expect
> > that above mentioned "set ssl cert combined.pem.rsa" would also update
> > the intermediate certificate - but the *previous* intermediate is still
> > being used by haproxy. I noticed this actually only because the "set ssl
> > ocsp-response" returned "Certificate ID does not match any certificate
> > or issuer". It took me quite a while to spot that the intermediate was
> > not updated.
> >
> > So the final question is, is this a bug or is the intermediate not
> > supposed to be updated along with the combined.pem but differently? A
> > reload or restart of haproxy will activate the new intermediate
> > certificate of course.
> >
>
> Looks like a bug to me, the intermediate certificate is indeed supposed
> to be updated, I'll look into this.
>
I made some tests and I can't reproduce the issue, could you check with
the CLI that the intermediate changed with "show ssl cert"
This is the test I made:
1 Root CA, 2 Intermediates, 2 server certificates made with each intermediates.
cat server1.key server1.crt intermediateCA1.crt > server1.fullchain.pem
cat server1.key server1.crt intermediateCA1.crt > server2.fullchain.pem
$ echo "@1 show ssl cert server1.fullchain.pem" | socat -
/tmp/master.socket
Filename: server1.fullchain.pem
Status: Used
Serial: 19018ED789D84428F15631EEDD946E254D3F
notBefore: Dec 8 13:30:47 2020 GMT
notAfter: Sep 4 13:30:47 2023 GMT
Subject Alternative Name:
Algorithm: RSA2048
SHA1 FingerPrint: 74BB48E0F47B89AEE68A8173774B446775CDA0A3
Subject: /C=AU/ST=Some-State/O=Foobar Server1/CN=server1.foobar.local
Issuer: /C=AU/ST=Some-State/O=Foobar Int/CN=int1.foobar.local
Chain Subject: /C=AU/ST=Some-State/O=Foobar Int/CN=int1.foobar.local
Chain Issuer: /C=AU/ST=Some-State/O=Foobar ROOT/CN=root.foobar.local
I then updated the certificate this way:
$ echo -e -n "@1 set ssl cert server1.fullchain.pem <<\n$(cat
server2.fullchain.pem)\n\n" | socat - /tmp/master.socket
Transaction created for certificate server1.fullchain.pem!
$ echo "@1 commit ssl cert server1.fullchain.pem" | socat -
/tmp/master.socket
Committing server1.fullchain.pem.
Success!
And checked that the certificate is correctly updated:
$ echo "@1 show ssl cert server1.fullchain.pem" | socat -
/tmp/master.socket
Filename: server1.fullchain.pem
Status: Used
Serial: 0808AAE72CD605D64FE5FEACA9FC8B3BA33F69E2
notBefore: Dec 8 13:33:26 2020 GMT
notAfter: Sep 4 13:33:26 2023 GMT
Subject Alternative Name:
Algorithm: RSA2048
SHA1 FingerPrint: E60B288CE48BDAEE9A234DCE16DF0A05E4C4E1BE
Subject: /C=AU/ST=Some-State/O=Foobar Server2/CN=server2.foobar.local
Issuer: /C=AU/ST=Some-State/O=Foobar Int2/CN=int2.foobar.local
Chain Subject: /C=AU/ST=Some-State/O=Foobar Int2/CN=int2.foobar.local
Chain Issuer: /C=AU/ST=Some-State/O=Foobar ROOT/CN=root.foobar.local
You can see at the end of the output that the certificate and the chain was
updated.
You can also check the chain returned by haproxy with `openssl s_client
-showcerts -connect localhost:8443 -servername server2.foobar.local`
Regards,
--
William Lallemand
Re: dynamic ssl certificate updates with changed intermediate
On Sat, Dec 05, 2020 at 02:57:03AM +0100, Björn Jacke wrote:
> Hi,
>
> I ran into an issue with haproxy 2.2.6, where I'm not sure if this is
> working as intended or not. I have a frontend, which has a ssl cert
> configured in a combined pam file, containing the private, public and
> intermediate certificate. The bind line looks like this:
>
> bind 203.0.113.1 ssl crt /certs/host.example.org/combined.pem.rsa ...
>
> If I renew the certificate, it works as also shown in
>
> https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/
>
> via
>
> echo "set ssl cert ${DIRNAME}/combined.pem.rsa" | socat ...
>
> Everything worked fine since quite a while ...
>
> until now the issuing intermediate certificate changed. I would expect
> that above mentioned "set ssl cert combined.pem.rsa" would also update
> the intermediate certificate - but the *previous* intermediate is still
> being used by haproxy. I noticed this actually only because the "set ssl
> ocsp-response" returned "Certificate ID does not match any certificate
> or issuer". It took me quite a while to spot that the intermediate was
> not updated.
>
> So the final question is, is this a bug or is the intermediate not
> supposed to be updated along with the combined.pem but differently? A
> reload or restart of haproxy will activate the new intermediate
> certificate of course.
>
Looks like a bug to me, the intermediate certificate is indeed supposed
to be updated, I'll look into this.
--
William Lallemand
dynamic ssl certificate updates with changed intermediate
Hi,
I ran into an issue with haproxy 2.2.6, where I'm not sure if this is
working as intended or not. I have a frontend, which has a ssl cert
configured in a combined pam file, containing the private, public and
intermediate certificate. The bind line looks like this:
bind 203.0.113.1 ssl crt /certs/host.example.org/combined.pem.rsa ...
If I renew the certificate, it works as also shown in
https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/
via
echo "set ssl cert ${DIRNAME}/combined.pem.rsa" | socat ...
Everything worked fine since quite a while ...
until now the issuing intermediate certificate changed. I would expect
that above mentioned "set ssl cert combined.pem.rsa" would also update
the intermediate certificate - but the *previous* intermediate is still
being used by haproxy. I noticed this actually only because the "set ssl
ocsp-response" returned "Certificate ID does not match any certificate
or issuer". It took me quite a while to spot that the intermediate was
not updated.
So the final question is, is this a bug or is the intermediate not
supposed to be updated along with the combined.pem but differently? A
reload or restart of haproxy will activate the new intermediate
certificate of course.
Thanks
Björn
