Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

> You don't have to explain the code to me. :-)

^^. Ok. I was also trying to explain what I understood so you can correct
me if I'm wrong.

Everything is Cristal clear now.

Thanks a million.

On Sun, 3 Oct 2021, 22:25 Viktor Dukhovni, 
wrote:

> On Sun, Oct 03, 2021 at 09:33:29PM +0200, Alex Robuchon wrote:
>
> > > Not quite, a candidate chain is constructed from whatever certificates
> the
> > > peer (server in your case) provided, and then passed to the callback
> with
> > > "preverify_ok" set to false (for the top certificate), because the
> chain is
> > > not trusted.
> >
> > This confuses me a little bit because I thought the callback function set
> > with *SSL_set_verify* would have the "trusted first" valid chain.
>
> But there is no trust store configured, so trust first is a no-op.
> The constructed chain consists entirely of certificates from the
> peer, and last certificate in the chain (the ultimate issuing CA)
> is still not trusted, so preverify_ok = 0.
>
>
> > Are you sure you are not speaking as if eventmachine was using the
> > *SSL_CTX_set_cert_verify_callback* ?
>
> I rewrote the OpenSSL chain construction code for OpenSSL 1.1.0.  Yes, I
> am sure.
>
> > - *build_chain* that will apply the trusted first algorithm and replace
> the
> > certificate chain passed by the server with the valid one ( if found ).
>
> The code in eventmachine does not configure any trusted certificates for
> the SSL_CTX handles used to create the SSL connections.  So
> trusted-first is a no-op.
>
> > - *internal_verify *which now has the new chain and will call
> *verify_callback
> > *( the callback function passed to SSL_set_verify ) for every certificate
> > in this new chain in reverse order.
>
> You don't have to explain the code to me. :-)
>
> > > But given all the evidence before me, I'd want to delete that code and
> > > never see it again.
> >
> > I hear you :).
>
> That code is beyond repair, it needs to be thrown out, and replaced with
> ground up by the book TLS connection establishment.
>
> #1. No compiled in private keys
>
> #2. Configure the SSL_CTX with the desired list of trust anchors
>
> #3. Reuse the SSL_CTX for multiple connections that share the same
> trust and other general settings
>
> #4. Prior to SSL_connect(), tell the SSL library what hostname you want
> to verify via SSL_set1_host(3).  Let OpenSSL do all the heavy
> lifting of certificate and name checks.
>
> #5. DO NOT attempt to override certificate chain verifiction in the
> verify callback.  Use the verify callback only for logging or
> similar diagnostic purposes.
>
> #6. Return "preverify_ok" unmodified, unless your application is doing
> opportunistic unauthenticated TLS, or wants to complete the
> handshake even on verification failure, and then gracefully tear
> down the application-layer session with the peer (after checking the
> verification status at connection completion).
>
> If preverify_ok is false (0), OpenSSL chain verification failed,
> your application is unlikely to do better.  Return 0, and the
> TLS handshake will be aborted, you can log the error reason,
> chain depth, ... in the verify callback.
>
> --
> Viktor.
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sun, Oct 03, 2021 at 09:33:29PM +0200, Alex Robuchon wrote:

> > Not quite, a candidate chain is constructed from whatever certificates the
> > peer (server in your case) provided, and then passed to the callback with
> > "preverify_ok" set to false (for the top certificate), because the chain is
> > not trusted.
> 
> This confuses me a little bit because I thought the callback function set
> with *SSL_set_verify* would have the "trusted first" valid chain.

But there is no trust store configured, so trust first is a no-op.
The constructed chain consists entirely of certificates from the
peer, and last certificate in the chain (the ultimate issuing CA)
is still not trusted, so preverify_ok = 0.


> Are you sure you are not speaking as if eventmachine was using the
> *SSL_CTX_set_cert_verify_callback* ?

I rewrote the OpenSSL chain construction code for OpenSSL 1.1.0.  Yes, I
am sure.

> - *build_chain* that will apply the trusted first algorithm and replace the
> certificate chain passed by the server with the valid one ( if found ).

The code in eventmachine does not configure any trusted certificates for
the SSL_CTX handles used to create the SSL connections.  So
trusted-first is a no-op.

> - *internal_verify *which now has the new chain and will call *verify_callback
> *( the callback function passed to SSL_set_verify ) for every certificate
> in this new chain in reverse order.

You don't have to explain the code to me. :-)

> > But given all the evidence before me, I'd want to delete that code and
> > never see it again.
> 
> I hear you :).

That code is beyond repair, it needs to be thrown out, and replaced with
ground up by the book TLS connection establishment.

#1. No compiled in private keys

#2. Configure the SSL_CTX with the desired list of trust anchors

#3. Reuse the SSL_CTX for multiple connections that share the same
trust and other general settings

#4. Prior to SSL_connect(), tell the SSL library what hostname you want
to verify via SSL_set1_host(3).  Let OpenSSL do all the heavy
lifting of certificate and name checks.

#5. DO NOT attempt to override certificate chain verifiction in the
verify callback.  Use the verify callback only for logging or
similar diagnostic purposes.

#6. Return "preverify_ok" unmodified, unless your application is doing
opportunistic unauthenticated TLS, or wants to complete the
handshake even on verification failure, and then gracefully tear
down the application-layer session with the peer (after checking the
verification status at connection completion).

If preverify_ok is false (0), OpenSSL chain verification failed,
your application is unlikely to do better.  Return 0, and the
TLS handshake will be aborted, you can log the error reason,
chain depth, ... in the verify callback.

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

>
> Not quite, a candidate chain is constructed from whatever certificates the
> peer (server in your case) provided, and then passed to the callback with
> "preverify_ok" set to false (for the top certificate), because the chain is
> not trusted.
>

This confuses me a little bit because I thought the callback function set
with *SSL_set_verify* would have the "trusted first" valid chain. Are you
sure you are not speaking as if eventmachine was using the
*SSL_CTX_set_cert_verify_callback* ?

>From what I see from the source code ( master )

The *tls_post_process_server_certificate* function calls
ssl/ssl_cert.c#ssl_verify_cert_chain


if (s->ctx->app_verify_callback != NULL)
i = s->ctx->app_verify_callback(ctx, s->ctx->app_verify_arg);
else
i = X509_verify_cert(ctx);
return i

Since eventmachine do not set the *app_verify_callback* so the default
*X509_verify_cert* is used from crypto/x509/x509_vfy.c (L261)


The function *X509_verify_cert* calls *verify_chain(ctx)* line 295 which in
turns calls:
- *build_chain* that will apply the trusted first algorithm and replace the
certificate chain passed by the server with the valid one ( if found ). At
the point the ctx has the new chain
- *internal_verify *which now has the new chain and will call *verify_callback
*( the callback function passed to SSL_set_verify ) for every certificate
in this new chain in reverse order.

During the build_chain process I notice the ctx->get_issuer ( which
actually points to X509_STORE_CTX_get1_issuer which I suppose return 0
because eventmachine do not properly set the store )


But the evenmachine callback ignores "preverify_ok" and goes through the
> motions of doing some sort of verification of each certificate.
>

yes indeed

But given all the evidence before me, I'd want to delete that code and
> never see it again.
>

I hear you :).

On Sun, Oct 3, 2021 at 6:48 PM Viktor Dukhovni 
wrote:

> > On 3 Oct 2021, at 12:33 pm, Alex Robuchon 
> wrote:
> >
> > So I suppose openssl skip the part that is supposed to build the chain
> when no store is configured.
>
> Not quite, a candidate chain is constructed from whatever certificates
> the peer (server in your case) provided, and then passed to the callback
> with "preverify_ok" set to false (for the top certificate), because the
> chain is not trusted.
>
> But the evenmachine callback ignores "preverify_ok" and goes through the
> motions of doing some sort of verification of each certificate.
>
> Ultimately, it will attempt to "verify" the leaf certificate, and if it
> can somehow do a fair job of that (by building its own chain, checking
> all the signatures, doing the name checks (for a name that does not
> appear to be passed to the verification function), then in theory
> the checks at depths above 0 are just silly opportunities to fail and
> the EE cert check would be enough.
>
> But given all the evidence before me, I'd want to delete that code and
> never see it again.
>
> --
> Viktor.
>
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

> On 3 Oct 2021, at 12:33 pm, Alex Robuchon  
> wrote:
> 
> So I suppose openssl skip the part that is supposed to build the chain when 
> no store is configured.

Not quite, a candidate chain is constructed from whatever certificates
the peer (server in your case) provided, and then passed to the callback
with "preverify_ok" set to false (for the top certificate), because the
chain is not trusted.

But the evenmachine callback ignores "preverify_ok" and goes through the
motions of doing some sort of verification of each certificate.

Ultimately, it will attempt to "verify" the leaf certificate, and if it
can somehow do a fair job of that (by building its own chain, checking
all the signatures, doing the name checks (for a name that does not
appear to be passed to the verification function), then in theory
the checks at depths above 0 are just silly opportunities to fail and
the EE cert check would be enough.

But given all the evidence before me, I'd want to delete that code and
never see it again.

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

I just had a big laugh Reading this. I never had to dig into openssl before
so I unfortunately had to try to understand your API with eventmachine as
an example. Silly me.


> On the other hand I suppose if we do not call this it would pick the
> "default" trust store from the system which seems to be the case here
> because it can find /usr/lib/ssl/certs/2e5ac55d.0 .
>

Actually the part responsible for this lookup is not part of the openssl
library but from the ruby callback function registered through
SSL_set_verify  which as you said tries to verify each certificate in
isolation ( with a configured store this time )

So I suppose openssl skip the part that is supposed to build the chain when
no store is configured.

Do whatever it takes to never rely on this code again.  Even abandon Ruby
> if that's what it takes...


 Hopefully we do not have a lot of code relying on eventmachine so we'll
see if we can improve the library or move away from it.

Thanks so much for your time, explanation and responsiveness.


On Sun, 3 Oct 2021, 17:37 Viktor Dukhovni, 
wrote:

> On Sun, Oct 03, 2021 at 01:54:44PM +0200, Alex Robuchon wrote:
>
> > Thanks for the detailed answer.
> >
> > From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
> >
> > When I use the eventmachine lib that uses the wrong cert chain I can see
> > with strace :
>
> Run as far away from eventmachine as you can.  The very first thing you
> see in the code is a compiled in default "private key" (for all the
> world to share).
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L30-L123
>
> Though applications can override these, and supply actual private keys,
> it does not get better from there...  The comments in the code show the
> author punting on understanding the OpenSSL API and just guessing what
> to do.
>
> Indeed the code creates SSL_CTX objects without specifying either the
> default or explicit trust stores.
>
> The real disaster is in:
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L675-L698
>
> with a completely broken SSL verification callback, that completely
> ignores all errors from the chain construction and signature
> verification code, and just attempts to "verify" each certificate in
> *isolation*.
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L693-L697
>
> This means:
>
> * No verification of chain signatures
> * No verification of path constraints
> * No verification of name constraints
> * No hostname checks.
> * ...
>
> This code was written by someone too clueless to know what they're
> doing and too lazy to bother to learn.  DO NOT USE IT.
>
> Do whatever it takes to never rely on this code again.  Even abandon
> Ruby if that's what it takes...
>
> --
> Viktor.
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sun, Oct 03, 2021 at 01:54:44PM +0200, Alex Robuchon wrote:

> Thanks for the detailed answer.
> 
> From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
> 
> When I use the eventmachine lib that uses the wrong cert chain I can see
> with strace :

Run as far away from eventmachine as you can.  The very first thing you
see in the code is a compiled in default "private key" (for all the
world to share).


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L30-L123

Though applications can override these, and supply actual private keys,
it does not get better from there...  The comments in the code show the
author punting on understanding the OpenSSL API and just guessing what
to do.

Indeed the code creates SSL_CTX objects without specifying either the
default or explicit trust stores.

The real disaster is in:


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L675-L698

with a completely broken SSL verification callback, that completely
ignores all errors from the chain construction and signature
verification code, and just attempts to "verify" each certificate in
*isolation*.


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L693-L697

This means:

* No verification of chain signatures
* No verification of path constraints
* No verification of name constraints
* No hostname checks.
* ...

This code was written by someone too clueless to know what they're
doing and too lazy to bother to learn.  DO NOT USE IT.

Do whatever it takes to never rely on this code again.  Even abandon
Ruby if that's what it takes...

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

Thanks for the detailed answer.

>From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1

When I use the eventmachine lib that uses the wrong cert chain I can see
with strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/2e5ac55d.0", {st_mode=S_IFREG|0644, st_size=1200,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/2e5ac55d.0", O_RDONLY) = 8
stat("/usr/lib/ssl/certs/2e5ac55d.1", 0x7ffe1a8e5d80) = -1 ENOENT (No such
file or directory)

When I use another lib that uses the correct cert chain I can see with
strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/8d33f237.0", 0x7fff1b4b0f90) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/4042bcee.0", {st_mode=S_IFREG|0644, st_size=1939,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/4042bcee.0", O_RDONLY) = 8

In the second case I can see it tries to load the R3 certificate
( 8d33f237.0 ). I wonder why in the first case it does not try to find each
certificate in the chain from the trust store at all. I wonder if it can be
related to the fact I do not see any call to SSL_CTX_set_cert_store in the
lib.  On the other hand I suppose if we do not call this it would pick the
"default" trust store from the system which seems to be the case here
because it can find /usr/lib/ssl/certs/2e5ac55d.0 .

This looks like to be an issue in the eventmachine lib itself. I need to
brush up my C skills and have a deeper look at this :).

Thanks

On Sat, Oct 2, 2021 at 8:11 PM Viktor Dukhovni 
wrote:

> On Sat, Oct 02, 2021 at 06:21:00PM +0100, Angus Robertson - Magenta
> Systems Ltd wrote:
>
> > > Yes.  To make things even more complex, a few sites also have an
> > > older version of R3 that is directly signed by the DST root:
> > >
> > >  - leaf <- R3 <- DST Root CA X3 (self-signed)
> > >
> > > but that's far from common at this point.
> >
> > That old R3 [CA] was issued last winter and got installed in Windows
> > Server 2018 intermediate stores then, and was still being sent out on
> > 29th and 30th, despite expiring on 29th.
>
> Not just Windows, at least one Unix system running Postfix is still
> vending a chain with the R3 signed by DST that expired on 2021-09-29:
>
> issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
> subject=C = US, O = Let's Encrypt, CN = R3
> notBefore=Oct  7 19:21:40 2020 GMT
> notAfter=Sep 29 19:21:40 2021 GMT
> SHA256
> Fingerprint=73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
> -BEGIN CERTIFICATE-
> MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
> MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
> DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
> MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
> AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
> jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
> Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
> U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
> gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
> /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
> oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
> BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
> ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
> p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
> AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
> Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
> LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
> r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
> AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
> ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
> S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
> qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
> O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
> UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
> -END CERTIFICATE-
>
> The EE (depth 0 server) certificate is not expired, and yet somehow the
> server is building a chain with a fresh leaf cert, and a rather stale
> issuer CA.
>
> It verifies via the DANE implementation in OpenSSL, because its "2 1 1"
> record with a fresh RRSIG specifies the R3 CA as trusted, and its
> expiration date is not in scope since it was signed by an entity outside
> the effective trust chain.
>
> Validation would fail for the same chain with WebPKI, unless there's a
> new improved R3 in the trust store (not just the roots).
>
> My DANE survey scan engine checks trust-anchor cert expiration date,
> even when an intermediate CA, 

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sat, Oct 02, 2021 at 06:21:00PM +0100, Angus Robertson - Magenta Systems Ltd 
wrote:

> > Yes.  To make things even more complex, a few sites also have an 
> > older version of R3 that is directly signed by the DST root:
> > 
> >  - leaf <- R3 <- DST Root CA X3 (self-signed)
> > 
> > but that's far from common at this point.
> 
> That old R3 [CA] was issued last winter and got installed in Windows
> Server 2018 intermediate stores then, and was still being sent out on
> 29th and 30th, despite expiring on 29th.  

Not just Windows, at least one Unix system running Postfix is still
vending a chain with the R3 signed by DST that expired on 2021-09-29:

issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
subject=C = US, O = Let's Encrypt, CN = R3
notBefore=Oct  7 19:21:40 2020 GMT
notAfter=Sep 29 19:21:40 2021 GMT
SHA256 
Fingerprint=73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
-BEGIN CERTIFICATE-
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-END CERTIFICATE-

The EE (depth 0 server) certificate is not expired, and yet somehow the
server is building a chain with a fresh leaf cert, and a rather stale
issuer CA.

It verifies via the DANE implementation in OpenSSL, because its "2 1 1"
record with a fresh RRSIG specifies the R3 CA as trusted, and its
expiration date is not in scope since it was signed by an entity outside
the effective trust chain.

Validation would fail for the same chain with WebPKI, unless there's a
new improved R3 in the trust store (not just the roots).

My DANE survey scan engine checks trust-anchor cert expiration date,
even when an intermediate CA, mostly because the implementation is
done that way, but I can retroactively justify it because it makes
sense to be more strict in tools that look for potential issues.

Implementations other than OpenSSL might similarly reject such a
suboptimal chain.

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Angus Robertson - Magenta Systems Ltd]

> Yes.  To make things even more complex, a few sites also have an 
> older version of R3 that is directly signed by the DST root:
> 
>  - leaf <- R3 <- DST Root CA X3 (self-signed)
> 
> but that's far from common at this point.

That old R3 root was issued last winter and got installed in Windows
Server 2018 intermediate stores then, and was still being sent out on
29th and 30th, despite expiring on 29th.  

Perhaps because IIS caches server certificates.  I had to delete it
from the Windows store and reboot the server to stop it being sent out
by IIS. 

Angus

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sat, Oct 02, 2021 at 05:24:26PM +0200, Alex Robuchon wrote:

> The project has a callback function registered with SSL_CTX_set_verify
> and failed to verify DST Root CA X3 since it expired.

That happens when either:

* The configured trust store (perhaps not the default system
  trust store) does not contain the ISRG root CA certificate

* The version of OpenSSL used (perhaps indirectly via some
  library that is linked with an older OpenSSL) is 1.0.x
  rather than 1.1.0 or later.

> From what I understood about the let's encrypt certificate chain, R3 is
> cross signed and two chained could be built:
> 
>- leaf <- R3 <- ISRG Root X1 <- DST Root CA X3 (self-signed)
>- leaf <- R3 <- ISRG Root X1 (self-signed)

Yes.  To make things even more complex, a few sites also have an older
version of R3 that is directly signed by the DST root:

 - leaf <- R3 <- DST Root CA X3 (self-signed)

but that's far from common at this point.

> The servers by default return the first chain but from what I
> understand depending on the openssl version/flags it should  use the
> second path if ISRG Root X1 is in the store

Yes, see above.

> My config :
> openssl version : OpenSSL 1.1.1f  31 Mar 2020

Are you sure there are no other versions of OpenSSL also installed on
the system, and/or no other trust stores that the application in
question might be using instead of the default system store?

An "strace" of the process should show which shared objects it is
loading, and which cert store files it is opening...  Best to not
post the complete strace output here, it may contain sensitive
data.  Try to interpret it on your end and post your findings.

> ISRG Root X1 self signed is in my cert store.

This presumes a *singular* "trust store", but the application could
easily be looking elsewhere.

> So from what I understand, trusted first is default in this version of
> openssl and the second path should be taken.

Yes, per the pre-requisite conditions.

> For the record s_client can valide the second path on my machine :
> $ echo | openssl s_client -connect retouche-pro.ch:443 -name retouche-pro.ch 
> -servername retouche-pro.ch
> CONNECTED(0003)
> depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
> verify return:1
> depth=1 C = US, O = Let's Encrypt, CN = R3
> verify return:1
> depth=0 CN = retouche-pro.ch
> verify return:1

As expected for 1.1.1f and a trust store that contains the ISRG X1 root.

> So I suppose the problem relies on the use of openssl in the ruby
> libs.

Are the ruby libraries linked with OpenSSL 1.1.1?  Are they overriding
the default system trust store?

> However what I'm a bit surprised with is the fact that
> SSL_CTX_set_verify is called with the "wrong" certificate chain.

I think you mean the callback specified via that function is called
with unexpected chain elements, not the function itself...

> The documentation says it's going to be called for each certificate in
> the chain but do not specify if it's the chain specified by the server
> or the one built by openssl with trusted first algorithm.

Except when various earlier errors take place during chain construction,
those callbacks take place as part of signature verification on the
already constructed chain, so the callback normally sees the "trusted"
chain, not the server provided certificate "heap" (which may not be
linearly ordered).

> Is it normal to have the callback defined in SSL_CTX_set_verify to be
> called with the wrong chain ? Or do you think something is not configured
> correctly on these gems ?

See above, the callbacks report the constructed chain, except when some
error happens during chain construction.

-- 
Viktor.