Re: "DST Root CA X3" expiry workaround for OpenSSL 1.0.1, 1.0.0 and 0.9.8 client applications

[Felipe Gasper]

> On Oct 3, 2021, at 7:09 PM, Rob Stradling  wrote:
> 
> The "Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2" blog 
> article [1] doesn't cover OpenSSL versions prior to 1.0.2, presumably because 
> they've been unsupported for a long time.  However, no doubt there are still 
> some users and applications that are stuck using even older versions of 
> OpenSSL.  

Indeed, CentOS/RHEL 7 and earlier still use OpenSSL 1.0.2. These are widely 
used in shared hosting.

We ended up implementing Workaround #2 in order to bring OpenSSL 1.0.x into 
parity with the behaviour of 1.1.0+. It would be nice--would have been helpful 
for me, at least--to have fleshed-out code examples.

Thank you to everyone who maintains OpenSSL and who’s helping us all through 
this transition.

cheers,
-Felipe Gasper

"DST Root CA X3" expiry workaround for OpenSSL 1.0.1, 1.0.0 and 0.9.8 client applications

[Rob Stradling]

The "Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2" blog 
article [1] doesn't cover OpenSSL versions prior to 1.0.2, presumably because 
they've been unsupported for a long time.  However, no doubt there are still 
some users and applications that are stuck using even older versions of 
OpenSSL.  Here are some solutions...

Firstly, it's worth noting that Workaround 1 (see [1]) also works with OpenSSL 
1.0.1n and above, which seems to be because the 1.0.2 "alternative chains" 
behaviour was backported (see [2]).

Secondly, I've come up with a variant of Workaround 1 (see "Workaround 0" 
below) that also works for even older OpenSSL releases.  I thought I'd share it 
here in case it helps anyone out...

---
Workaround 0 (on clients with OpenSSL >=0.9.8m)

Modify the "notAfter" date in the expired root certificate (DST Root CA X3) 
from "Sep 30 14:01:15 2021 GMT" to "Sep 30 18:14:03 2024 GMT" (which is when 
the new DST->ISRG cross-certificate is due to expire - see [3] and [4]); then, 
in the OpenSSL-based client application's trust store, replace the expired root 
certificate with the modified version.  Here's one way to create the modified 
certificate:

wget -O DSTRootCAX3_Extended.crt "https://crt.sh/?d=8395;
sed -i "s/xMDkzMDE0MDExNVow/0MDkzMDE4MTQwM1ow/g" DSTRootCAX3_Extended.crt

You should find that "openssl s_client" with "-CAfile DSTRootCAX3_Extended.crt" 
will return "Verify return code: 0 (ok)" when connecting to any server that 
serves Let's Encrypt's preferred certificate chain (i.e., the chain that 
includes [4]).  For example:

openssl s_client -connect letsencrypt.org:443 -CAfile 
../DSTRootCAX3_Extended.crt

Why does this work?
Modifying the expired root certificate obviously invalidates its self-signed 
signature.  However, OpenSSL 0.9.8m and above don't check self-signed 
signatures by default (due to [5]), which is one factor in why this workaround 
works (whereas 0.9.8l and below do check self-signed signatures).  The other 
factor is that OpenSSL treats the modified certificate as a root certificate, 
because the Issuer and Subject Names are identical; since it's not treated as 
an intermediate certificate, older versions of OpenSSL are able to treat it as 
a trust anchor even though they lack the "partial chains" functionality that 
was added in 1.0.2 (see [6]).

It is not necessary to add the new ISRG Root X1 self-signed certificate to the 
client's trust store for Workaround 0 to work.  Note however that Workaround 0 
won't work beyond Sep 30 18:14:03 2024 GMT.


[1] 
https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

[2] 
https://github.com/openssl/openssl/commit/f7bf8e02dfcb2c02bc12a59276d0a3ba43e6c204

[3] https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

[4] https://crt.sh/?id=3958242236

[5] 
https://github.com/openssl/openssl/commit/1e53b797f65ef6d3c2eb1052797683fec27a9ff5

[6] 
https://github.com/openssl/openssl/commit/9a1f59cd3128ddac73d3e0721ecd55935f53ba8b


--
Rob Stradling
Senior Research & Development Scientist
Sectigo Limited

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

> You don't have to explain the code to me. :-)

^^. Ok. I was also trying to explain what I understood so you can correct
me if I'm wrong.

Everything is Cristal clear now.

Thanks a million.

On Sun, 3 Oct 2021, 22:25 Viktor Dukhovni, 
wrote:

> On Sun, Oct 03, 2021 at 09:33:29PM +0200, Alex Robuchon wrote:
>
> > > Not quite, a candidate chain is constructed from whatever certificates
> the
> > > peer (server in your case) provided, and then passed to the callback
> with
> > > "preverify_ok" set to false (for the top certificate), because the
> chain is
> > > not trusted.
> >
> > This confuses me a little bit because I thought the callback function set
> > with *SSL_set_verify* would have the "trusted first" valid chain.
>
> But there is no trust store configured, so trust first is a no-op.
> The constructed chain consists entirely of certificates from the
> peer, and last certificate in the chain (the ultimate issuing CA)
> is still not trusted, so preverify_ok = 0.
>
>
> > Are you sure you are not speaking as if eventmachine was using the
> > *SSL_CTX_set_cert_verify_callback* ?
>
> I rewrote the OpenSSL chain construction code for OpenSSL 1.1.0.  Yes, I
> am sure.
>
> > - *build_chain* that will apply the trusted first algorithm and replace
> the
> > certificate chain passed by the server with the valid one ( if found ).
>
> The code in eventmachine does not configure any trusted certificates for
> the SSL_CTX handles used to create the SSL connections.  So
> trusted-first is a no-op.
>
> > - *internal_verify *which now has the new chain and will call
> *verify_callback
> > *( the callback function passed to SSL_set_verify ) for every certificate
> > in this new chain in reverse order.
>
> You don't have to explain the code to me. :-)
>
> > > But given all the evidence before me, I'd want to delete that code and
> > > never see it again.
> >
> > I hear you :).
>
> That code is beyond repair, it needs to be thrown out, and replaced with
> ground up by the book TLS connection establishment.
>
> #1. No compiled in private keys
>
> #2. Configure the SSL_CTX with the desired list of trust anchors
>
> #3. Reuse the SSL_CTX for multiple connections that share the same
> trust and other general settings
>
> #4. Prior to SSL_connect(), tell the SSL library what hostname you want
> to verify via SSL_set1_host(3).  Let OpenSSL do all the heavy
> lifting of certificate and name checks.
>
> #5. DO NOT attempt to override certificate chain verifiction in the
> verify callback.  Use the verify callback only for logging or
> similar diagnostic purposes.
>
> #6. Return "preverify_ok" unmodified, unless your application is doing
> opportunistic unauthenticated TLS, or wants to complete the
> handshake even on verification failure, and then gracefully tear
> down the application-layer session with the peer (after checking the
> verification status at connection completion).
>
> If preverify_ok is false (0), OpenSSL chain verification failed,
> your application is unlikely to do better.  Return 0, and the
> TLS handshake will be aborted, you can log the error reason,
> chain depth, ... in the verify callback.
>
> --
> Viktor.
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sun, Oct 03, 2021 at 09:33:29PM +0200, Alex Robuchon wrote:

> > Not quite, a candidate chain is constructed from whatever certificates the
> > peer (server in your case) provided, and then passed to the callback with
> > "preverify_ok" set to false (for the top certificate), because the chain is
> > not trusted.
> 
> This confuses me a little bit because I thought the callback function set
> with *SSL_set_verify* would have the "trusted first" valid chain.

But there is no trust store configured, so trust first is a no-op.
The constructed chain consists entirely of certificates from the
peer, and last certificate in the chain (the ultimate issuing CA)
is still not trusted, so preverify_ok = 0.


> Are you sure you are not speaking as if eventmachine was using the
> *SSL_CTX_set_cert_verify_callback* ?

I rewrote the OpenSSL chain construction code for OpenSSL 1.1.0.  Yes, I
am sure.

> - *build_chain* that will apply the trusted first algorithm and replace the
> certificate chain passed by the server with the valid one ( if found ).

The code in eventmachine does not configure any trusted certificates for
the SSL_CTX handles used to create the SSL connections.  So
trusted-first is a no-op.

> - *internal_verify *which now has the new chain and will call *verify_callback
> *( the callback function passed to SSL_set_verify ) for every certificate
> in this new chain in reverse order.

You don't have to explain the code to me. :-)

> > But given all the evidence before me, I'd want to delete that code and
> > never see it again.
> 
> I hear you :).

That code is beyond repair, it needs to be thrown out, and replaced with
ground up by the book TLS connection establishment.

#1. No compiled in private keys

#2. Configure the SSL_CTX with the desired list of trust anchors

#3. Reuse the SSL_CTX for multiple connections that share the same
trust and other general settings

#4. Prior to SSL_connect(), tell the SSL library what hostname you want
to verify via SSL_set1_host(3).  Let OpenSSL do all the heavy
lifting of certificate and name checks.

#5. DO NOT attempt to override certificate chain verifiction in the
verify callback.  Use the verify callback only for logging or
similar diagnostic purposes.

#6. Return "preverify_ok" unmodified, unless your application is doing
opportunistic unauthenticated TLS, or wants to complete the
handshake even on verification failure, and then gracefully tear
down the application-layer session with the peer (after checking the
verification status at connection completion).

If preverify_ok is false (0), OpenSSL chain verification failed,
your application is unlikely to do better.  Return 0, and the
TLS handshake will be aborted, you can log the error reason,
chain depth, ... in the verify callback.

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

>
> Not quite, a candidate chain is constructed from whatever certificates the
> peer (server in your case) provided, and then passed to the callback with
> "preverify_ok" set to false (for the top certificate), because the chain is
> not trusted.
>

This confuses me a little bit because I thought the callback function set
with *SSL_set_verify* would have the "trusted first" valid chain. Are you
sure you are not speaking as if eventmachine was using the
*SSL_CTX_set_cert_verify_callback* ?

>From what I see from the source code ( master )

The *tls_post_process_server_certificate* function calls
ssl/ssl_cert.c#ssl_verify_cert_chain


if (s->ctx->app_verify_callback != NULL)
i = s->ctx->app_verify_callback(ctx, s->ctx->app_verify_arg);
else
i = X509_verify_cert(ctx);
return i

Since eventmachine do not set the *app_verify_callback* so the default
*X509_verify_cert* is used from crypto/x509/x509_vfy.c (L261)


The function *X509_verify_cert* calls *verify_chain(ctx)* line 295 which in
turns calls:
- *build_chain* that will apply the trusted first algorithm and replace the
certificate chain passed by the server with the valid one ( if found ). At
the point the ctx has the new chain
- *internal_verify *which now has the new chain and will call *verify_callback
*( the callback function passed to SSL_set_verify ) for every certificate
in this new chain in reverse order.

During the build_chain process I notice the ctx->get_issuer ( which
actually points to X509_STORE_CTX_get1_issuer which I suppose return 0
because eventmachine do not properly set the store )


But the evenmachine callback ignores "preverify_ok" and goes through the
> motions of doing some sort of verification of each certificate.
>

yes indeed

But given all the evidence before me, I'd want to delete that code and
> never see it again.
>

I hear you :).

On Sun, Oct 3, 2021 at 6:48 PM Viktor Dukhovni 
wrote:

> > On 3 Oct 2021, at 12:33 pm, Alex Robuchon 
> wrote:
> >
> > So I suppose openssl skip the part that is supposed to build the chain
> when no store is configured.
>
> Not quite, a candidate chain is constructed from whatever certificates
> the peer (server in your case) provided, and then passed to the callback
> with "preverify_ok" set to false (for the top certificate), because the
> chain is not trusted.
>
> But the evenmachine callback ignores "preverify_ok" and goes through the
> motions of doing some sort of verification of each certificate.
>
> Ultimately, it will attempt to "verify" the leaf certificate, and if it
> can somehow do a fair job of that (by building its own chain, checking
> all the signatures, doing the name checks (for a name that does not
> appear to be passed to the verification function), then in theory
> the checks at depths above 0 are just silly opportunities to fail and
> the EE cert check would be enough.
>
> But given all the evidence before me, I'd want to delete that code and
> never see it again.
>
> --
> Viktor.
>
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

> On 3 Oct 2021, at 12:33 pm, Alex Robuchon  
> wrote:
> 
> So I suppose openssl skip the part that is supposed to build the chain when 
> no store is configured.

Not quite, a candidate chain is constructed from whatever certificates
the peer (server in your case) provided, and then passed to the callback
with "preverify_ok" set to false (for the top certificate), because the
chain is not trusted.

But the evenmachine callback ignores "preverify_ok" and goes through the
motions of doing some sort of verification of each certificate.

Ultimately, it will attempt to "verify" the leaf certificate, and if it
can somehow do a fair job of that (by building its own chain, checking
all the signatures, doing the name checks (for a name that does not
appear to be passed to the verification function), then in theory
the checks at depths above 0 are just silly opportunities to fail and
the EE cert check would be enough.

But given all the evidence before me, I'd want to delete that code and
never see it again.

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

I just had a big laugh Reading this. I never had to dig into openssl before
so I unfortunately had to try to understand your API with eventmachine as
an example. Silly me.


> On the other hand I suppose if we do not call this it would pick the
> "default" trust store from the system which seems to be the case here
> because it can find /usr/lib/ssl/certs/2e5ac55d.0 .
>

Actually the part responsible for this lookup is not part of the openssl
library but from the ruby callback function registered through
SSL_set_verify  which as you said tries to verify each certificate in
isolation ( with a configured store this time )

So I suppose openssl skip the part that is supposed to build the chain when
no store is configured.

Do whatever it takes to never rely on this code again.  Even abandon Ruby
> if that's what it takes...


 Hopefully we do not have a lot of code relying on eventmachine so we'll
see if we can improve the library or move away from it.

Thanks so much for your time, explanation and responsiveness.


On Sun, 3 Oct 2021, 17:37 Viktor Dukhovni, 
wrote:

> On Sun, Oct 03, 2021 at 01:54:44PM +0200, Alex Robuchon wrote:
>
> > Thanks for the detailed answer.
> >
> > From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
> >
> > When I use the eventmachine lib that uses the wrong cert chain I can see
> > with strace :
>
> Run as far away from eventmachine as you can.  The very first thing you
> see in the code is a compiled in default "private key" (for all the
> world to share).
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L30-L123
>
> Though applications can override these, and supply actual private keys,
> it does not get better from there...  The comments in the code show the
> author punting on understanding the OpenSSL API and just guessing what
> to do.
>
> Indeed the code creates SSL_CTX objects without specifying either the
> default or explicit trust stores.
>
> The real disaster is in:
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L675-L698
>
> with a completely broken SSL verification callback, that completely
> ignores all errors from the chain construction and signature
> verification code, and just attempts to "verify" each certificate in
> *isolation*.
>
>
> https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L693-L697
>
> This means:
>
> * No verification of chain signatures
> * No verification of path constraints
> * No verification of name constraints
> * No hostname checks.
> * ...
>
> This code was written by someone too clueless to know what they're
> doing and too lazy to bother to learn.  DO NOT USE IT.
>
> Do whatever it takes to never rely on this code again.  Even abandon
> Ruby if that's what it takes...
>
> --
> Viktor.
>

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Viktor Dukhovni]

On Sun, Oct 03, 2021 at 01:54:44PM +0200, Alex Robuchon wrote:

> Thanks for the detailed answer.
> 
> From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
> 
> When I use the eventmachine lib that uses the wrong cert chain I can see
> with strace :

Run as far away from eventmachine as you can.  The very first thing you
see in the code is a compiled in default "private key" (for all the
world to share).


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L30-L123

Though applications can override these, and supply actual private keys,
it does not get better from there...  The comments in the code show the
author punting on understanding the OpenSSL API and just guessing what
to do.

Indeed the code creates SSL_CTX objects without specifying either the
default or explicit trust stores.

The real disaster is in:


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L675-L698

with a completely broken SSL verification callback, that completely
ignores all errors from the chain construction and signature
verification code, and just attempts to "verify" each certificate in
*isolation*.


https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L693-L697

This means:

* No verification of chain signatures
* No verification of path constraints
* No verification of name constraints
* No hostname checks.
* ...

This code was written by someone too clueless to know what they're
doing and too lazy to bother to learn.  DO NOT USE IT.

Do whatever it takes to never rely on this code again.  Even abandon
Ruby if that's what it takes...

-- 
Viktor.

Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )

[Alex Robuchon]

Thanks for the detailed answer.

>From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1

When I use the eventmachine lib that uses the wrong cert chain I can see
with strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/2e5ac55d.0", {st_mode=S_IFREG|0644, st_size=1200,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/2e5ac55d.0", O_RDONLY) = 8
stat("/usr/lib/ssl/certs/2e5ac55d.1", 0x7ffe1a8e5d80) = -1 ENOENT (No such
file or directory)

When I use another lib that uses the correct cert chain I can see with
strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/8d33f237.0", 0x7fff1b4b0f90) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/4042bcee.0", {st_mode=S_IFREG|0644, st_size=1939,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/4042bcee.0", O_RDONLY) = 8

In the second case I can see it tries to load the R3 certificate
( 8d33f237.0 ). I wonder why in the first case it does not try to find each
certificate in the chain from the trust store at all. I wonder if it can be
related to the fact I do not see any call to SSL_CTX_set_cert_store in the
lib.  On the other hand I suppose if we do not call this it would pick the
"default" trust store from the system which seems to be the case here
because it can find /usr/lib/ssl/certs/2e5ac55d.0 .

This looks like to be an issue in the eventmachine lib itself. I need to
brush up my C skills and have a deeper look at this :).

Thanks

On Sat, Oct 2, 2021 at 8:11 PM Viktor Dukhovni 
wrote:

> On Sat, Oct 02, 2021 at 06:21:00PM +0100, Angus Robertson - Magenta
> Systems Ltd wrote:
>
> > > Yes.  To make things even more complex, a few sites also have an
> > > older version of R3 that is directly signed by the DST root:
> > >
> > >  - leaf <- R3 <- DST Root CA X3 (self-signed)
> > >
> > > but that's far from common at this point.
> >
> > That old R3 [CA] was issued last winter and got installed in Windows
> > Server 2018 intermediate stores then, and was still being sent out on
> > 29th and 30th, despite expiring on 29th.
>
> Not just Windows, at least one Unix system running Postfix is still
> vending a chain with the R3 signed by DST that expired on 2021-09-29:
>
> issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
> subject=C = US, O = Let's Encrypt, CN = R3
> notBefore=Oct  7 19:21:40 2020 GMT
> notAfter=Sep 29 19:21:40 2021 GMT
> SHA256
> Fingerprint=73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
> -BEGIN CERTIFICATE-
> MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
> MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
> DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
> MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
> AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
> jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
> Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
> U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
> gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
> /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
> oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
> BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
> ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
> p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
> AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
> Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
> LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
> r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
> AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
> ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
> S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
> qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
> O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
> UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
> -END CERTIFICATE-
>
> The EE (depth 0 server) certificate is not expired, and yet somehow the
> server is building a chain with a fresh leaf cert, and a rather stale
> issuer CA.
>
> It verifies via the DANE implementation in OpenSSL, because its "2 1 1"
> record with a fresh RRSIG specifies the R3 CA as trusted, and its
> expiration date is not in scope since it was signed by an entity outside
> the effective trust chain.
>
> Validation would fail for the same chain with WebPKI, unless there's a
> new improved R3 in the trust store (not just the roots).
>
> My DANE survey scan engine checks trust-anchor cert expiration date,
> even when an intermediate CA,