Let's Encrypt's Root Certificate is expiring!
By SCOTT HELME 20 SEP 2021
https://scotthelme.co.uk/lets-encrypt-old-root-expiration/
On 30th September 2021, the root certificate that Let's Encrypt are currently
using, the IdentTrust DST Root CA X3 certificate, will expire.
You may or may not need to do anything about this Root CA expiring, but I'm
betting a few things will probably break on that day so here's what you need to
know!
There's a long backstory
This seems like a shameless plug, but if you really want to know more about how
Certificate Authorities (CA) and Certificate Chains work, you should consider
joining me on the Practical TLS and PKI training course that I deliver. For
everyone else, this blog post, and the details I'm going to link to, should be
enough to understand what's going to happen and why.
Ultimately, all certificates that power HTTPS on the Web are issued by a CA, a
trusted organisation recognised by your device/OS. Here you can see the list of
"Trusted Root Certificate Authorities" on my current Windows 10 device:
(Graphic)
These certificates are built into your OS and are generally updated as part of
the normal process of updating your OS. The certificate in here that is going
to cause a problem is this one, the IdenTrust DST Root CA X3.
(Graphic)
As you can see, the clock is ticking and we are getting close to the expiration
date of 30th Sep 2021 but it's not just an expiration date, it's an expiration
timestamp that we call notAfter:
(Graphic)
That's converted to BST for me, but if I parse the certificate using OpenSSL
X509 you can see the UTC timestamp for expiration:
(Graphic)
That gives us quite a specific time for when this certificate will expire:
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Once this root CA has expired, clients, like web browsers, will no longer trust
certificates that have been issued by this CA.
The end is nigh!
This will not be the first time a root CA certificate has expired and I imagine
it will follow the same trend as previous expirations where things break. If
the root certificate that your certificate chain anchors on is expired then
there's a good chance it's going to cause things to fail.
This happened last year, on May 30th at 10:48:38 2020 GMT to be exact, when the
AddTrust External CA Root expired and took a bunch of things with it.
Organisations like Roku, Stripe, Spreedly and many others had problems and they
weren't the only ones, even RedHat had something to say about the event.
If you're interested in more details on the past AddTrust expiration, what
might be in store for the IdenTrust expiration and what you can do about it,
this 4-part series I wrote will help you, but be warned, it's not for the
fainthearted!
In normal circumstances this event, a root CA expiring, wouldn't even be worth
talking about because the transition from an old root certificate to a new root
certificate is completely transparent. The reason we're having a problem at all
is because clients don't get updated regularly and if the client doesn't get
updated, then the new root CA that replaces the old, expiring root CA is not
downloaded onto the device.
Let's Encrypt have grown
In the last year alone, Let's Encrypt have grown their market share quite a lot
and as a CA becomes larger, it's certificates enable more of the Web to operate
and as a result, when something like this comes along they have the potential
to cause more problems. This is nothing to do with what Let's Encrypt have
done, or have not done, this still comes down to the same underlying problem
that devices out in the ecosystem aren't being updated as they should be.
Given the relative size difference between Let's Encrypt and AddTrust, I have a
feeling that the IdenTrust root expiry has the potential to cause more
problems. Nobody really knows how much of a problem it could be, it could be of
similar consequence to the AddTrust expiry, or there could be some unforeseen
circumstances and it could be far worse, your guess is as good as mine.
What are Let's Encrypt doing about it?
As I said above, this issue isn't happening because of anything that Let's
Encrypt have or have not done, it's happening because all certificates
eventually expire and if devices aren't being updated then they won't receive
the new, replacement certificates. That said, Let's Encrypt have not sat around
and twiddled their thumbs as the expiration date has approached, they've been
working hard trying to figure out a solution.
Back in April 2019 I wrote Let's Encrypt to transition to ISRG root, where
Let's Encrypt had planned to move away from the IdenTrust root to their own
root, ISRG Root X1, that expires on 4th June 2035, giving us quite a number of
years. The problem was, not many devices had received the necessary updates
that include this new ISRG Root X1, issued 4 years prior in 2015! If a large
selection of devices had not received an update to include this new root
certificate, they simply won't trust it. This is basically the same problem
we're experiencing now with the IdenTrust root expiring, because client devices
haven't been updated, they also haven't received the new ISRG Root X1. The
transition was postponed.
In September 2020 I had to write another article, Let's Encrypt postpone the
ISRG Root transition, to explain, for the third time, that Let's Encrypt had
again postponed the transition. They cited the following concerns:
Due to concerns about insufficient ISRG root propagation on Android devices we
have decided to move the date on which we will start serving a chain to our own
root to January 11, 2021.
This loosely translates to Android devices not having received an update for
over 4 years, meaning those devices had still not received the ISRG Root X1,
meaning they wouldn't trust it. Let's Encrypt can't move to issuing from the
new root, but the IdenTrust root still has 1 year of life left and the clock is
really ticking now.
In the end, something a little unexpected has happened which might just reduce
the serious impact of this event and make it a little more palatable. Because
old Android devices don't check the expiration date of a root certificate when
they use it, Let's Encrypt may be able to continue to chain down to the expired
root certificate without any problem on those older devices. This does
introduce some complexity going forwards, but ultimately the goal is Extending
Android Device Compatibility for Let's Encrypt Certificates.
For this to work, Let's Encrypt had to get a cross-sign for their own ISRG Root
X1 certificate from the expiring IdenTrust DST Root CA X3, but that wouldn't
help at all unless the cross-signed root was valid for longer than the signing
root, which it is. The new ISRG Root X1 certificate is valid for longer than
the IdenTrust DST Root CA X3 that signed it!
As we now know, the IdenTrust DST Root CA X3 expires on 30th September 2021,
but the new, cross-signed, ISRG Root X1 does not expire until 30th September
2024!
By extending the validity of the new cross-signed root beyond that of the
signing root, Let's Encrypt have found a way to sneak past the rules and buy us
another 3 years until this problem happens all over again. Some people are not
happy with the sneaky play, but it does seem that it falls within the rules,
though not perhaps what everyone would have expected or preferred. This new,
cross-signed ISRG Root X1 is also not to be confused with the existing ISRG
Root X1 that hasn't changed and further details can be found here.
Hopefully, this will help alleviate a lot of the problems that were pending,
but it's not a solution to all problems as any client that enforces the
expiration date of the root certificate that it anchors on, will still fail.
Affected Clients
One of the notable clients that will still be affected by this expiration is
anything depending on the OpenSSL 1.0.2 or earlier library, release 22nd
January 2015 and last update as OpenSSL 1.0.2u on 20th December 2019. OpenSSL
have released a blog post detailing what action those affected can take, but
they all require manual intervention to prevent breakage, full details are
here. The brief overview of options is:
Remove the IdenTrust DST Root CA X3 root certificate and manually install the
ISRG Root X1 root certificate (not the cross-signed one).
If you're using OpenSSL commands like verify or s_client you can add the
--trusted_first flag if possible.
Have the server serve an alternate certificate chain that goes directly to the
ISRG Root X1 (not the cross-signed one), but this will break the above
mentioned Android devices.
This Let's Encrypt docs page contains a list of clients that only trust the
IdenTrust DST Root CA X3 certificate and after that is the list of platforms
that trust ISRG Root X1. I've blended these two lists together to produce the
following list of clients that will break after the IdenTrust DST Root CA X3
expires.
OpenSSL <= 1.0.2
Windows < XP SP3
macOS < 10.12.1
iOS < 10 (iPhone 5 is the lowest model that can get to iOS 10)
Android < 7.1.1 (but >= 2.3.6 will work if served ISRG Root X1 cross-sign)
Mozilla Firefox < 50
Ubuntu < 16.04
Debian < 8
Java 8 < 8u141
Java 7 < 7u151
NSS < 3.26
Amazon FireOS (Silk Browser)
Platforms that I'm still unsure of and will need some further investigation to
see if they will fail after the IdenTrust DST Root CA X3 expires are:
Cyanogen > v10
Jolla Sailfish OS > v1.1.2.16
Kindle > v3.4.1
Blackberry >= 10.3.3
PS4 game console with firmware >= 5.00
IIS
The answer to the question "what will happen when the IdenTrust root expires?"
depends on how widespread the types of clients listed above are. I don't know
what's floating around out there on the Web, and I don't know what depends on
those things either. One thing that I do know, though, is that at least
something, somewhere is going to break.
Update 08:08 BST 21st Sep 2021
I've added IIS to the 'unsure' list as some reports [1][2] suggest manual
intervention is required for a smooth transition.
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
