Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
Hi all, The reason this is manifesting as a giant problem unexpectedly since Saturday and it did not only hit obscure ancient java programs etc. as Sectigo predicted, is: SSL providers such as Namecheap SSLs.com (and probably many others) were issuing certificates good until mid-2021 with an expiring .ca-bundle file in the download zip! So anyone up until about new years was getting these files and installing expiring-in-may bundle certs onto their server along with their actual cet (expiring in 1 or 2 years usually). may 30th came along and that stopped working out so well for a LOT more users than expected. Server side mitigation: This problem can/should be fixed on the server side by server administrators updating their CA certificate bundle: (i.e. SSLCACertificateFile or SSLCertificateChainFile if you use Apache) to replace the expired Comodo or Sectigo root certs with the correct non-expired ones. The correct bundle cert depends on who your SSL is signed by, but they can be found here: https://www.ssls.com/knowledgebase/sectigo-root-certificate-expiring-may-30-2020 and also here: https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l0117LT However, Until all the servers are patched, or the add trust external CA cert is removed from the ca-certificates bundle, this will continue to explode for clients. Client side Mitigation: As specified here, mitigation on the client side is: - remove AddTrust_External_Root.crt from /etc/ca-certificates.conf - regenerate with update-ca-certificates -f -v Debian fix: Should be to update ca-certificates as per this bug and remove the expired cert. There is actually a reason to do this, as per the openSSL bug in question: https://rt.openssl.org/Ticket/Display.html?id=3359#txn-40958 "The current fix is to not have expired certificates in the trust store." So given the version of OpenSSL in stretch, the only sane fix at least for stretch (and Jessie LTS if still possible) is to remove that certificate. Many of these notes were gleaned from this reddit discussion thread: https://www.reddit.com/r/linux/comments/gshh70/sectigo_root_ca_expiring_may_not_be_handled_well/ which I just discovered now, after spending a weekend fixing up certs server side on hundreds of customer sites. Damn you, Namecheap!!! D.
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
This is accurate to my understanding of the situation. On Mon, Jun 1, 2020 at 11:33 AM Kurt Roeckx wrote: > Just to clarify, as I understand it, openssl 1.0.2 (so libssl1.0.2 > in oldstable) still has the problem, which means things like > libcurl in oldstable have that problem. And removing the > certificate from the trust store fixes it. > > > Kurt > >
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
Just to clarify, as I understand it, openssl 1.0.2 (so libssl1.0.2 in oldstable) still has the problem, which means things like libcurl in oldstable have that problem. And removing the certificate from the trust store fixes it. Kurt
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
On Mon, Jun 01, 2020 at 01:29:39AM +0200, Axel Beckert wrote: > OpenSSL 1.1.1g 21 Apr 2020 > → openssl s_client -connect mirror.sinavps.ch:443 > CONNECTED(0003) > depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = > AddTrust External CA Root > verify error:num=10:certificate has expired > notAfter=May 30 10:48:38 2020 GMT > verify return:1 > depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = > AddTrust External CA Root > notAfter=May 30 10:48:38 2020 GMT > verify return:1 > depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, > CN = USERTrust RSA Certification Authority > verify error:num=10:certificate has expired > notAfter=May 30 10:48:38 2020 GMT > verify return:1 > depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, > CN = USERTrust RSA Certification Authority > notAfter=May 30 10:48:38 2020 GMT > verify return:1 > depth=1 C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = > GlobeSSL DV Certification Authority 2 > notAfter=Sep 9 23:59:59 2024 GMT > verify return:1 > depth=0 OU = Domain Control Validated, OU = Globe Standard SSL, CN = > mirror.sinavps.ch > notAfter=Jul 16 23:59:59 2021 GMT > verify return:1 > --- > Certificate chain > 0 s:OU = Domain Control Validated, OU = Globe Standard SSL, CN = > mirror.sinavps.ch > i:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = > GlobeSSL DV Certification Authority 2 > 1 s:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = > GlobeSSL DV Certification Authority 2 >i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN > = USERTrust RSA Certification Authority > 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN > = USERTrust RSA Certification Authority >i:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = > AddTrust External CA Root > --- > [...] Verify return code: 0 (ok) I think openssl's output is confusing, but it always has that "verify return:1" for each certificate at the top. Kurt
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
On Mon, Jun 1, 2020 at 1:29 AM Axel Beckert wrote:
> > You will need to workaround this. As such this motivates critical me
> think.
>
> I think "grave" is severe enough, as it "only" breaks HTTPS including
> apt with HTTPS-based mirrors (as the one mentioned above) and hence
> only "unrelated software/packages", not the whole system (like the
> kernel or the bootloader would do if the system won't boot anymore
> after an upgrade).
>
ok.
I read the description about unrelated software a bit differently indeed.
("makes unrelated software on the system (or the whole system) break, or
causes serious data loss, or introduces a security hole on systems where
you install the package.")
> > just doing a straight up curl will hang until timeout. With the expired
> > cert disabled this is bypassaed (without curl -k).
>
> Nope. curl exits immediately for me, at least in unstable (7.68.0-1):
>
Indeed. Sorry, me being inaccurate. I was testing this on old stable.
As you noted later on as well =)
Ack, stretch is affected, too, at least with lynx and — funnily again
> — curl (7.52.1-5+deb9u10).
>
Thanks for digging further into this issue.
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
Hi, Axel Beckert wrote: > Certificate chain > 0 s:OU = Domain Control Validated, OU = Globe Standard SSL, CN = > mirror.sinavps.ch > i:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = > GlobeSSL DV Certification Authority 2 > 1 s:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = > GlobeSSL DV Certification Authority 2 >i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN > = USERTrust RSA Certification Authority > 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN > = USERTrust RSA Certification Authority >i:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = > AddTrust External CA Root > --- https://archive.raspberrypi.org/ also seems to have been affected (four hours ago, about 20:30 UTC) but is no more as of writing this mail. Common demoniater with the affected https://mirror.sinavps.ch/ is the above mentioned "USERTrust RSA Certification Authority" certificate. > The longer I think about the more I think it is a bug in both OpenSSL > and GnuTLS, because the certificate above is totally valid because the > second last CA is actually no more an Intermediate CA but in > ca-certificates, too. > > But for some reason, even though the third certificate in the chain is > trusted, both, OpenSSL and GnuTLS seem to see the fourth certificate > and only seem to check if that one is trusted and not any inbetween. This might be related to the used Intermediate CA certificate used on the server side. Because if https://archive.raspberrypi.org/ could be fixed on the server side, this smells a lot like the Intermediate CA certificate. So if that Intermediate CA certificate on the server includes the "USERTrust RSA Certification Authority" certification, the client doesn't seem to trust it even if a certificate with the same serial is in it's own list of trusted certificates, and it tries to verify the included signature, which is from the expired AddTrust. So the amount of "bug" which could be argued is in OpenSSL and GnuTLS is probably rather small. It's more a kind of missing feature to check every Intermediate CA certificate if it is also by chance in the local list of trusted CAs. Regards, Axel -- ,''`. | Axel Beckert , https://people.debian.org/~abe/ : :' : | Debian Developer, ftp.ch.debian.org Admin `. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5 `-| 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
Bug#961907: ca-certificates: Remove expired mozilla/AddTrust_External_Root.crt
Control: affects -1 + lynx libwww-perl wget links links2 apt aptitude w3m curl openssl dillo mpv epiphany vlc luakit surf aptitude-robot Hi, Rémi Denis-Courmont wrote: > The AddTrust_External_Root.crt certificate has expired, and its > continued inclusion in the ca-certificates set is causing GnuTLS-based > client applications (and OpenSSL 1.0.x) to barf on a lot of sites. Not only OpenSSL 1.0.x, also OpenSSL in unstable is affected: → openssl version OpenSSL 1.1.1g 21 Apr 2020 → openssl s_client -connect mirror.sinavps.ch:443 CONNECTED(0003) depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:1 depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root notAfter=May 30 10:48:38 2020 GMT verify return:1 depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:1 depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority notAfter=May 30 10:48:38 2020 GMT verify return:1 depth=1 C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = GlobeSSL DV Certification Authority 2 notAfter=Sep 9 23:59:59 2024 GMT verify return:1 depth=0 OU = Domain Control Validated, OU = Globe Standard SSL, CN = mirror.sinavps.ch notAfter=Jul 16 23:59:59 2021 GMT verify return:1 --- Certificate chain 0 s:OU = Domain Control Validated, OU = Globe Standard SSL, CN = mirror.sinavps.ch i:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = GlobeSSL DV Certification Authority 2 1 s:C = US, ST = DE, L = Wilmington, O = "Globe Hosting, Inc.", CN = GlobeSSL DV Certification Authority 2 i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority i:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root --- [...] > It could probably be argued that this is a bug in GnuTLS rather than > ca-certificates, The longer I think about the more I think it is a bug in both OpenSSL and GnuTLS, because the certificate above is totally valid because the second last CA is actually no more an Intermediate CA but in ca-certificates, too. But for some reason, even though the third certificate in the chain is trusted, both, OpenSSL and GnuTLS seem to see the fourth certificate and only seem to check if that one is trusted and not any inbetween. Because "USERTrust RSA Certification Authority" is actually not expired, just a certificate, which signed it (obviously as shown above): (on a buster system) $ openssl x509 -in /etc/ssl/certs/USERTrust_RSA_Certification_Authority.pem -noout -text | egrep 'Not After|Subject:' Not After : Jan 18 23:59:59 2038 GMT Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority $ openssl x509 -in /etc/ssl/certs/AddTrust_External_Root.pem -noout -text | egrep 'Not After|Subject:' Not After : May 30 10:48:38 2020 GMT Subject: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root > but I don't see the point in keeping an expired certificate here. Ack. > The problem is confirmed to affect Epiphany and VLC. Tons more: I ran into it with aptitude-robot (via aptitude and apt) first and was able to confirm it in nearly all browsers and streaming videoplayer I could think of. The only exceptions were firefox-esr, chromium, and — to my surprise — qutebrowser. Martin Bagge / brother wrote: > severity: critical > Thanks > > You will need to workaround this. As such this motivates critical me think. I think "grave" is severe enough, as it "only" breaks HTTPS including apt with HTTPS-based mirrors (as the one mentioned above) and hence only "unrelated software/packages", not the whole system (like the kernel or the bootloader would do if the system won't boot anymore after an upgrade). > just doing a straight up curl will hang until timeout. With the expired > cert disabled this is bypassaed (without curl -k). Nope. curl exits immediately for me, at least in unstable (7.68.0-1): → time curl https://mirror.sinavps.ch curl: (60) SSL certificate problem: certificate has expired More
