Interesting you mention that - it's what cPanel flagged as a
contributing factor towards how long their restoration took.
"While we were aware of Lets Encrypt's upcoming root expiration, we were
not aware that they would and still are providing us a chain to that
expired root as it is the default chain when using their API.
Identifying and understanding this certificate behaviour ahead of time
would have helped us pre-empt these issues and I can assure you that the
importance of this early confirmation in the future is being emphasised."
The main forum threads on the LE site are rather vocal shall we say!
Understandably so.
Hopefully it's a learning exercise that helps with future transitions.
Cheers,
Luke Thompson
Operations Manager
On 6/10/21 2:49 pm, Beeson, Ayden wrote:
This isn't just old devices either, my Android phone is running near
enough to the latest release and this broke my radius auth against a
Lets encrypt cert.
I had to manually edit the CA to remove the final root cert, which
Lets Encrypt are still including via the renewal tool I'm using.
I knew it was coming, but didn't expect it to cause this problem....
------------------------------------------------------------------------
*From:* AusNOG <[email protected]> on behalf of Luke
Thompson <[email protected]>
*Sent:* Friday, 1 October 2021 12:44:44 PM
*To:* Lachlan Gilmour; Mark Andrews
*Cc:* [email protected]
*Subject:* Re: [AusNOG] Global DNS yuck?
cPanel also failed to plan for the expiry, so we're seeing workarounds
then revocations (oops, that didn't work - etc). Still no real headway
after 12~ hours.
The root cert expiry was a long time coming, though if you check
Twitter it seems like it's caught many out.
Cheers,
Luke Thompson
Operations Manager
On 1/10/21 12:40 pm, Lachlan Gilmour wrote:
I believe it is related to the Lets Encrypt root cert that expired
overnight.
I've seen quite a few older devices today having issues accessing
sites using lets encrypt certs.
More info on the issue can be found here:
https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZhMDU5Mjk2ZGMwY2FkNGNjYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjk4NjkwOGJlM2E1YTI9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG9jcyUyRWNlcnRpZnl0aGV3ZWIlMkVjb20lMkZkb2NzJTJGa2IlMkZrYi0yMDIxMDktbGV0c2VuY3J5cHQlMkY=>
On Fri, Oct 1, 2021 at 12:36 PM Mark Andrews <[email protected]
<mailto:[email protected]>> wrote:
More correctly they had working DNSSEC deployed
(https://dnsviz.net/d/slack.com/YVXX_g/dnssec/
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiMTQ0NjA3MDliZDFkZGRhYz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZjA4NjkwOGJlM2E1OWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZG5zdml6JTJFbmV0JTJGZCUyRnNsYWNrJTJFY29tJTJGWVZYWCU1RmclMkZkbnNzZWMlMkY=>)
and then pulled both the DS records for slack.com
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>
and the DNSSEC records in slack.com
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>
AT THE SAME TIME resulting in DNSSEC validation failures. Cached
DS records said slack.com
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>
is signed but the answers from the slack.com
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>
servers where missing the DNSSEC records. They failed to wait for
the DS records to expire from DNS caches before removing the
DNSSEC records in slack.com
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM4Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZiZmFjYjk4Y2FlZGU1YWI9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZzbGFjayUyRWNvbQ==>.
Failure to wait for unsigned responses to clear caches before
publishing DS records can also cause issues with multiple levels
of caching.
> On 1 Oct 2021, at 08:23, Scott Howard <[email protected]
<mailto:[email protected]>> wrote:
>
> They broke (and subsequently fixed) their DNSSEC configuration
many hours ago, but it was broken long enough to get cached by
some servers for up to 24 hours so some users are still having
issues connecting.
>
> Short of the classic "have your ISP clear their DNS cache" not
much anyone can do except wait it out...
>
> https://status.slack.com/2021-09/06c1e17de93e7dc2
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiODUxMmM3ZjlhYzFkZWQ2OD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY4YTk5OGMxZDVhM2JhZmY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3RhdHVzJTJFc2xhY2slMkVjb20lMkYyMDIxLTA5JTJGMDZjMWUxN2RlOTNlN2RjMg==>
>
> Scott
>
>
> On Thu, Sep 30, 2021 at 3:19 PM Andrew Yager
<[email protected] <mailto:[email protected]>> wrote:
> Hi,
>
> Slack is down and finding a few other (non slack) services etc
being broken seemingly with DNS things. Anyone know what’s going on?
>
> A
> _______________________________________________
> AusNOG mailing list
> [email protected] <mailto:[email protected]>
> http://lists.ausnog.net/mailman/listinfo/ausnog
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>
> _______________________________________________
> AusNOG mailing list
> [email protected] <mailto:[email protected]>
> http://lists.ausnog.net/mailman/listinfo/ausnog
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: [email protected]
<mailto:[email protected]>
_______________________________________________
AusNOG mailing list
[email protected] <mailto:[email protected]>
http://lists.ausnog.net/mailman/listinfo/ausnog
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYTVmMjgzYWRhYzdjNTk0Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiZhZmFjMTlmODllZmU0ZTk9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZsaXN0cyUyRWF1c25vZyUyRW5ldCUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRmF1c25vZw==>
--
Lachlan Gilmour
w : surfpacific.com.au
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NGM3OTg5ZTk0ZD02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiY0ZWVkYTk1OTRlZmU5YWY9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGc3VyZnBhY2lmaWMlMkVjb20lMkVhdSUyRg==>
p : +61 7 5571 1161 <tel:+61755711161>
f : +61 7 5676 6652
e : [email protected]
<mailto:[email protected]>
a
: Suite 30307, Level 3, Tower 3 Southport Central Commercial,
9 Lawson Street, Southport, Queensland 4215, Australia.
<http://antispam.csu.edu.au:32224/?dmVyPTEuMDAxJiZiYzQ0M2I2NDhlOGQ5ZWM5Yz02MTU2NzYzNV83NTY2MV8xNTEwNl8xJiYyZjZjNzg3ODFhMGY5YjM9MTMzMyYmdXJsPWh0dHAlM0ElMkYlMkZyZW1vdGUlMkVzdXJmcGFjaWZpYyUyRWNvbSUyRg==>
------------------------------------------------------------------------
*Legal Notice:* If this email message is received by other than the
named addressee(s), then the recipient is requested immediately to
notify us and delete the email from the recipient’s computer memory
and to destroy all hard and other copies of it. Privilege is not
waived or lost by reason of a mistaken delivery or transmission to
other than the addressee. Please
_______________________________________________
AusNOG mailing list
[email protected]
http://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
[email protected]
http://lists.ausnog.net/mailman/listinfo/ausnog
