Re: [dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
On Wed, Nov 01, 2023 at 04:49:01PM +0100, Mark Andrews wrote: > It shouldn’t take any time as the bogus records shouldn’t have been cached. > Right, unlike mismatched parent-side DS RRs, RRSIG expiration heals fairly promptly once the zone is resigned at the origin. I am repeatedly surprised when I hear of operators finding out about RRSIG expiration after the fact from 3rd parties. Somehow the reflexive knowlege that DNS monitoring means not only: - Is it still working at this very moment but also: - Is it about to stop working if nothing is done soon appears to not have become an ingrained part of the operator culture. * What can we as a community do to get the message out? * What tooling improvements could make this easier for operators? Specifically, in the case of signed zones, monitoring MUST also include regular checks of the remaining expiration time of at least the core zone apex records (DNSKEY, SOA and NS), and ideally the whole zone, both on the primary server and the secondaries. There needs to be a minimum acceptable remaining RRSIG time that's some reasonable fraction of the total RRSIG lifetime, which if crossed leaves enough time for the responsible operator to react and rectify any issues. My tiny zones are monitored to not go below ~π days of remaining RRSIG validity. :-) ldns-verify-zone -e P0Y0M3DT3H23M54S -V1 ... [ Of course that minimum time needs to be less than the threshold at which extant records are normally resigned. ] Should authoritative resolvers have knobs to perform internal checks on the signed zones they serve and at least syslog loud warnings? If there were some protocol to get a message into a monitoring system, that would be even better... Ideally, if operators cannot or do not on their own implement the requisite monitoring, is it possible to make it easy enough for them to do, and is sufficiently prominently documented or otherwise becomes well known, that they start doing it? "Unmonitored critical service", especially when it involves security, should be an oxymoron. -- Viktor. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
It shouldn’t take any time as the bogus records shouldn’t have been cached. -- Mark Andrews > On 1 Nov 2023, at 15:06, Paul de Weerd wrote: > > Dear Matthew, > >> On 01/11/2023 12:13, Matthew Richardson via dns-operations wrote: >> Our systems use some RIPE Atlas anchors for general connectivity >> monitoring. Just now, they all failed. >> If looks as if DNSSEC has expired:- >> https://dnsviz.net/d/anchors.atlas.ripe.net/dnssec/ >> It also looks as if other things in ripe.net may also have expired (eg >> www.ripe.net when looking for a contact to advise of this). > > Indeed, there was an issue with the DNSSEC signatures on the ripe.net zone > expiring earlier today (20231101104448). As Stephane commented, this was > resolved at 12:15 (UTC) on our end, but as usual it may take some time for > the fixed zone to propagate to all caches. > > We are working on a post mortem about this incident and will share that with > the community ASAP. > > For future reference, in case of issues with the ripe.net services, > https://status.ripe.net/ should be the go-to place. Admittedly, with the > ripe.net zone bogus, that was also unavailable - something more to consider > going forward. > > Best regards, > > Paul de Weerd > Manager Global Information Infrastructure team > RIPE NCC > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
Dear Matthew, On 01/11/2023 12:13, Matthew Richardson via dns-operations wrote: Our systems use some RIPE Atlas anchors for general connectivity monitoring. Just now, they all failed. If looks as if DNSSEC has expired:- https://dnsviz.net/d/anchors.atlas.ripe.net/dnssec/ It also looks as if other things in ripe.net may also have expired (eg www.ripe.net when looking for a contact to advise of this). Indeed, there was an issue with the DNSSEC signatures on the ripe.net zone expiring earlier today (20231101104448). As Stephane commented, this was resolved at 12:15 (UTC) on our end, but as usual it may take some time for the fixed zone to propagate to all caches. We are working on a post mortem about this incident and will share that with the community ASAP. For future reference, in case of issues with the ripe.net services, https://status.ripe.net/ should be the go-to place. Admittedly, with the ripe.net zone bogus, that was also unavailable - something more to consider going forward. Best regards, Paul de Weerd Manager Global Information Infrastructure team RIPE NCC ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
On Wed, Nov 01, 2023 at 01:37:14PM +0100, Stephane Bortzmeyer wrote a message of 17 lines which said: > > If looks as if DNSSEC has expired:- > > It seems it has been repaired around 1215 UTC. https://twitter.com/ripencc/status/1719712189496311986 "Our services have been restored and all services are operational. We believe the root cause of the issue was DNSSEC-related, and we are continuing to monitor the situation. We will soon share a postmortem on our status page: https://status.ripe.net; ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
On Wed, Nov 01, 2023 at 11:13:15AM +, Matthew Richardson via dns-operations wrote a message of 64 lines which said: > If looks as if DNSSEC has expired:- It seems it has been repaired around 1215 UTC. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
[dns-operations] anchors.atlas.ripe.net/ripe.net - DNSSEC bogus due expiration
--- Begin Message --- Our systems use some RIPE Atlas anchors for general connectivity monitoring. Just now, they all failed. If looks as if DNSSEC has expired:- https://dnsviz.net/d/anchors.atlas.ripe.net/dnssec/ It also looks as if other things in ripe.net may also have expired (eg www.ripe.net when looking for a contact to advise of this). -- Best wishes, Matthew --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
