RE: dnssec-delegation seems to be broken from .gov to bls.gov
Point taken and understood. But you know how it is when there is major outage the push from upper management is always for "fix it now" and get us up and running do your RCA later. Thanks Sandeep -Original Message- From: Mark Andrews Sent: Wednesday, December 6, 2023 10:19 PM To: Bhangui, Sandeep - BLS CTR Cc: Nick Tait ; bind-users@lists.isc.org Subject: Re: dnssec-delegation seems to be broken from .gov to bls.gov CAUTION: This email originated from outside of BLS. DO NOT click (select) links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails through the "Phish Alert Report" button on your email toolbar. More to the point why was the old KSK removed *before* checking that the DS record for the new KSK was published and had been for the TTL of the DS RRset? With proper procedures this should not happen. When something goes wrong / is delayed in a key rollover the process should stall until that step is complete, not proceed blindly ahead. > On 7 Dec 2023, at 07:35, Bhangui, Sandeep - BLS CTR via bind-users > wrote: > > The problem has been resolved. > The automatic KSK rollover on the dotgov.gov did not happen properly and > once we manually updated the DS record with the correct KSK keytags and keys > things were fixed. > All is good now. > Now to see if we can find out as to why the automatic KSK failover on the > dotgov.gov did not happen correctly. > Thanks > Sandeep > From: bind-users On Behalf Of Nick > Tait via bind-users > Sent: Wednesday, December 6, 2023 3:23 PM > To: bind-users@lists.isc.org > Subject: Re: dnssec-delegation seems to be broken from .gov to bls.gov > CAUTION: This email originated from outside of BLS. DO NOT click (select) > links or open attachments unless you recognize the sender and know the > content is safe. Please report suspicious emails through the “Phish Alert > Report” button on your email toolbar. On 7/12/2023 9:05 am, Nick Tait via > bind-users wrote: > I could be wrong, but based on the output above it looks like the current TTL > is 0, which means that doing this should provide immediate relief. > Sorry it looks like the DNS server on the Wi-Fi network I'm connected to has > done something weird with the TTL. > This is what I get when querying one of the "gov." authoritative servers > directly: > $ dig -t ds bls.gov @a.ns.gov +norecurse > > ; <<>> DiG 9.18.18-0ubuntu2-Ubuntu <<>> -t ds bls.gov @a.ns.gov > +norecurse ;; global options: +cmd ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32241 ;; flags: qr > aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 > > ;; OPT PSEUDOSECTION: > ; EDNS: version: 0, flags:; udp: 1232 > ;; QUESTION SECTION: > ;bls.gov. IN DS > > ;; ANSWER SECTION: > bls.gov.3600IN DS 50951 8 2 > E6B0A294066904F20A2B8EBA3FA9920F9A1822802977F59D706B30A1 77F7DC0C > > ;; Query time: 16 msec > ;; SERVER: 2001:503:ff40::1#53(a.ns.gov) (UDP) ;; WHEN: Thu Dec 07 > 09:19:24 NZDT 2023 ;; MSG SIZE rcvd: 84 This means when you remove > the DS record, it will take 1 hour to fully take effect (assuming no delay > replicating between authoritative servers). > Nick. > -- > Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe > from this list > > ISC funds the development of this software with paid support subscriptions. > Contact us at https://www.isc.org/contact/ for more information. > > > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnssec-delegation seems to be broken from .gov to bls.gov
More to the point why was the old KSK removed *before* checking that the DS record for the new KSK was published and had been for the TTL of the DS RRset? With proper procedures this should not happen. When something goes wrong / is delayed in a key rollover the process should stall until that step is complete, not proceed blindly ahead. > On 7 Dec 2023, at 07:35, Bhangui, Sandeep - BLS CTR via bind-users > wrote: > > The problem has been resolved. > The automatic KSK rollover on the dotgov.gov did not happen properly and > once we manually updated the DS record with the correct KSK keytags and keys > things were fixed. > All is good now. > Now to see if we can find out as to why the automatic KSK failover on the > dotgov.gov did not happen correctly. > Thanks > Sandeep > From: bind-users On Behalf Of Nick Tait > via bind-users > Sent: Wednesday, December 6, 2023 3:23 PM > To: bind-users@lists.isc.org > Subject: Re: dnssec-delegation seems to be broken from .gov to bls.gov > CAUTION: This email originated from outside of BLS. DO NOT click (select) > links or open attachments unless you recognize the sender and know the > content is safe. Please report suspicious emails through the “Phish Alert > Report” button on your email toolbar. On 7/12/2023 9:05 am, Nick Tait via > bind-users wrote: > I could be wrong, but based on the output above it looks like the current TTL > is 0, which means that doing this should provide immediate relief. > Sorry it looks like the DNS server on the Wi-Fi network I'm connected to has > done something weird with the TTL. > This is what I get when querying one of the "gov." authoritative servers > directly: > $ dig -t ds bls.gov @a.ns.gov +norecurse > > ; <<>> DiG 9.18.18-0ubuntu2-Ubuntu <<>> -t ds bls.gov @a.ns.gov +norecurse > ;; global options: +cmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32241 > ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 > > ;; OPT PSEUDOSECTION: > ; EDNS: version: 0, flags:; udp: 1232 > ;; QUESTION SECTION: > ;bls.gov. IN DS > > ;; ANSWER SECTION: > bls.gov.3600IN DS 50951 8 2 > E6B0A294066904F20A2B8EBA3FA9920F9A1822802977F59D706B30A1 77F7DC0C > > ;; Query time: 16 msec > ;; SERVER: 2001:503:ff40::1#53(a.ns.gov) (UDP) > ;; WHEN: Thu Dec 07 09:19:24 NZDT 2023 > ;; MSG SIZE rcvd: 84 > This means when you remove the DS record, it will take 1 hour to fully take > effect (assuming no delay replicating between authoritative servers). > Nick. > -- > Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from > this list > > ISC funds the development of this software with paid support subscriptions. > Contact us at https://www.isc.org/contact/ for more information. > > > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: dnssec-delegation seems to be broken from .gov to bls.gov
The problem has been resolved. The automatic KSK rollover on the dotgov.gov did not happen properly and once we manually updated the DS record with the correct KSK keytags and keys things were fixed. All is good now. Now to see if we can find out as to why the automatic KSK failover on the dotgov.gov did not happen correctly. Thanks Sandeep From: bind-users On Behalf Of Nick Tait via bind-users Sent: Wednesday, December 6, 2023 3:23 PM To: bind-users@lists.isc.org Subject: Re: dnssec-delegation seems to be broken from .gov to bls.gov CAUTION: This email originated from outside of BLS. DO NOT click (select) links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails through the “Phish Alert Report” button on your email toolbar. On 7/12/2023 9:05 am, Nick Tait via bind-users wrote: I could be wrong, but based on the output above it looks like the current TTL is 0, which means that doing this should provide immediate relief. Sorry it looks like the DNS server on the Wi-Fi network I'm connected to has done something weird with the TTL. This is what I get when querying one of the "gov." authoritative servers directly: $ dig -t ds bls.gov @a.ns.gov +norecurse ; <<>> DiG 9.18.18-0ubuntu2-Ubuntu <<>> -t ds bls.gov @a.ns.gov +norecurse ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32241 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;bls.gov. IN DS ;; ANSWER SECTION: bls.gov.3600IN DS 50951 8 2 E6B0A294066904F20A2B8EBA3FA9920F9A1822802977F59D706B30A1 77F7DC0C ;; Query time: 16 msec ;; SERVER: 2001:503:ff40::1#53(a.ns.gov) (UDP) ;; WHEN: Thu Dec 07 09:19:24 NZDT 2023 ;; MSG SIZE rcvd: 84 This means when you remove the DS record, it will take 1 hour to fully take effect (assuming no delay replicating between authoritative servers). Nick. -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnssec-delegation seems to be broken from .gov to bls.gov
On 7/12/2023 9:05 am, Nick Tait via bind-users wrote: I could be wrong, but based on the output above it looks like the current TTL is 0, which means that doing this should provide immediate relief. Sorry it looks like the DNS server on the Wi-Fi network I'm connected to has done something weird with the TTL. This is what I get when querying one of the "gov." authoritative servers directly: $ dig -t ds bls.gov @a.ns.gov +norecurse ; <<>> DiG 9.18.18-0ubuntu2-Ubuntu <<>> -t ds bls.gov @a.ns.gov +norecurse ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32241 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;bls.gov. IN DS ;; ANSWER SECTION: bls.gov.3600IN DS 50951 8 2 E6B0A294066904F20A2B8EBA3FA9920F9A1822802977F59D706B30A1 77F7DC0C ;; Query time: 16 msec ;; SERVER: 2001:503:ff40::1#53(a.ns.gov) (UDP) ;; WHEN: Thu Dec 07 09:19:24 NZDT 2023 ;; MSG SIZE rcvd: 84 This means when you remove the DS record, it will take 1 hour to fully take effect (assuming no delay replicating between authoritative servers). Nick. -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnssec-delegation seems to be broken from .gov to bls.gov
On 7/12/2023 1:53 am, Bhangui, Sandeep - BLS CTR via bind-users wrote: Hi It seems the DNSSEC delegation is broken from “.gov” to bls.gov domain and due to which the records for bls.gov are considered as bogus and we are having issues at our site. It looks like we were in the process of KSK rollover and that may have caused the issue as things were fine till yesterday. As we troubleshoot this issue was wondering whether from our master DNS server can we use some option in named.conf so that dnssec verification is NOT done for any bls.gov DNS lookups from outside to get a quick fix to this problem. Currently DNS lookups from outside are flaky and I believe the reason behind that being that the DNSSEC delegation is broken. From the output at dnsviz.net analyzing for bls.gov it seems that KSK rollover for bls.gov is the issue. Basically, trying to see if I can get a quick interim fix till we resolve the issue correctly. Please advise. Thanks Sandeep Hi Sandeep. Probably the simplest workaround for broken chain of trust would be to remove your zone's DS records from the parent zone. $ dig -t ds bls.gov ; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> -t ds bls.gov ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27975 ;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;bls.gov. IN DS ;; ANSWER SECTION: bls.gov. 0 IN DS 50951 8 2 E6B0A294066904F20A2B8EBA3FA9920F9A1822802977F59D706B30A1 77F7DC0C ;; Query time: 0 msec ;; SERVER: 172.20.192.1#53(172.20.192.1) (UDP) ;; WHEN: Thu Dec 07 09:01:33 NZDT 2023 ;; MSG SIZE rcvd: 80 I could be wrong, but based on the output above it looks like the current TTL is 0, which means that doing this should provide immediate relief. Add a new DS record once you've fixed your KSK issues. Nick. -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
