Re: [dns-operations] Destination-adjacent source address spoofed DNS queries
--- Begin Message --- On Tue, Mar 5, 2024 at 2:24 PM John Kristoff wrote: > This seems DNS operationally relevant and I hope no one will mind the > plug. It was fun to write up a small piece on some curious spoofed DNS > queries we observed. Something that probably would have been overlooked > otherwise. We could probably do this 24x7. :-) > > <https://open.substack.com/pub/dataplane/p/destination-adjacent-source-address> > > John That's a fascinating development. I hope we learn more. I like reading DNS research but I don't always like reading the logs researchers generate. :-D (Never read your DNS logs, you don't want to know. ;-) For what it's worth, I saw something similar -- but distinctly different -- in 2021-2022 using the domain dnsdrakkarv6.com. They'd send two similar queries, one from a real-looking IP, and the other spoofed with my IP + 1. For example (very edited and redacted): 2021-03-22 IP6 2001:db8::2.1234 > 2001:db8::1.53: 5678+ A? xXXxXX.[38 decimal digits].s67.vp1.v6.dnsdrakkarv6.com. (91) 2021-03-22 IP6 2001:912:800:212::61.1234 > 2001:db8::1.53: 5678+ A? xXXxXX.[38 decimal digits].n67.vp1.v6.dnsdrakkarv6.com. (91) 2022-02-24 IP6 2001:db8:0:2::.1234 > 2001:db8:0:1::::.53: 5678+ A? xxXxXx.[38 decimal digits].s128.vp2.v6.dnsdrakkarv6.com. (92) 2022-02-24 IP6 2a10:a080:1100:1000::1.1234 > 2001:db8:0:1::::.53: 5678+ A? xxXxXx.[38 decimal digits].n128.vp2.v6.dnsdrakkarv6.com. (92) -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] cmdns.dev.dns-oarc.net down?
--- Begin Message --- On Mon, Sep 4, 2023 at 12:29 PM Christoph via dns-operations wrote: > Hello! > > https://dnsviz.net/d/cmdns.dev.dns-oarc.net/dnssec/ > > since cmdns.dev.dns-oarc.net appears to be down, > what are other good websites that show end users their recursive > resolver source IPs? > > best regards, > Christoph A few weeks ago, I went looking for a couple websites I remembered -- didn't CZ.NIC or SIDN run one? -- but couldn't find any of them. I did find <https://dnscheck.tools/>, which looks pretty neat, though I might just be a sucker for fixed-size fonts. :-) -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region
--- Begin Message --- On Tue, Jul 18, 2023 at 7:53 PM Gavin McCullagh wrote: > On Tue, Jul 18, 2023 at 12:45 PM Shumon Huque wrote: >> Yes, I agree. A resolver can't really tell that a response with an expired >> signature wasn't an attacker trying to replay old data. For robustness >> against attacks, it must re-query other available other servers if they >> exist. >> >> Also, I was under the impression that most resolvers already had this robust >> behavior. Since Unbound was mentioned, I just tested an unbound resolver >> against a test DNS record that I have provisioned with an intentionally >> expired DNSSEC signature - it sent queries to all 4 servers for the zone >> before giving up and returning SERVFAIL. > > > Interesting. As I understand it, in the event we're talking about, 4/13 > nameservers would have been stale - so it might be that it did retry but not > enough to work around the problem. We definitely saw Unbound returning > SERVFAIL for unsigned com domains though. I didn't get around to retesting > the specific circumstances yet, but if Unbound already retries on this, then > we can just work to understand the details better. > > Gavin My past experience with Unbound (a few years ago) was that it very aggressively tried every nameserver when it encountered problems, DNSSEC or otherwise. If someone had asked me a month ago, I would have joked that the only way it would have returned SERVFAIL is if your Unbound clusters DDoSed Verisign offline. :-) This is only a guess, but maybe it hit a limit like the new max-sent-count setting? For a while now, resolvers have been focusing on limiting queries for things like the NXNSAttack, which works against your goal here. With all due respect to Verisign and their highly-provisioned servers, that is a real concern. It doesn't help anything when a zone is completely bogus *and* the zone and its parents are getting 1000x their normal query volume in retries. Maybe you hit an edge case or Unbound's defaults should be tuned a little differently? -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region
--- Begin Message --- On Tue, Jul 18, 2023 at 6:21 PM Gavin McCullagh wrote: > Hi, > > sorry to dredge this back up, but I just want to give anyone the chance to > object. > > My read of what Viktor and others have indicated here is that, when a > validating resolver receives a response with expired rrsigs, it's okay (and > encouraged?) for that resolver to treat that as an invalid response and retry > against other nameservers, similarly to how it would handle a REFUSED or > SERVFAIL response from an authority (i.e. with similar care to limit retry > storms). > > The purpose of this is so that a single stale pop or authoritative host would > not cause an outage to dnssec signed domains, as resolvers will retry against > others. > > I'd like to reach out to NLNet about changing Unbound to do this, so I want > to make sure people have a chance to disagree. Feel free to voice your > disagreement (and reasons) here if you do. > > Gavin This is just a comment, but I've reported TLD secondary nameservers with expired RRSIGs ~4 different times.¹ ² ³ I never would have noticed most problems if I had been using a resolver that retried other authoritative nameservers for DNSSEC issues. Who knows how long it would have taken for the problems to have been discovered or fixed. Of course it's good for resolution to succeed, but it also papers over problems and causes them to linger forever. (Just like happens with other DNS problems now, like a nameserver timing out or returning SERVFAIL.) ¹ And also in-addr.arpa/ip6.arpa. ² And that time one of the root servers borked arpa zone apex queries. ³ Actually the xn--wgbh1c TLD is broken right now but I haven't told anyone. -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Enabling DNSSEC signing for pagerduty.com
--- Begin Message --- On Wed, Jun 7, 2023 at 6:09 AM Viktor Dukhovni wrote: > On Tue, Jun 06, 2023 at 11:00:29AM -0600, Andy Smith via dns-operations wrote: > > We (PagerDuty) are in the process of enabling DNSSEC signing across > > our domains, and today (June 6th) we’re planning to enable it for > > pagerduty.com and associated subdomains (e.g. eu.pagerduty.com). Given > > the potential impact and the large number of organizations using our > > services, we thought it would be a good idea to let people know it’s > > happening in case any problems occur. If you do see any issues, feel > > free to contact me directly or support via supp...@pagerduty.com. > > According to DNSViz: <https://dnsviz.net/d/pagerduty.com/ZIAdCA/dnssec/> > sibling glue is missing for ns-474.awsdns-59.com. This is not > something pagerduty.com can do anything about (except pass it along to > AWS). Since I am not much of a fan of sibling glue anyway, not a > problem really, but if there is IPv4 glue for an NS host it should > also have IPv6 glue if the auth RR exists. I tried to ask about that on the old AWS forum but Amazon didn't really understand me. And then they deleted the forum. :-( It's resolvable with extra queries on an IPv6-only network because the awsdns-59.com zone itself uses nameservers with IPv6 glue (unless those nameservers are all down) but it's still unfortunate. They have some other issues outside of the Route 53 service, too. (E.g. ns1.amzndns.com or ns1-ec2-rdns.amazonaws.com.) -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Enabling DNSSEC signing for pagerduty.com
--- Begin Message --- On Tue, Jun 6, 2023 at 5:46 PM Andy Smith via dns-operations wrote: > Hi all, > > We (PagerDuty) are in the process of enabling DNSSEC signing across > our domains, and today (June 6th) we’re planning to enable it for > pagerduty.com and associated subdomains (e.g. eu.pagerduty.com). Given > the potential impact and the large number of organizations using our > services, we thought it would be a good idea to let people know it’s > happening in case any problems occur. If you do see any issues, feel > free to contact me directly or support via supp...@pagerduty.com. > > As an aside, we’ve talked with a few other organizations that have > enabled DNSSEC signing and have gotten a lot of useful information as > a result. We’d be more than happy to hear from other people who have > gone through the process and also to share what we’ve learned in the > future in case it helps anyone else! > > Cheers, > > Andy. Hold on, did y'all skip a step? <https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-dnssec-enable-signing.html>: > 2. Wait for at least the previous zone’s maximum TTL. > >Wait for resolvers to flush all unsigned records from their cache. To > achieve this you should wait for at least the previous zone’s maximum TTL. In > the example.com zone above, the wait time would be 1 day. I think the zone was signed and then the DS record was added within no more than about 2 hours? But the pagerduty.com./NS record set TTL is 172,000. But most or all other records have TTLs of no more than 300? It's probably fine but maybe not? Clients don't normally look up NS records, but I don't know if there are cases where a recursive resolver's internal logic could notice or care? [ABSOLUTELY DO NOT PANIC AND UNSIGN THE ZONE. YOU CAN PANIC AND DELETE THE DS RECORD BUT FOR THE LOVE OF GOD DO NOT UNSIGN THE ZONE.] -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] ENT NXDOMAIN problem at .BS nameserver ns36.cdns.net
--- Begin Message --- On Thu, Sep 22, 2022 at 9:17 AM Warren Kumari wrote: > [ - bs ] > > There is a very similar issue with 'production.cloudflare.docker.com' > (https://dnsviz.net/d/production.cloudflare.docker.com/dnssec/): > > A query for production.cloudflare.docker.com results in a NOERROR response, > while a query for its ancestor, cloudflare.docker.com, returns a name error > (NXDOMAIN), which indicates that subdomains of cloudflare.docker.com, > including production.cloudflare.docker.com, don't exist. > > This broke my ability to use docker for a while — I'd enabled strict qname > minimization as a test, and then needed to update some containers in an > emergency. It took a while to debug the issues… > > W That's Amazon Route 53 for you. There were at least 2 threads about ENTs on the old AWS forum (one started by yours truly) before they got rid of it. IIRC, they were reluctant to fix it because they were concerned that changing (correcting) ENT wildcard behavior would break things for some of their users. At least one AWS team has deployed the other "fix", a pointless TXT record: $ dig elb.amazonaws.com txt (In signed responses, Route 53 uses NSEC black lies. ENTs are handled appropriately.) -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Opportunity to operate a non-gTLD non-ccTLD TLD
--- Begin Message --- On Fri, Jul 8, 2022 at 9:47 PM John R Levine wrote: > >> The incumbent is Verisign. Any reason to believe they are looking to > >> replace them? > > > > It might just be the USG has to go through a procurement process for this > > sort of thing every few years. > > Sure. Just wondering if there are reasons that the USG might be unhappy with > VRSN. > > The RFP says they currently do not provide DNS service to registrants but > want the new contractor to do so, to make the DNS more secure and > reliable. I would imagine VRSN could do that. It would have been easier before Verisign sold that whole business line to Neustar, but sure. :-D -- Matt Nordhoff --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] dynect.net outage
On Mon, May 30, 2022 at 6:48 AM Ralf Weber wrote: > Moin! > > On 30 May 2022, at 8:34, Robert Edmonds wrote: > >> So how do you expect the domain to be resolved if all of your out > >> of bailiwick name server names no longer point to an IP address? > > > > By using the working nameservers with resolvable names specified in the > > delegation from the parent zone, which never changed in this particular > > case. This is what Unbound's resolution algorithm does if there are not > > too many nonexisting nameserver target names in the child's NS RRset, > > and what other resolver algorithms do. > So you mean the parent provided additional records (A/) when issuing > a referral? Otherwise I can not see how from an empty cache you can > resolve this domain if all of the name server names supplied are NXDOMAIN. > > > There is more than one resolver implementation, and they differ in the > > results of resolving a zone with this type of misconfiguration, and none > > of them are the reference implementation of DNS. So just looking at a > > particular resolver algorithm returning SERVFAIL when encountering a > > particular data pattern starting from a cold cache cannot tell us > > whether the algorithm or the data is at fault. > I agree on this, however the difference in implementation are less > when it comes to resolving from a cold cache and all the explanations > given so far for me point to the domain being unresolvable for all > implementations from an empty cache. > > So long > -Ralf > ——- > Ralf Weber The domain was always resolvable from a cold cache because the delegation was always* correct. <https://dnsviz.net/d/dynect.net/YpA8MQ/dnssec/> The outage happened when the authoritative NS records were changed to *completely different* names that do not exist. A totally parent-centric resolver would never have noticed anything wrong. The "bug" in Unbound is that, in this precise error situation, it apparently returns SERVFAIL before trying to fall back on the parent-side NS records. * As a separate issue, half of the nameservers have out-of-date glue records with IPs that don't respond. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Input from dns-operations on NCAP proposal
itional layer of likely caching difficulties, potential resolution failure and name collisions. > Configuration 3: Use a properly configured empty zone with correct NS and SOA > records. Queries for the single label TLD would return a NOERROR and NODATA > response. > > > > The level of disruption to existing private use of such labels by this > restricted form of name delegation would be reasonably expected to be > minimal; however, the series of referrals and responses received by resolvers > are different from a direct NXDOMAIN response from the root server system and > deviate from the DNS protocol. It is possible that even this slight > difference could impact application resolution processes, such as search list > processing. The NCAP would appreciate any technical insights from a risk > perspective the community may be able to provide regarding the proposal. > > > > Best, > > > > Matt Thomas > > NCAP Co-chair -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] TLD .law - non-signing KSK with referenced DS
On Wed, Jan 19, 2022 at 1:34 PM Einar Bjarni Halldórsson wrote: > On 14.1.2022 10:30, Viktor Dukhovni wrote: > > On Fri, Jan 14, 2022 at 10:09:04AM +, Matthew Richardson wrote: > > > >> Looking visually at the DNSViz output, the KSK 16819 does look strange as > >> it is referenced by a DS but does not sign anything. > >> > >> Out of interest, do folks think this is a valid configuration? > > Looks valid to me, because another KSK for the same algorithm and > > choice of hash does sign the DNSKEY RRset: > I thought it was just the same algorithm, not necessarily the same hash > type? > > We're finishing up a test migration of a signed zone, doing a key > rollover, and the old DS record is algorithm 8, digest type 2. The new > key has two DS records, both algorithm 8, one digest type 2, one type 4. > > We saw the error in zonemaster, but DNSviz and probes in RIPE Atlas > never flagged an error. > > .einar RFC 4509 (SHA-256 DS) section 3 [0] says: "Validator implementations SHOULD ignore DS RRs containing SHA-1 digests if DS RRs with SHA-256 digests are present in the DS RRset." Skimming RFC 6605 [1] -- which was nominally about ECDSA DNSKEYs but also added SHA-384 DS -- I don't see anything directly about that, but it does incorporate 4509's Security Considerations [2], which are entirely about the DS downgrade issue. PowerDNS Recursor used to ignore SHA-256 records in the face of SHA-384 records, but this was considered a bug and recently fixed. [3] I don't know if any other resolvers behave the same way. It would be prudent not to chance it. [0]: <https://datatracker.ietf.org/doc/html/rfc4509#section-3> [1]: <https://datatracker.ietf.org/doc/html/rfc6605> [2]: <https://datatracker.ietf.org/doc/html/rfc4509#section-6> [3]: <https://github.com/PowerDNS/pdns/pull/10908> -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Obsoleting 1024-bit RSA ZSKs (move to 1280 or algorithm 13)
This email is technically a reply to Viktor, but it's not really a reply to anyone in particular. There are challenges to upgrading to larger keys, or rolling algorithms, but how insurmountable are they? There are at least 139 TLDs using 2048-bit or larger DNSSEC keys *right now* -- and that's excluding about two dozen that are in the middle of migrating to a different registry and downgrading to 1280-bit ZSKs. In mid-2020, one registry was doing some kind of migration and left a few hundred TLDs entirely double-signed with 1280-bit RSA for a couple of months. They were paying double their usual cryptography cost, and the TCP cost (which was, granted, lower pre-Flag Day), and the fragmentation cost. Whatever expenses, deployment and planning difficulties there are, all of these registries were able to handle them. Many commercial DNS operators are multi-hundred-million or even multi-billion dollar companies. Some of them could probably afford to buy literally every HSM in the world. A few wouldn't even put a dent in their quarterly numbers. And if they want to get out of the industry, their customers can still go elsewhere (despite the last decade of consolidation). -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] .NL and 1024-bit RSA ZSKs.
On Fri, Oct 8, 2021 at 5:55 PM Viktor Dukhovni wrote:
> > On 8 Oct 2021, at 1:12 pm, Puneet Sood via dns-operations
> > wrote:
> >
> > This is another case where NSEC3 opt-out interferes with effective
> > NSEC{3} response caching which would reduce queries to the TLD.
>
> Speaking of the .NL zone DNSSEC parameters, the ZSK is 1024-bit RSA,
> and .NL is the largest zone (by signed delegation count) with RSA
> keys less than 1280 bits.
>
> The .COM TLD uses 1280-bit RSA ZSKs, while .BR, .CZ, .CH, .FR and .DK
> all use ECDSA P256.
>
> The next batch of TLDs with 1024-bit RSA ZSKs are .EU, .NO, .BE and .ORG.
>
> While we don't have compelling evidence that 1024-bit RSA DNSKEYs,
> rotated sufficiently often are at a realistic risk of brute-force
> cryptanalytic attacks, the broader cryptographic community has
> left 1024-bit RSA behind, and we now have better options:
>
> * 1280-bit RSA is practical and improves the safety margin
> * P256 has been successfully adopted by 45 TLDs and has
> near universal resolver support, on par with RSA.
>
> So I'd like to suggest that .NL consider either a stronger ZSK,
> or an algorithm rollover.
>
> Not all is stuck in the past, over the last ~1 year, the use of
> algorithm 7 has dropped from a peak of ~2.2 million zones to
> just ~350k zones and lately continuing to fall ~10k/day.
>
> So progress is possible, it just does not happen on all fronts
> at the same time.
>
> For those not yet caught up on last-night's OARC "Town Square"
> Mattermost channel, it would be good to have auth operators
> look more closely at their use of RSA and as needed move to a
> set of best-practice key algorithms/sizes.
>
> RSA: 2048-bit KSK, 1280 or 1536-bit ZSK
> P256: Fortunately no further tunables
A couple hundred TLDs even use 2048-bit RSA ZSKs with success. It's
not as efficient, but it must be practical enough, since people in
fact do it.
(Resolvers with poor TCP handling might not appreciate it if .com ever
deployed it, though...)
> The only potential tweak in ECDSA is whether signatures use
> a random nonce, or a deterministic variant that derives the
> nonce from the message:
>
> https://datatracker.ietf.org/doc/html/rfc6979
>
> Deterministic ECDSA signing should be well suited for
> zone signing when the software stack supports it, and
> can be more performant if the RNG is a bottleneck.
>
> [ I don't know which HSMs, if any, support deterministic
> ECDSA signing. ]
--
Matt Nordhoff
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Lot's of TXT queries from Google
On Thu, Oct 7, 2021 at 11:53 AM Moritz Müller via dns-operations wrote: > Hi, > > For the second time in a few weeks we noticed a significant increase in > queries for NS and TXT records at our .nl name servers, originating almost > exclusively from the Public DNS resolvers of Google > Did someone else noticed something similar or has an explanation? > > In comparison to beginning of September, the number of NS queries increased 2 > fold and the number of TXT queries almost 6 fold. > At some point, 25% of all queries to our name servers for .nl where for TXT > record. > > The resolvers query either for a domain name following the pattern > _dmarc.foo.nl or default._domainkey.foo.nl. > Where foo is a random string, 12 characters long. > > Examples are: > _dmarc.mdvlxtagogij.nl. > default._domainkey.vppj4svmbclt.nl. > > The queried second level domain names are not registered and queries for the > same domain name are repeated 3 to 5 times. > At some point, 80% of all TXT queries from google had these patterns, 36% of > all queries from Google resolvers. > > The queries started ramping up around 2021-09-05 and reached their peak at > 2021-09-18. They never reached a concerning level, but we first noticed them > because our machine processing the incoming PCAP files couldn’t cope anymore. > > We assume that this is likely not an attack but some tests/measurements, > which got a bit out of hand. But since we don’t see the origin of the queries > behind the Google resolvers, we’re not sure to whom to reach out. From another perspective, I own some domains in a different ccTLD, and they get a constant low volume of similar DNS queries, and daily DMARC reports from major mail providers showing that spam is being sent from spoofed random subdomains of my domains. It's mostly died down over the last week. Maybe the spammers switched to .nl? -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Registrars supporting ED25519
On Sat, Jul 31, 2021 at 5:53 PM Eric Germann via dns-operations wrote: > I’d like to publish NS records into .dev but Namecheap, AWS and name.com > won’t let you select algorithm 15 What's going wrong with Name.com? The algorithm drop-down in their DS record manager includes algorithm 15 (and algorithm 16). It's certainly *supposed* to work, last I heard. I'm not sure I've ever *used* algorithm 15 with Name.com, though. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Inconsistent NSEC response for unsigned zone from AWS
On Tue, Jun 22, 2021 at 3:54 AM Viktor Dukhovni wrote: > On Tue, Jun 22, 2021 at 03:30:39AM +0000, Matt Nordhoff wrote: > > > > Indeed I see the same: > > > > > > $ dig +noall +dnssec +norecur +nocrypto +ans +auth +add -t NS > > > corp.ibexglobal.com @ns-725.awsdns-26.net. > > > corp.ibexglobal.com.172800 IN NS ns-1415.awsdns-48.org. > > > corp.ibexglobal.com.172800 IN NS > > > ns-1804.awsdns-33.co.uk. > > > corp.ibexglobal.com.172800 IN NS ns-29.awsdns-03.com. > > > corp.ibexglobal.com.172800 IN NS ns-945.awsdns-54.net. > > > corp.ibexglobal.com.86400 IN NSEC > > > \000.corp.ibexglobal.com. RRSIG NSEC > > > corp.ibexglobal.com.86400 IN RRSIG NSEC 13 3 86400 > > > 20210623012420 20210621232420 36517 ibexglobal.com. [omitted] > > > > > > This violates <https://datatracker.ietf.org/doc/html/rfc4035#section-2.3>: > > > > I haven't spent the time to understand precisely what this thread is > > talking about, but that's how NSEC white/black lies work. NS1 does the > > same thing (give or take possible bugs) as AWS. > > No, there's a subtle difference, this qname actually exists, and has an > NS RRSet. The NSEC bitmap needs to reflect this. Ah, now I understand the issue, thank you. I'm sorry if my previous email seemed glib. I didn't get enough sleep to easily understand NSEC problems, but thought it was probably better to reply now than to wait. My email might still matter, but it's not relevant to this thread. [FWIW, Cloudflare handles insecure referrals correctly, AFAIK. I have no idea about NS1, but there's no reason to suspect anything.] -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Inconsistent NSEC response for unsigned zone from AWS
On Tue, Jun 22, 2021 at 12:36 AM Viktor Dukhovni wrote: > On Mon, Jun 21, 2021 at 07:45:44PM -0400, Puneet Sood wrote: > > > $ dig corp.ibexglobal.com -t NS +dnssec +nocrypto +nocomment > > @ns-725.awsdns-26.net. > > > > ;corp.ibexglobal.com. IN NS > > corp.ibexglobal.com.172800 IN NS ns-1415.awsdns-48.org. > > corp.ibexglobal.com.172800 IN NS ns-1804.awsdns-33.co.uk. > > corp.ibexglobal.com.172800 IN NS ns-29.awsdns-03.com. > > corp.ibexglobal.com.172800 IN NS ns-945.awsdns-54.net. > > corp.ibexglobal.com.86400 IN NSEC\000.corp.ibexglobal.com. > > RRSIG NSEC > > corp.ibexglobal.com.86400 IN RRSIG NSEC 13 3 86400 > > 20210623002754 20210621222754 36517 ibexglobal.com. [omitted] > > Indeed I see the same: > > $ dig +noall +dnssec +norecur +nocrypto +ans +auth +add -t NS > corp.ibexglobal.com @ns-725.awsdns-26.net. > corp.ibexglobal.com.172800 IN NS ns-1415.awsdns-48.org. > corp.ibexglobal.com.172800 IN NS ns-1804.awsdns-33.co.uk. > corp.ibexglobal.com.172800 IN NS ns-29.awsdns-03.com. > corp.ibexglobal.com.172800 IN NS ns-945.awsdns-54.net. > corp.ibexglobal.com.86400 IN NSEC\000.corp.ibexglobal.com. > RRSIG NSEC > corp.ibexglobal.com.86400 IN RRSIG NSEC 13 3 86400 > 20210623012420 20210621232420 36517 ibexglobal.com. [omitted] > > This violates <https://datatracker.ietf.org/doc/html/rfc4035#section-2.3>: > >... > >An NSEC record (and its associated RRSIG RRset) MUST NOT be the only >RRset at any particular owner name. That is, the signing process >MUST NOT create NSEC or RRSIG RRs for owner name nodes that were not >the owner name of any RRset before the zone was signed. The main >reasons for this are a desire for namespace consistency between >signed and unsigned versions of the same zone and a desire to reduce >the risk of response inconsistency in security oblivious recursive >name servers. > >... > >The bitmap for the NSEC RR at a delegation point requires special >attention. Bits corresponding to the delegation NS RRset and any >RRsets for which the parent zone has authoritative data MUST be set; >bits corresponding to any non-NS RRset for which the parent is not >authoritative MUST be clear. I haven't spent the time to understand precisely what this thread is talking about, but that's how NSEC white/black lies work. NS1 does the same thing (give or take possible bugs) as AWS. Cloudflare's implementation is slightly different but also has NSEC records like this. (Cloudflare's "NSEC shotgun" 'NODATA' records put every record type they support in the bitmap; their 'NXDOMAIN' records just set NSEC and RRSIG). I note that RFC 7129 is informational and updates no other RFCs. $ dig +dnssec nxdomain.cloudflare.com | awk '$4 == "NSEC"' nxdomain.cloudflare.com. 300 IN NSEC \000.nxdomain.cloudflare.com. RRSIG NSEC $ dig +dnssec nxdomain.ns1.com | awk '$4 == "NSEC"' nxdomain.ns1.com. 3600IN NSEC\000.nxdomain.ns1.com. RRSIG NSEC -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anybody awake over at comcast.net?
On Wed, Feb 10, 2021 at 3:47 AM Peterson (AWS), Alec wrote: > Hey Matt, sorry we missed your forum post, we do need to stay on top of those. > > There are actually 2 hours of overlap, not 1. For the zone I just provided, > here is the previous and current RRSIG: > > DNSKEY 13 2 3600 2021020901 2021020815 38680 hilander.com. > J5WR2nU1Gl3xI5ehHBsnI7OXiThuYZKc1XpV6brYf85BgurOcZkb5+6eLbsvV+ykODaPrEnEIDu/HRYcaIRrNg== > DNSKEY 13 2 3600 2021020909 2021020823 38680 hilander.com. > 735uL43A++phhPv/edwq02zANvAfEsas1j9HM+ghe+t7b6FLCAF6RZfXA9L+TBukYmhIkPiF87GbHDWXmETBNQ== > > So when we roll the signature over, the one that we were previously using > still has 2 hours of validity, not 1 hour. So using your example, we would > roll it over at 2300, not , giving us room for 1 hour of clock skew. > Think that’s reasonable. > > Alec I haven't looked recently, but when I checked some poor customer's domain at 2020-12-31 23:59:59, that wasn't the case. It returned the 2020-12-31 15:00:00 - 2021-01-01 01:00:00 records: <https://gist.github.com/mnordhoff/02e173d419387c3098cd0f90c7b75f34> If the record rollover has been moved back one hour, the inception should correspondingly be moved back to 06:00:00 / 14:00:00 / 22:00:00, though. :-D > On Feb 9, 2021, at 7:37 PM, Matt Nordhoff wrote: > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > On Wed, Feb 10, 2021 at 1:44 AM Peterson (AWS), Alec via > dns-operations wrote: > > +1 for short RRSIG times and the discipline it enforces. We went down this > path when building DNSSEC for Route 53, ZSK signatures are on the order of 10 > hours: > > hilander.com. 3599 IN RRSIG DNSKEY 13 2 3600 2021021009 ( > 2021020923 38680 hilander.com. > 3H3QZt3qC2XbqkbumqsvRVeVrtgJVVRVGC/TZkc7vMuN > IdlL/wZrw+qBfYaSOex7dOp2PUP7pwW+NUgCXc2F7Q== ) > > A bunch of risks with this approach that needs to be mitigated, especially > around static stability in the face of an issue with the ZSK signing process. > But all solvable. As part of this we also automated ZSK rotation (which > happens less often, but still on the order of once a week). > > > Speaking of which, the DNSKEY RRSIG lifetime should be 11 hours, not > 10 hours. The current expiration doesn't allow for client caching plus > clock skew. E.g.: > > 23:59:59: Recursive resolver queries authoritative servers for > hilander.com DNSKEY, gets back record set with TTL 3600, RRSIG > inception 15:00:00, expiration 01:00:00. > (00:00:00: Authoritative servers switch to new DNSKEY record set, TTL > 3600, inception 23:00:00, expiration 09:00:00.) > 00:59:58: Validating stub resolver queries recursive resolver for > hilander.com DNSKEY. gets back cached record set with TTL 1, RRSIG > inception 15:00:00, RRSIG expiration 01:00:00. > > If the stub resolver's clock is 3 seconds fast -- 01:00:01 -- and it > doesn't make an extra allowance for expired records or clock skew, it > will treat the record set as bogus. > > (All other records except for DNSKEY are live signed, with the RRSIG > expiration set 1 hour after the TTL.) > > <https://forums.aws.amazon.com/thread.jspa?threadID=05=0> > > On Feb 8, 2021, at 9:27 PM, Paul Vixie wrote: > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > On Mon, Feb 08, 2021 at 01:45:06AM -0500, Viktor Dukhovni wrote: > > ... > I do not recommend either X.509 certificate or RRSIG lifetimes quite > this long. Shorter lifetimes IMHO promote better discipline. > > > for my own zones i think i'm using one year signatures and regenerating them > from "cron" once per week -- just to be safe. so, not better discipline unless > you deliberately _live_ on the edge, which i think is an unwise practice. > > i expect i'll crib together some bourne shellack to check my whole signature > chains and warn me when there's less than 72 hours remaining in any validity > period. going into SERVFAIL like this is an operational risk i shouldn't take. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] anybody awake over at comcast.net?
On Wed, Feb 10, 2021 at 1:44 AM Peterson (AWS), Alec via dns-operations wrote: > +1 for short RRSIG times and the discipline it enforces. We went down this > path when building DNSSEC for Route 53, ZSK signatures are on the order of 10 > hours: > > hilander.com. 3599 IN RRSIG DNSKEY 13 2 3600 2021021009 ( > 2021020923 38680 hilander.com. > 3H3QZt3qC2XbqkbumqsvRVeVrtgJVVRVGC/TZkc7vMuN > IdlL/wZrw+qBfYaSOex7dOp2PUP7pwW+NUgCXc2F7Q== ) > > A bunch of risks with this approach that needs to be mitigated, especially > around static stability in the face of an issue with the ZSK signing process. > But all solvable. As part of this we also automated ZSK rotation (which > happens less often, but still on the order of once a week). Speaking of which, the DNSKEY RRSIG lifetime should be 11 hours, not 10 hours. The current expiration doesn't allow for client caching plus clock skew. E.g.: 23:59:59: Recursive resolver queries authoritative servers for hilander.com DNSKEY, gets back record set with TTL 3600, RRSIG inception 15:00:00, expiration 01:00:00. (00:00:00: Authoritative servers switch to new DNSKEY record set, TTL 3600, inception 23:00:00, expiration 09:00:00.) 00:59:58: Validating stub resolver queries recursive resolver for hilander.com DNSKEY. gets back cached record set with TTL 1, RRSIG inception 15:00:00, RRSIG expiration 01:00:00. If the stub resolver's clock is 3 seconds fast -- 01:00:01 -- and it doesn't make an extra allowance for expired records or clock skew, it will treat the record set as bogus. (All other records except for DNSKEY are live signed, with the RRSIG expiration set 1 hour after the TTL.) <https://forums.aws.amazon.com/thread.jspa?threadID=05=0> > On Feb 8, 2021, at 9:27 PM, Paul Vixie wrote: > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > On Mon, Feb 08, 2021 at 01:45:06AM -0500, Viktor Dukhovni wrote: > > ... > I do not recommend either X.509 certificate or RRSIG lifetimes quite > this long. Shorter lifetimes IMHO promote better discipline. > > > for my own zones i think i'm using one year signatures and regenerating them > from "cron" once per week -- just to be safe. so, not better discipline unless > you deliberately _live_ on the edge, which i think is an unwise practice. > > i expect i'll crib together some bourne shellack to check my whole signature > chains and warn me when there's less than 72 hours remaining in any validity > period. going into SERVFAIL like this is an operational risk i shouldn't take. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] .com delegation responses when glue addresses don't fit
On Wed, Aug 19, 2020 at 5:49 PM Mukund Sivaraman wrote: > We notice the following response from .com's namesevers: > > [muks@mx ~]$ dig +nord +dnssec +bufsize=512 @2001:502:1ca1::30 infoblox.com > > ; <<>> DiG 1.1.1.20200608151533.e8a2352e96 <<>> +nord +dnssec +bufsize=512 > @2001:502:1ca1::30 infoblox.com > ; (1 server found) > ;; global options: +cmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15448 > ;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 11, ADDITIONAL: 1 > > ;; OPT PSEUDOSECTION: > ; EDNS: version: 0, flags: do; udp: 4096 > ;; QUESTION SECTION: > ;infoblox.com. IN A > > ;; AUTHORITY SECTION: > infoblox.com. 172800 IN NS ns1.infoblox.com. > infoblox.com. 172800 IN NS ns2.infoblox.com. > infoblox.com. 172800 IN NS ns3.infoblox.com. > infoblox.com. 172800 IN NS ns4.infoblox.com. > infoblox.com. 172800 IN NS ns5.infoblox.com. > infoblox.com. 172800 IN NS ns6.infoblox.com. > infoblox.com. 86400 IN DS 33613 5 2 > 339462CBAEB1773800EA8B688D2CA048FCAB0EB2933A97AEE2B86A9A 212F37C5 > infoblox.com. 86400 IN DS 33613 5 1 > 629C2D6C060E2133CD0F4470F3ECC8834DA4FAD6 > infoblox.com. 86400 IN DS 49879 5 2 > 605656DB7C9DFE4D8A453C350B3DA63039A78878DA089AD4247AB9A0 D3B43998 > infoblox.com. 86400 IN DS 49879 5 1 > C1DB78AD9A8928CB15A7E0CE9E4468D433F5C638 > infoblox.com. 86400 IN RRSIG DS 8 2 86400 20200823050241 > 20200816035241 24966 com. > 0s/TnWuxLdVzCQqY0tVauNXeCpirT5rYacvEpxaQfTxCjP2XfZkqHy4A > SNoGyYWGZQdxTa7zXVgrKuWOoKZ2CKxC/kd++VnEJKoFw3llOoq56Wz+ > lq65BS7E6/ZlE4Qgce8rhbBQVkE6Sk1YXkuxDbwoPYfvkHlfWaboeiNO > 6y731Xcrq3vjqdG6YZCHyH64SSnVFypUiRN26H2HPsYsSg== > > ;; Query time: 19 msec > ;; SERVER: 2001:502:1ca1::30#53(2001:502:1ca1::30) > ;; WHEN: Wed Aug 19 17:30:29 GMT 2020 > ;; MSG SIZE rcvd: 512 > > [muks@mx ~]$ > > > Glue address records are required in this delegation response, but none > are returned. TC=1 is not set. This causes problems with some resolvers. > > Can someone at Verisign please check correctness of this response, and > set TC=1 for such responses? > > It appears to be the problem statement of: > https://tools.ietf.org/html/draft-andrews-dnsop-glue-is-not-optional-01 > > Mukund There was a previous thread about this issue: <https://lists.dns-oarc.net/pipermail/dns-operations/2019-January/018259.html> (The domain mentioned in that post is no longer problematic with typical EDNS buffer sizes; you have to try 700 bytes or so.) Verisign didn't respond at the time, unfortunately. IIRC, someone from Verisign might have commented on draft-andrews-dnsop-glue-is-not-optional on the dnsop list. I'd have to check. If someone has a contact with them, that might be better. TLDs operated by other organizations were also affected to a lesser extent -- I don't know of any affected domains in the wild, but it was possible to reproduce the behavior on some of their servers. (I assume they deploy multiple nameserver implementations.) Of course, the whole point of DNS resilience is that every server should work well, so it would still be good if everyone updated their nameservers, and the draft was standardized. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] .COM Zone DNSSEC Operational Update -- ZSK length change
On Thu, Jan 2, 2020 at 9:38 PM Wessels, Duane wrote: > > On Dec 28, 2019, at 8:50 AM, Matt Nordhoff wrote: > > On Mon, Oct 14, 2019 at 6:34 PM Wessels, Duane via dns-operations > > wrote: > >> All, > >> > >> Verisign is in the process of increasing the size and strength of > >> the DNSSEC Zone Signing Keys (ZSKs) for the top-level domains that > >> it operates. As part of this process, the ZSK for the .COM zone will > >> be increased in size from 1024 to 1280 bits. > >> > >> On October 10, 2019 the 1280 bit ZSK was pre-published in the .COM zone. > >> On October 15, we plan to sign the .COM zone with the 1280 bit ZSK. > >> On October 20, we plan to remove the old 1024 bit ZSK from the zone. > > > > D'y'all have an updated ETA on step 3? > > > > Matt, > > My apologies for the incorrect information in the initial message. The old > 1024-bit ZSK was post-published for an extended period of time. It was > removed > as of Jan 1. > > DW [insert GitHub party popper emoji] That's great news. Congratulations on completing the upgrade! :-) Once the RRSIG on the previous DNSKEY record set expires in 9 days, 1024-bit RSA on Verisign-operated TLDs will be absolutely dead and buried. :-) -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] .COM Zone DNSSEC Operational Update -- ZSK length change
On Mon, Oct 14, 2019 at 6:34 PM Wessels, Duane via dns-operations wrote: > All, > > Verisign is in the process of increasing the size and strength of > the DNSSEC Zone Signing Keys (ZSKs) for the top-level domains that > it operates. As part of this process, the ZSK for the .COM zone will > be increased in size from 1024 to 1280 bits. > > On October 10, 2019 the 1280 bit ZSK was pre-published in the .COM zone. > On October 15, we plan to sign the .COM zone with the 1280 bit ZSK. > On October 20, we plan to remove the old 1024 bit ZSK from the zone. D'y'all have an updated ETA on step 3? > We do not anticipate any problems from this upgrade. In accordance > with our normal operating procedures we have a rollback process should > it become necessary to revert to the 1024 bit ZSK. > > DW -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Anyone with contacts at Paypal and/or Ultradns?
On Wed, Dec 11, 2019 at 6:47 AM Tom Ivar Helbekkmo via dns-operations wrote: > Mail from Paypal to me is failing, hard, because I run a resolver with > DNSSEC verification and qname minimization, and an MTA that implements > DMARC. Out of the four name servers they've got configured, the two at > Ultradns are mishandling empty non-terminals. I get SERVFAIL responses > for slc.paypal.com and _domainkey.paypal.com, both of which are needed > for email, because their MTAs are under the former, and their DKIM keys > under the latter. > > The problems are visible using dnsviz: > > https://dnsviz.net/d/slc.paypal.com/dnssec/ > https://dnsviz.net/d/_domainkey.paypal.com/dnssec/ > > I've tried writing to hostmas...@paypal.com about this, but have > received no response. > > -tih It looks like this is a more widespread Neustar issue. A domain using Namecheap's DNS service -- which now outsources the DNS servers to Neustar -- ran into the same issue. <https://community.letsencrypt.org/t/dns-problem-servfail-looking-up-caa-solved/109397> <https://mn0.us/SCfQ> They have since 'solved' the problem by adding a TXT record so that their empty non-terminal isn't empty anymore, but there's a dig showing the issue in the thread. Going by the RRSIG inception and expiration periods, Namecheap is using PowerDNS as a signer, and PayPal is using something else. Namecheap and PayPal could still be using the same software at another point in their stacks, but it seems more likely that the problem is on Neustar's end. -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] sophosxl.net problem?
On Wed, Oct 30, 2019 at 11:30 PM Mark Andrews wrote: > > On 31 Oct 2019, at 12:02 am, Bob Harold wrote: > > On Tue, Oct 29, 2019 at 9:07 PM Paul Vixie wrote: > > Mark Andrews wrote on 2019-10-27 19:24: > > > ... > > > > > > BIND tried to fix named to reject AA=0 from authoritative servers a > > > few years back but pandora.tv was returning AA=0 from all servers at > > > the time and we had to back the change out. We still want to make > > > that change. > > > > please consider making this a config option so that those of us who are > > willing to endure outages for nonconforming domains can turn it on. it > > could even become part of some annual so-called dns flag day. > > > > -- > > P Vixie > > > > I agree. > > > > But if someone thinks that is too drastic, would it be reasonable to make a > > config option, plus an exception list? Then someone could make exceptions > > for the known cases, but break any new cases, to avoid this problem getting > > any worse. > > > > -- > > Bob Harold > > First thing is to get Google, Cloudflare etc. on board. “But it works using > 8.8.8.8 or 1.1.1.1” etc. > is the biggest problem with actually being able to deploy fixes. The second > problem is being able > to contact the server administrators. For y'all's information, PowerDNS Recursor rejects non-AA responses. It used to accept them until, I believe, earlier this year. They're tracking broken zones in an issue: <https://github.com/PowerDNS/pdns/issues/8150> -- Matt Nordhoff ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
