Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
--- Begin Message --- if the same IP is asking the same qname over and over, then you might want to look into DNS RRL, which was originally a BIND thing but which all open source name servers now possess in some form. it was crafted for authority (really, root and TLD) servers, but does also work for recursives. http://www.redbarn.org/dns/ratelimits (i guess i'll go fix the https version of that service, again.) vixie --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
> On 9 Jan 2023, at 15:22, Viktor Dukhovni wrote: > > On Mon, Jan 09, 2023 at 01:55:29PM +, Roy Arends wrote: > >> I’ve often seen this behaviour. >> >> One confirmed explanation was (but there may be more/other) that this >> is the result of a stateful firewall. While the rules are pushed, >> traffic through it is buffereduntil the last rule is pushed, after >> which the buffer is flushed to world, resulting in a barrage of >> queries from the resolver behind the firewall. It depends on the >> resolver what happens with the ID. Some will re-issue the query after >> no response, some re-issue with new ID. > > The repetition of the same DNS query ID and exclusively the same qname > somewhat argues against the firewall theory, because ~100 instances of > just retransmissions of the same query from a resolver seems unlikely, > especially within the time it takes a firewall to reload its ruleset. This was a confirmed case (the bulk same q-id q-name q-type src-addr thing stood out). Repeatable. It may not be the only explanation, though, but it is not theory. It took a few seconds for the specific firewall to reload rules (Checkpoint was the fw in question iirc). The resolver box would receive a dst host/net unreachable from the FW box, which was about 5 ms away, which resulted in the resolver box re-sending the exact same query, and this looped a bit. The FW would buffer the request and upon the “allow 53 UDP” rule loading, a burst of buffered queries were send (partly towards our DNS servers). I have no access to the specific details, as I’ve left Nominet. However, colleagues posted a few of similar stories about spammy DNS related behaviour at the time. ymmv Roy ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
Coming from the spam filtering side of my job... I wonder if they could be part of a filter resolving URLS (based on queries for www. during a burst of identical emails being sent. Do you know what is at two IP addresses? (no PTR records for them) -- William Brown WNYRIC/Erie 1 BOCES -Original Message- From: dns-operations On Behalf Of sth...@nethelp.no Sent: Monday, January 9, 2023 3:50 AM To: dns-operations@lists.dns-oarc.net Subject: [dns-operations] Resolvers seeing repeated bursts of identical queries This email originated from outside of the organization. Use caution when replying, opening attachment(s), and/or clicking on URL's. We are receiving a significant amount of query bursts on our resolvers with the following characteristics: - A client IP doing a burst of queries for the same name repeatedly, very quickly. - The query is typically an A query. - A burst often has 50 - 100 queries for the same name within a few milliseconds. - All the queries within one burst have the same DNS query ID (but different IP id and source port number). - The same client IP producing such bursts of identical queries also sends regular queries (one query per name, DNS query IDs vary). Example of (part of) query burst - in this case the client sends bursts of 84 queries within less than 1 ms: 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) ... 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) followed by another burst of 84 queries in around 1.1 ms: 09:24:56.594416 IP 194.19.79.131.38426 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594475 IP 194.19.79.131.42976 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594501 IP 194.19.79.131.58089 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594560 IP 194.19.79.131.14419 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594561 IP 194.19.79.131.56931 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594562 IP 194.19.79.131.18576 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) ... 09:24:56.595596 IP 194.19.79.131.41232 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) I *suspect* the bursts and the regular queries are actually produced by different clients on the inside of a firewall with NAT - but note I don't *know* this is the case. Does anybody know of software / applications that would produce such query bursts? Note that I don't believe the query bursts are caused by L2 loops or similar, because - These problems have lasted for weeks - And they occur for several different (unrelated) customers Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
On Mon, Jan 09, 2023 at 01:55:29PM +, Roy Arends wrote: > I’ve often seen this behaviour. > > One confirmed explanation was (but there may be more/other) that this > is the result of a stateful firewall. While the rules are pushed, > traffic through it is buffereduntil the last rule is pushed, after > which the buffer is flushed to world, resulting in a barrage of > queries from the resolver behind the firewall. It depends on the > resolver what happens with the ID. Some will re-issue the query after > no response, some re-issue with new ID. The repetition of the same DNS query ID and exclusively the same qname somewhat argues against the firewall theory, because ~100 instances of just retransmissions of the same query from a resolver seems unlikely, especially within the time it takes a firewall to reload its ruleset. Wild speculation: If not some sort of DoS attack, perhaps a stub resolver or application bug or configuration issue? For example, some APIs have timeouts in μs rather than seconds, maybe some code is setting exceedingly short lookup timeouts? Still the retry *count* is rather high. > > - A burst often has 50 - 100 queries for the same name within a few > > milliseconds. > > - All the queries within one burst have the same DNS query ID (but > > different IP id and source port number). > > - The same client IP producing such bursts of identical queries also > > sends regular queries (one query per name, DNS query IDs vary). Multiple systems behind a shared NAT, just one of them is the problem system? -- Viktor. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
I’ve often seen this behaviour. One confirmed explanation was (but there may be more/other) that this is the result of a stateful firewall. While the rules are pushed, traffic through it is buffereduntil the last rule is pushed, after which the buffer is flushed to world, resulting in a barrage of queries from the resolver behind the firewall. It depends on the resolver what happens with the ID. Some will re-issue the query after no response, some re-issue with new ID. I never got confirmation of the firewall make. This was about 8 years ago. Roy > On 9 Jan 2023, at 08:50, sth...@nethelp.no wrote: > > We are receiving a significant amount of query bursts on our resolvers > with the following characteristics: > > - A client IP doing a burst of queries for the same name repeatedly, > very quickly. > - The query is typically an A query. > - A burst often has 50 - 100 queries for the same name within a few > milliseconds. > - All the queries within one burst have the same DNS query ID (but > different IP id and source port number). > - The same client IP producing such bursts of identical queries also > sends regular queries (one query per name, DNS query IDs vary). > > Example of (part of) query burst - in this case the client sends > bursts of 84 queries within less than 1 ms: > > 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > ... > 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > > followed by another burst of 84 queries in around 1.1 ms: > > 09:24:56.594416 IP 194.19.79.131.38426 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > 09:24:56.594475 IP 194.19.79.131.42976 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > 09:24:56.594501 IP 194.19.79.131.58089 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > 09:24:56.594560 IP 194.19.79.131.14419 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > 09:24:56.594561 IP 194.19.79.131.56931 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > 09:24:56.594562 IP 194.19.79.131.18576 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > ... > 09:24:56.595596 IP 194.19.79.131.41232 > 193.75.75.193.53: 28221+ A? > www.facebook.com. (34) > > I *suspect* the bursts and the regular queries are actually produced > by different clients on the inside of a firewall with NAT - but note I > don't *know* this is the case. > > Does anybody know of software / applications that would produce such > query bursts? Note that I don't believe the query bursts are caused by > L2 loops or similar, because > > - These problems have lasted for weeks > - And they occur for several different (unrelated) customers > > Steinar Haug, AS2116 > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
--- Begin Message --- >> Example of (part of) query burst - in this case the client sends >> bursts of 84 queries within less than 1 ms: >> >> 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) >> ... >> 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? >> www.jointraining.com. (38) > > Have you looked at the IP TTL within each of these bursts? > > A random distributionmight suggest a dispersed set of sources (or ALGs or > NATs or something). Patterns might give other clues. Good point. However, all of the queries within one burst have the same IP TTL, which suggests they were generated by the same host. Steinar Haug, AS2116 --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Resolvers seeing repeated bursts of identical queries
Hey, On Mon, Jan 9, 2023 at 03:50, wrote: > Example of (part of) query burst - in this case the client sends > bursts of 84 queries within less than 1 ms: > > 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) > ... > 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? > www.jointraining.com. (38) Have you looked at the IP TTL within each of these bursts? A random distributionmight suggest a dispersed set of sources (or ALGs or NATs or something). Patterns might give other clues. Joe >___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
[dns-operations] Resolvers seeing repeated bursts of identical queries
We are receiving a significant amount of query bursts on our resolvers with the following characteristics: - A client IP doing a burst of queries for the same name repeatedly, very quickly. - The query is typically an A query. - A burst often has 50 - 100 queries for the same name within a few milliseconds. - All the queries within one burst have the same DNS query ID (but different IP id and source port number). - The same client IP producing such bursts of identical queries also sends regular queries (one query per name, DNS query IDs vary). Example of (part of) query burst - in this case the client sends bursts of 84 queries within less than 1 ms: 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) ... 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) followed by another burst of 84 queries in around 1.1 ms: 09:24:56.594416 IP 194.19.79.131.38426 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594475 IP 194.19.79.131.42976 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594501 IP 194.19.79.131.58089 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594560 IP 194.19.79.131.14419 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594561 IP 194.19.79.131.56931 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594562 IP 194.19.79.131.18576 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) ... 09:24:56.595596 IP 194.19.79.131.41232 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) I *suspect* the bursts and the regular queries are actually produced by different clients on the inside of a firewall with NAT - but note I don't *know* this is the case. Does anybody know of software / applications that would produce such query bursts? Note that I don't believe the query bursts are caused by L2 loops or similar, because - These problems have lasted for weeks - And they occur for several different (unrelated) customers Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
