Re: named tcp dos?
On Thursday, August 02, 2018 18:13:21 Randy Bush wrote:
> > We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP
> > queries.
>
> that is quite a variance
>
> > In comparison, we get about 25-30% IPv6 queries.
>
> wonder how that compares to others
>
On the secondaries for a Fortune 50 company with a sizeable ecommerce presence,
we see ~17% of queries come in over IPv6, and ~2.5% are TCP queries. With
respect to the Internet, the v6 percentage is probably low, as the servers I
checked answer quite a lot of queries from internal IPv4 networks.
For grins, I turned on query logging on one server (BIND 9.11.4) for a short
time and produced a histogram of the unique query attribute combinations:
$ awk '"query:"==$10 {print $(NF-1)}' /var/log/daemon.2 | sort | uniq -c | sort
-rn | tee >(awk '{s+=$1}END{print s}')
38111265 -E(0)DC
4963452 -E(0)D
4784394 -
3268810 -E(0)
896136 +E(0)DC
551934 -E(0)TDC
406856 -E(0)DCV
318068 -E(0)DV
282536 -E(0)DCK
173078 -T
149780 -E(0)TD
132303 -E(0)DK
107240 -C
105752 -E(0)T
32748 -E(0)TDV
24677 +
21722 -E(0)TDCV
10958 -E(0)C
10907 +T
337 -E(0)TDCK
174 +E(0)
135 -TC
131 -E(0)TDK
98 +E(0)TDC
19 +E(0)D
18 +E(0)K
8 -E(0)TC
3 +E(0)T
54353539
FWIW, this indicates that most TCP queries come from clients that claim to
support EDNS0.
--
Greg Rivers
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Randy Bush wrote: > > an aside: folk seem to be in the 20% range for ipv6, while overall > backbone traffic stats are about half that. are dns caches more likely > to be v6 enabled than the average bear? I get the impression from various discussions that yes, they are. Actual citation: http://www.potaroo.net/ispcol/2016-10/dnsipv6.html Tony. -- f.anthony.n.finchhttp://dotat.at/ Bailey: Southwest, becoming cyclonic 5 to 7, decreasing 4 at times. Moderate or rough. Showers. Good, occasionally poor. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
> We have slightly less then 25% for IPv6 queries. > And about 4-5% TCP queries. considering we share the load of the same non-trivial signed cctld, i should be seeing similarly. though i am sure both of us serve a few more . and tony and hugo (the latter privately) are seeing similar, though maybe slightly less v6. or they admit to more variance :) > In our case, the default for "tcp-clients" setting is still good enough. > > In BIND 9.9/9.10 it is 100 by default > In BIND 9.11/9.12 it is 150 by default. i am currently running default on 9.10 > If you want the future, you can set it to 200 ;-) tony's reply/advice on this is interesting. i am considering his > minimal-responses yes; > minimal-any yes; but maybe i should just suck it up; tcp and tls are the wave of the dns future. an aside: folk seem to be in the 20% range for ipv6, while overall backbone traffic stats are about half that. are dns caches more likely to be v6 enabled than the average bear? yet another measurement project for which we have no time. hi duane :) and thanks for the real numbers. much better than, though not as amusing as, the email i received from two frat boys who probably should not drink and type. reminiscences of the usenet! randy ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Matus UHLAR - fantomas wrote: > On 03.08.18 15:09, Tony Finch wrote: > > minimal-any definitely reduces truncated responses - that's why I > > implemented it :-) > > - are they so common that it does matter? Well, they used to be, but Chris Thompson and I have done a lot to make ANY queries on our servers unattractive to attackers. Tony. -- f.anthony.n.finchhttp://dotat.at/ oppose all forms of entrenched privilege and inequality ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
On 03.08.18 12:10, Tony Finch wrote: > I have a few config options which can affect TCP usage. These two should > reduce it: > >minimal-responses yes; >minimal-any yes; Matus UHLAR - fantomas wrote: I don't think so. minimal-responses only skip unnecessary info, so they should have no effect on TCP retries. On 03.08.18 15:09, Tony Finch wrote: minimal-any definitely reduces truncated responses - that's why I implemented it :-) - are they so common that it does matter? - if anyone wants to get full reply now, they do need to use TCP now, even if the response would fit to 1420 chars... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. A day without sunshine is like, night. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Matus UHLAR - fantomas wrote: > > it's the max-udp-size 1420 apparently. > > I set it to similar value because of problematic L3 switch in front of our > DNS servers long ago. > > Should not be needed now. I don't have that because of my network (which works OK), but because of other people's broken networks that screw up fragmented responses and so have trouble resolving our domain names. Tony. -- f.anthony.n.finchhttp://dotat.at/ Cromarty, Forth: Variable, mainly west, 3 or 4, occasionally 5 for a time. Smooth or slight. Thundery showers, fog patches. Moderate or good, occasionally very poor. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Matus UHLAR - fantomas wrote: > On 03.08.18 12:10, Tony Finch wrote: > > I have a few config options which can affect TCP usage. These two should > > reduce it: > > > > minimal-responses yes; > > minimal-any yes; > > I don't think so. minimal-responses only skip unnecessary info, so they > should have no effect on TCP retries. minimal-any definitely reduces truncated responses - that's why I implemented it :-) Tony. -- f.anthony.n.finchhttp://dotat.at/ Cromarty, Forth: Variable, mainly west, 3 or 4, occasionally 5 for a time. Smooth or slight. Thundery showers, fog patches. Moderate or good, occasionally very poor. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Daniel Stirnimann wrote: >> In comparison, we get about 25-30% IPv6 queries. We have slightly less then 25% for IPv6 queries. On 03.08.18 12:19, Tony Finch wrote: Hmm, I have 20% on one server and 22% on another. it's the max-udp-size 1420 apparently. I set it to similar value because of problematic L3 switch in front of our DNS servers long ago. Should not be needed now. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Save the whales. Collect the whole set. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Randy Bush wrote: estimate or measure the distribution of the ratio of udp to tcp queries on say 100 cctld servers. On 03.08.18 12:10, Tony Finch wrote: On a recently rebooted auth server, which hosts zones for a handful of universities with and without DNSSEC, slightly less than 1% of queries are over TCP. $ curl -Ssf http://authdns1.csx.cam.ac.uk:8053/json/v1 | jq '[ .nsstats.QryUDP, .nsstats.QryTCP ]' [ 6994195, 61575 ] I have a few config options which can affect TCP usage. These two should reduce it: minimal-responses yes; minimal-any yes; I don't think so. minimal-responses only skip unnecessary info, so they should have no effect on TCP retries. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Windows 2000: 640 MB ought to be enough for anybody ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Daniel Stirnimann wrote: > >> In comparison, we get about 25-30% IPv6 queries. > > We have slightly less then 25% for IPv6 queries. Hmm, I have 20% on one server and 22% on another. Tony. -- f.anthony.n.finchhttp://dotat.at/ a fair voting system for all elections ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Randy Bush wrote:
>
> estimate or measure the distribution of the ratio of udp to tcp queries
> on say 100 cctld servers.
On a recently rebooted auth server, which hosts zones for a handful of
universities with and without DNSSEC, slightly less than 1% of queries are
over TCP.
$ curl -Ssf http://authdns1.csx.cam.ac.uk:8053/json/v1 |
jq '[ .nsstats.QryUDP, .nsstats.QryTCP ]'
[
6994195,
61575
]
I have a few config options which can affect TCP usage. These two should
reduce it:
minimal-responses yes;
minimal-any yes;
These ones can increase it:
rate-limit {
responses-per-second 10;
ipv4-prefix-length 32;
exempt-clients { cudn; };
};
max-udp-size 1420;
(The latter is to avoid UDP fragmentation.)
This is not a very beefy server so I haven't increased the TCP concurrency
very much:
tcp-clients 256;
Tony.
--
f.anthony.n.finchhttp://dotat.at/
Fisher, German Bight: South, veering west 3 or 4, occasionally 5 later. Smooth
or slight. Showers. Moderate or good.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
On 03.08.18 03:13, Randy Bush wrote: >> We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP >> queries. > > that is quite a variance > >> In comparison, we get about 25-30% IPv6 queries. > > wonder how that compares to others We have slightly less then 25% for IPv6 queries. And about 4-5% TCP queries. In our case, the default for "tcp-clients" setting is still good enough. In BIND 9.9/9.10 it is 100 by default In BIND 9.11/9.12 it is 150 by default. If you want the future, you can set it to 200 ;-) Daniel ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
> We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP > queries. that is quite a variance > In comparison, we get about 25-30% IPv6 queries. wonder how that compares to others thanks for actual data randy ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: named tcp dos?
> -Original Message- > From: bind-users [mailto:bind-users-boun...@lists.isc.org] On Behalf Of > Randy Bush > Sent: Friday, 3 August 2018 6:08 AM > > >> ... are there that many folk doing tcp out there? > > All name servers fall back to TCP when they receive truncated replies. > > we know the protocol. [ and we know folk have idiot middleboxen ] > > what i was asking was the distribution of this in the wild. We run about 300 TLD's on our DNS platform and get roughly 5-10% TCP queries. In comparison, we get about 25-30% IPv6 queries. Those rates are fairly consistent. > > randy Stuart ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
> ... are there that many folk doing tcp out there? All name servers fall back to TCP when they receive truncated replies. >>> >>> we know the protocol. [ and we know folk have idiot middleboxen ] >>> >>> what i was asking was the distribution of this in the wild >> >> one word: DNSSEC > Indeed, DNSSEC is a prime example. My point was that TCP queries to > your servers are determined largely by the size of the RRSETs you > serve. If your answers don't fit in 512 bytes (without EDNS) or ~4096 > bytes (with EDNS), you're going to be serving over TCP. as i said, let's assume we know the protocol. > Obviously you're way more likely to see TCP queries from systems that > don't support EDNS. Perhaps you have many such systems (and or idiot > middleboxen) querying you? two $dayjobs are interfering with my trying to schedule the time to actually measure what i am seeing on my servers. :) there are a fair number of zones here, including a large cctld with a lot of signage. so my guess (i.e. no real measurements [0]) is that at least that server sees a higher tcp ratio than the average bear., but if i get those data, are they 'normal?' are they similar to what others see? randy [0] - i confess to being a measurement researcher in one of my real lives. so i take measurement a bit seriously. but i have not been measuring dns for a couple of decades. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
On Thursday, August 02, 2018 22:12:38 Reindl Harald wrote: > > Am 02.08.2018 um 22:07 schrieb Randy Bush: > >>> ... are there that many folk doing tcp out there? > >> All name servers fall back to TCP when they receive truncated replies. > > > > we know the protocol. [ and we know folk have idiot middleboxen ] > > > > what i was asking was the distribution of this in the wild > > one word: DNSSEC > Indeed, DNSSEC is a prime example. My point was that TCP queries to your servers are determined largely by the size of the RRSETs you serve. If your answers don't fit in 512 bytes (without EDNS) or ~4096 bytes (with EDNS), you're going to be serving over TCP. Obviously you're way more likely to see TCP queries from systems that don't support EDNS. Perhaps you have many such systems (and or idiot middleboxen) querying you? -- Greg ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
>> estimate or measure the distribution of the ratio of udp to tcp >> queries on say 100 cctld servers > > bla - 512 bytes are easily exceeded > > more than 10 years ago i also thought i am smart and TCP 53 is only > needed for zone-transfers until i realized that random e-mail errors > where the result of large TXT records including way too verbose SPF > > open TCP 53 for the world and all problems where gone forever in my case, tcp was open and that is where the problem arose :) i am trying to tune per daniel's > Depending on your "max-udp-size" value (default 4096) you may also > want to increase "tcp-clients" setting (default 150). and am trying to understand the space randy ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
On 08/02/2018 04:16 PM, Randy Bush wrote: it is in a contest with ipv6 for non-deployment I read this mail list ALL the time and finally something appears that quite literally made me call over a few guys to point at my screen. Well done. Let's make up a tee-shirt with that on it : DNSSEC? IPv6? Which will deploy last? Something similar .. maybe a cartoon is needed. Dennis ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
... are there that many folk doing tcp out there? >>> All name servers fall back to TCP when they receive truncated replies. >> >> we know the protocol. [ and we know folk have idiot middleboxen ] >> >> what i was asking was the distribution of this in the wild > > one word: DNSSEC i.e. it is in a contest with ipv6 for non-deployment :( let me try and phrase my question to narrow the result to a number as opposed to a religion. estimate or measure the distribution of the ratio of udp to tcp queries on say 100 cctld servers. randy ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
>> ... are there that many folk doing tcp out there? > All name servers fall back to TCP when they receive truncated replies. we know the protocol. [ and we know folk have idiot middleboxen ] what i was asking was the distribution of this in the wild. randy ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
On Thursday, August 02, 2018 12:58:32 Randy Bush wrote: > ... are there that many folk doing tcp out there? > All name servers fall back to TCP when they receive truncated replies. -- Greg Rivers ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
> mdig @147.28.0.39 -f queries.txt
>
> queries.txt contains 40x
> switch.ch A
>
> I would suggest something like this:
>
> rate-limit {
>// start rate-limiting if more then X identical
>// responses per second, default 0 i.e. unlimited
>responses-per-second 25;
>nxdomains-per-second 25;
>errors-per-second 25;
>// credit/penalty WINDOW, default 15
>window 10;
>// send TC for every X-th rate-limited response, default 2
>slip 1;
> };
ok. done. thank you.
> Depending on your "max-udp-size" value (default 4096) you may also want
> to increase "tcp-clients" setting (default 150).
both are default. suggestions? are there that many folk doing tcp out
there?
randy
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: named tcp dos?
Hello Randy,
> so, i guess there is a named tcp dos going around. using bind9, is
> there an amelioration? or am i misconfigured in some way?
It looks to me that this is a side effect of a very permissive RRL
configuration. My tests with the following command indicate that you
have set responses-per-second to 5.
mdig @147.28.0.39 -f queries.txt
queries.txt contains 40x
switch.ch A
I would suggest something like this:
rate-limit {
// start rate-limiting if more then X identical
// responses per second, default 0 i.e. unlimited
responses-per-second 25;
nxdomains-per-second 25;
errors-per-second 25;
// credit/penalty WINDOW, default 15
window 10;
// send TC for every X-th rate-limited response, default 2
slip 1;
};
Depending on your "max-udp-size" value (default 4096) you may also want
to increase "tcp-clients" setting (default 150).
Daniel
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
