Re: DNS - connection limit (without any extra hardware)
What is this group's name? Oh yeah. So that means you have one of two choices ;-) Smart NANOGers have taken the time to read the NANOG charter here: http://www.nanog.org/charter.html which says... The purpose of NANOG is to provide forums in the North American region for education and the sharing of knowledge for the Internet operations community. --Michael Dillon
Re: DNS - connection limit (without any extra hardware)
On Dec 8, 2006, at 9:56 AM, Petri Helenius wrote: Has anyone figured out a remote but lawful way to repair zombie machines? Having remote power control over all of our customer's equipment. Though the customer might not consider that a repair, I do :-) -- Jo Rhett senior geek Silicon Valley Colocation
Re: DNS - connection limit (without any extra hardware)
On Dec 8, 2006, at 10:36 AM, Scott Weeks wrote: Lawful in which country? What is this group's name? Oh yeah. So that means you have one of two choices ;-) -- Jo Rhett senior geek Silicon Valley Colocation
Re: DNS - connection limit (without any extra hardware)
Lawful in which country? What is this group's name? Oh yeah. So that means you have one of two choices ;-) i know this will come as a shock to many, but there are more than two countries in north america. and like afnog, nanog is not as isolationist or jingoist as the current us administration. randy
Re: DNS - connection limit (without any extra hardware)
--- [EMAIL PROTECTED] wrote: On Dec 8, 2006, at 10:36 AM, Scott Weeks wrote: Lawful in which country? What is this group's name? Oh yeah. So that means you have one of two choices ;-) I was speaking about 'the internet' and not just the NA part of it. It's my goal to keep pointing out that 'the internet' isn't only the US and, therefore, a particular government's rules (whatever government that may be) don't apply. Unless, of course, every country gets their own 'Great Firewall of China'. Also, NANOG is the closest thing I can find to a GNOG (Global). scott
Re: DNS - connection limit (without any extra hardware)
of course, my company is working on two main tasks: the first team is focused on discovering what is the virus, and what is the best anti-virus. instead, my team has already scaled our DNS service, by doubling the number of DNSs. I'm not completely satisfied by the scaling solution: I wish to find a solution that could grant a good quality of the service without placing a lot of DNS in my web-farms Thanks Best Regards Luke On 12/8/06, Matt Ghali [EMAIL PROTECTED] wrote: On Fri, 8 Dec 2006, Simon Waters wrote: I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9. Indeed. It is generally accepted that it is easier to simply scale your service to provide adequate headroom than implement per-client traffic policies. of course, you could also work on cleaning up the mess, but I will charitably assume you are working the problem from both directions simultaneously. matto [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Re: DNS - connection limit (without any extra hardware)
I use to slave . which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross). I'm not sure to understand your solution. You configure your name-server as a slave-root-server? On 12/8/06, Simon Waters [EMAIL PROTECTED] wrote: On Friday 08 December 2006 14:40, you wrote: For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. That may trap an email server or two. Did you consider checking what they are looking up, and lying to them about the TTL/answer 127.0.0.1 for a week maybe better than NXDOMAIN. I use to slave . which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross). I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
Re: DNS - connection limit (without any extra hardware)
On Monday 11 December 2006 16:15, you wrote: I use to slave . which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross). I'm not sure to understand your solution. You configure your name-server as a slave-root-server? Yes. Most of the root server traffic is answering queries with NXDOMAIN for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain. The drawback is you provide the answer with the authority bit set, which isn't what the world's DNS clients should expect, but DNS clients don't care about that one bit (sorry). If the root zone file changed quickly it might also cause other problems! Paul V was very cautious about it as a method of running a DNS server, but if the recursive servers are being barraged with queries for (different) non-existent top level domains I think it is probably preferable to the servers being flattened (and/or passing that load onto the root name servers). If the queries are for existing, or the same, domains each time, it won't provide significant improvement. I suppose any server issuing more than 2000 or so queries a day to the root servers would potentially save bandwidth, and provide a more responsive experience for the end user. But one also has to handle the case of the root zone potentially expiring, not something I ever allowed to happen, but then I'm not the average DNS administrator. I've used this technique extensively myself in the past with no issues, but I'm not using it operationally at the moment. Since the load average on our DNS server is 0.00 to two decimal places I doubt it would make a lot of difference, and we host websites, and email, not randomly misconfigured, home, or business user PCs. So mostly we do lookups in in-addr.arpa, a depressingly large proportion of which fail, or look-ups for a small set of servers we forward email to (most of which exist, or I delete the forward).
Re: DNS - connection limit (without any extra hardware)
On Mon, 11 Dec 2006, Simon Waters wrote: Yes. Most of the root server traffic is answering queries with NXDOMAIN for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain. That would require configuring my caching server with authoritative zones, and it seems prevailing wisdom (at least with BIND configurations?) is to keep the peanut butter seperate from the chocolate, no matter how great they taste together, to the best of my knowledge. matto [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Re: DNS - connection limit (without any extra hardware)
In article [EMAIL PROTECTED] you write: On Mon, 11 Dec 2006, Simon Waters wrote: Yes. Most of the root server traffic is answering queries with NXDOMAIN for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain. That would require configuring my caching server with authoritative zones, and it seems prevailing wisdom (at least with BIND configurations?) is to keep the peanut butter seperate from the chocolate, no matter how great they taste together, to the best of my knowledge. matto No. The wisdom is to not make your authoritative servers caches. This is not the same as not making your caches authoritative for certain zones. Just don't have the caches listed in the NS RRsets. Note: You will need to configure your master server(s) to notify the caches for the zone that slave as the automatic mechanisms won't discover them. Mark [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Re: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Petri Helenius wrote: Has anyone figured out a remote but lawful way to repair zombie machines? Pete Virtual patching. -Hank
DNS - connection limit (without any extra hardware)
Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
RE: DNS - connection limit (without any extra hardware)
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? George Roettger -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Luke Sent: Friday, December 08, 2006 9:41 AM To: [EMAIL PROTECTED] Subject: DNS - connection limit (without any extra hardware) Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
Re: DNS - connection limit (without any extra hardware)
On Friday 08 December 2006 14:40, you wrote: For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. That may trap an email server or two. Did you consider checking what they are looking up, and lying to them about the TTL/answer 127.0.0.1 for a week maybe better than NXDOMAIN. I use to slave . which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross). I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
RE: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Geo. wrote: I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? George Roettger Atually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us. Luke: It is possible the DNS queries made are for non existent domains, fake replies, perhaps even making them something in 1918 space, and they MAY stop being not nice netizens. Gadi. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Luke Sent: Friday, December 08, 2006 9:41 AM To: [EMAIL PROTECTED] Subject: DNS - connection limit (without any extra hardware) Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
Re: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Luke wrote: Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? I have a bots infested network, they really task my services! How can I make my services ignore them so that the clients start calling me and spending my tech support budget? Thanks Best Regards Luke Gadi.
RE: DNS - connection limit (without any extra hardware)
Actually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us. Infected machines (bots) will cause a lot more than just DNS issues. Issues like this have a way of getting worse all by themselves if not addressed. Anyway, to play nice.. how about using a router to dampen traffic much like icmp dampening? Would it be possible to do DNS dampening? Geo.
Re: DNS - connection limit (without any extra hardware)
Geo. wrote: I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? Has anyone figured out a remote but lawful way to repair zombie machines? Pete George Roettger -Original Message- *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of *Luke *Sent:* Friday, December 08, 2006 9:41 AM *To:* [EMAIL PROTECTED] *Subject:* DNS - connection limit (without any extra hardware) Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
Re: DNS - connection limit (without any extra hardware)
On 8-Dec-2006, at 11:52, Geo. wrote: Actually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us. Infected machines (bots) will cause a lot more than just DNS issues. Issues like this have a way of getting worse all by themselves if not addressed. Anyway, to play nice.. how about using a router to dampen traffic much like icmp dampening? Would it be possible to do DNS dampening? I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router. There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle). You may find that you can install ipfw (or similar) rules on your nameservers themselves to do this kind of thing. Take careful note of what happens when the client population becomes large, though -- the garbage collection ought to be smooth and painless, or you'll just wind up swapping one worm proliferation failure mode for another. Host-based per-client rate limits scale better if there are many hosts providing service, e.g. behind a load balancer or using something like http://www.isc.org/pubs/tn/isc-tn-2004-1.html. As to the wider question, cleaning up the infected hosts is an excellent goal, but it'd certainly be nice if your DNS servers continued to function while you were doing so. Having every non- infected customer phone up screaming at once can be an unwelcome distraction when you already have more man hours of work to do per day than you have (staff * 24). Joe
Re: DNS - connection limit (without any extra hardware)
I have a bots infested network, they really task my services! How can I make my services ignore them so that the clients start calling me and spending my tech support budget? Or: I have bots on my network and as part of a multi-pronged approach to cleaning my network while keeping the services available to those who aren't infected, I'd like to research ways that I can minimize the effect these bots have on the rest of my customers Cheers, .pm
Re: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Petri Helenius wrote: Geo. wrote: I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? Has anyone figured out a remote but lawful way to repair zombie machines? Microsoft auto-update, the telephone line, going to a different country with a different set of rules. Gadi.
Re: DNS - connection limit (without any extra hardware)
On 12/8/06, Petri Helenius [EMAIL PROTECTED] wrote: Has anyone figured out a remote but lawful way to repair zombie machines? sure, null route the customer until they clean their hosts up
Re: DNS - connection limit (without any extra hardware)
--- [EMAIL PROTECTED] wrote: From: Petri Helenius [EMAIL PROTECTED] Geo. wrote: I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? :: Has anyone figured out a remote but lawful way to repair :: zombie machines? Lawful in which country? scott
Re: DNS - connection limit (without any extra hardware)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sorry for the top-post, but wanted to retain context here. Also, sorry for the specific product mention, but much of is mentioned below is something that we are doing with ICSS/BASE: http://www.trendmicro.com/en/products/nss/icss/evaluate/overview.htm $.02, - - ferg - -- Joe Abley [EMAIL PROTECTED] wrote: On 8-Dec-2006, at 11:52, Geo. wrote: Actually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us. Infected machines (bots) will cause a lot more than just DNS issues. Issues like this have a way of getting worse all by themselves if not addressed. Anyway, to play nice.. how about using a router to dampen traffic much like icmp dampening? Would it be possible to do DNS dampening? I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router. There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle). You may find that you can install ipfw (or similar) rules on your nameservers themselves to do this kind of thing. Take careful note of what happens when the client population becomes large, though -- the garbage collection ought to be smooth and painless, or you'll just wind up swapping one worm proliferation failure mode for another. Host-based per-client rate limits scale better if there are many hosts providing service, e.g. behind a load balancer or using something like http://www.isc.org/pubs/tn/isc-tn-2004-1.html. As to the wider question, cleaning up the infected hosts is an excellent goal, but it'd certainly be nice if your DNS servers continued to function while you were doing so. Having every non- infected customer phone up screaming at once can be an unwelcome distraction when you already have more man hours of work to do per day than you have (staff * 24). Joe -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.5.1 (Build 1557) wj8DBQFFebFQq1pz9mNUZTMRAk+xAKCg1dPMivTo6ee5Nj1I4yjVXQzvCQCgnBSI NV3RnsEijPJcHNawWS4uWog= =pawb -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: DNS - connection limit (without any extra hardware)
Aaron Glenn wrote: On 12/8/06, Petri Helenius [EMAIL PROTECTED] wrote: Has anyone figured out a remote but lawful way to repair zombie machines? sure, null route the customer until they clean their hosts up My question was specifically directed towards zombies that are not local to the ISP. Pete
Re: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Simon Waters wrote: I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9. Indeed. It is generally accepted that it is easier to simply scale your service to provide adequate headroom than implement per-client traffic policies. of course, you could also work on cleaning up the mess, but I will charitably assume you are working the problem from both directions simultaneously. matto [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
RE: DNS - connection limit (without any extra hardware)
On Fri, 8 Dec 2006, Gadi Evron wrote: Luke: It is possible the DNS queries made are for non existent domains, fake replies, perhaps even making them something in 1918 space, and they MAY stop being not nice netizens. Configuring your nameservers to randomly give bad answers isn't considered being a nice netizen either, the last time I checked. [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
RE: DNS - connection limit (without any extra hardware)
You could also look at Cloudshield. I was following the EveryDNS issue this weekend and this item among the regular VON press release blast jumped out at me: http://www.cloudshield.com/news_events/2006_Releases/EveryDNS%20FINAL.pdf Regards, Frank _ From: Frank Bulk Sent: Friday, December 08, 2006 8:59 AM To: '[EMAIL PROTECTED]' Subject: DNS - connection limit (without any extra hardware) Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
Re: DNS - connection limit (without any extra hardware)
On Dec 8, 2006, at 6:40 AM, Luke wrote: Hi, as a consequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? One effective strategy is to make 0wning your customer's system less profitable. Here is a good article by Suresh Ramasubramanian: http://www.circleid.com/posts/ port_25_blocking_or_fix_smtp_and_leave_port_25_alone_for_the_sake_of_spa m/ Some have been successful with notification tools such as those offered by: http://www.perftech.com/ Customers are directed to a free scrub that does not depend upon OS validation status, such as Housecall. -Doug
