limiting number of requests of a single hosts
We have a problem with one of our firewalls caused by DNS peaks. Once or twice a day a DNS burst (20K requests/15sec) kills all connections on the firewall. The firewall is due for replacement but in the mean time we would like to stop these peaks at their origin or at least try to limit their impact. We have 6 dns servers (bind) on our campus, that are all authoritative for our domains and also resolver for our campus hosts. Most of our clients however use our AD/LDAP/DNS Microsoft servers as their resolver, which on their turn contact our 6 dns servers for further resolving. What we figured out by packet capturing, is that at a certain point in time these AD/LDAP/DNS servers start 'collecting' dns requests without sending them further and then in a burt pass them on to our 6 dns servers which try to resolve these queries. Due to the fact that one request of a client mostly results in several queries of our dns servers to the outside world (root server contact, NS record resolving,..) , this results in a burst of dns requests through our firewalls, killing them. I have 2 questions, one, is there a way to rate-limit the amount of request a single client (the AD servers in this case) can have standing out against a bind server ? Kind of rate-limiting parameter for bind name server. Two, has anyone already seen this type of behavior on a Microsoft AD/LDAP/DNS server and has a clue what could cause this stalling ? Solving that would be the best solution. Thanks in advance for any suggestion, answer, Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
Holemans Wim wim.holem...@ua.ac.be wrote: I have 2 questions, one, is there a way to rate-limit the amount of request a single client (the AD servers in this case) can have standing out against a bind server ? Kind of rate-limiting parameter for bind name server. There isn't a way to do this in BIND. If you are running on Linux you might try the iptables hashlimit module, http://www.bortzmeyer.org/rate-limiting-dns-open-resolver.html (The recently announced response rate limiting patch won't work for you since it takes effect too late in the resolution process. http://www.redbarn.org/dns/ratelimits) I'm afraid I don't have an answer to your other question. Tony. -- f.anthony.n.finch d...@dotat.at http://dotat.at/ Plymouth, Northwest Biscay: Southwesterly 5 to 7, occasionally gale 8 in Plymouth. Rough or very rough, occasionally high in west Plymouth. Showers. Good, occasionally poor.___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
bind-users-bounces+wbrown=e1b@lists.isc.org wrote on 06/15/2012 04:25:16 AM: We have a problem with one of our firewalls caused by DNS peaks. Once or twice a day a DNS burst (20K requests/15sec) kills all connections on the firewall. The firewall is due for replacement but in the mean time we would like to stop these peaks at their origin or at least try to limit their impact. We have 6 dns servers (bind) on our campus, that are all authoritative for our domains and also resolver for our campus hosts. Most of our clients however use our AD/LDAP/DNS Microsoft servers as their resolver, which on their turn contact our 6 dns servers for further resolving. What we figured out by packet capturing, is that at a certain point in time these AD/LDAP/DNS servers start ?collecting? dns requests without sending them further and then in a burt pass them on to our 6 dns servers which try to resolve these queries. Due to the fact that one request of a client mostly results in several queries of our dns servers to the outside world (root server contact, NS record resolving,..) , this results in a burst of dns requests through our firewalls, killing them. I have 2 questions, one, is there a way to rate-limit the amount of request a single client (the AD servers in this case) can have standing out against a bind server ? Kind of rate-limiting parameter for bind name server. Two, has anyone already seen this type of behavior on a Microsoft AD/LDAP/DNS server and has a clue what could cause this stalling ? Solving that would be the best solution. Any chance of using network devices (firewalls, intelligent switches) to rate limit connections from the AD/DNS server to the bind server? Is the odd behavior of the AD/DNS server causing issues with the clients making the original request? Have you tried tracking down the original source of the query? Could that be the ultimate source of the traffic burst? It seems unlikely that MSDNS would intentionally hold DNS requests. Have you tried troubleshooting that? Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: limiting number of requests of a single hosts
Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp One of the problems is that these firewalls are going to be replaced soon and we don't want to spend to much effort in trying to fix what seems an annoying side-effect of something caused by a DNS system. We actually captured dns traffic around our AD server and were we see an average of 500 dns packets/5s in/out in normal conditions, this drops to about 100 for 20 seconds and then rises to 2000 dns packets/5sec causing our resolving servers to send a multiple amount of requests to the outside world killing the firewall. We know changed the settings on the AD server to only use 2 of the resolving servers (which have a max recursive clients implemented) and checked the box, saying that the AD server could do his own lookups if the forwarders are not available. Any chance of using network devices (firewalls, intelligent switches) to rate limit connections from the AD/DNS server to the bind server? Is the odd behavior of the AD/DNS server causing issues with the clients making the original request? Have you tried tracking down the original source of the query? Could that be the ultimate source of the traffic burst? It seems unlikely that MSDNS would intentionally hold DNS requests. Have you tried troubleshooting that? Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
On Fri, Jun 15, 2012 at 9:37 PM, Holemans Wim wim.holem...@ua.ac.be wrote: Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp One of the problems is that these firewalls are going to be replaced soon and we don't want to spend to much effort in trying to fix what seems an annoying side-effect of something caused by a DNS system. You DO realize that DNS is (mostly) UDP packets, and an attacker (or in your case, the ADs) can simply send UDP packet floods to kill your firewall (in your current state), regardless how your DNS server is configured, even when the DNS server is down? -- Fajar ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: limiting number of requests of a single hosts
Yes we know and new firewalls are on their way (already partly installed), but we can't activate them for the moment as we are in an examination period in which we can't make any change to our network as students should be able to take online tests 24/24... Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp -Original Message- From: Fajar A. Nugraha [mailto:w...@fajar.net] Sent: vrijdag 15 juni 2012 17:02 To: Holemans Wim Cc: bind-users@lists.isc.org Subject: Re: limiting number of requests of a single hosts On Fri, Jun 15, 2012 at 9:37 PM, Holemans Wim wim.holem...@ua.ac.be wrote: Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp One of the problems is that these firewalls are going to be replaced soon and we don't want to spend to much effort in trying to fix what seems an annoying side-effect of something caused by a DNS system. You DO realize that DNS is (mostly) UDP packets, and an attacker (or in your case, the ADs) can simply send UDP packet floods to kill your firewall (in your current state), regardless how your DNS server is configured, even when the DNS server is down? -- Fajar ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
You DO realize that DNS is (mostly) UDP packets, and an attacker (or in your case, the ADs) can simply send UDP packet floods to kill your firewall (in your current state), regardless how your DNS server is configured, even when the DNS server is down? Once we had the firewall for DNS, when it get bunk of queries from the suspect addresses, it returns truncating message and indicates the client to use TCP for queries. -- Email/Jabber/Gtalk: pa...@riseup.net Free DNS Hosting with www.DNSbed.com ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
On Jun 15, 2012, at 4:25 AM, Holemans Wim wrote: We have a problem with one of our firewalls caused by DNS peaks. Yes. EOM W Once or twice a day a DNS burst (20K requests/15sec) kills all connections on the firewall. The firewall is due for replacement but in the mean time we would like to stop these peaks at their origin or at least try to limit their impact. We have 6 dns servers (bind) on our campus, that are all authoritative for our domains and also resolver for our campus hosts. Most of our clients however use our AD/LDAP/DNS Microsoft servers as their resolver, which on their turn contact our 6 dns servers for further resolving. What we figured out by packet capturing, is that at a certain point in time these AD/LDAP/DNS servers start ‘collecting’ dns requests without sending them further and then in a burt pass them on to our 6 dns servers which try to resolve these queries. Due to the fact that one request of a client mostly results in several queries of our dns servers to the outside world (root server contact, NS record resolving,..) , this results in a burst of dns requests through our firewalls, killing them. I have 2 questions, one, is there a way to rate-limit the amount of request a single client (the AD servers in this case) can have standing out against a bind server ? Kind of rate-limiting parameter for bind name server. Two, has anyone already seen this type of behavior on a Microsoft AD/LDAP/DNS server and has a clue what could cause this stalling ? Solving that would be the best solution. Thanks in advance for any suggestion, answer, Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users -- Curse the dark, or light a match. You decide, it's your dark. -- Valdis Kletnieks ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
Hi there, On Fri, 15 Jun 2012, Holemans Wim wrote: ... Once or twice a day a DNS burst (20K requests/15sec) kills all connections on the firewall. Have you disabled firewall connection tracking for DNS requests? We have 6 dns servers (bind) on our campus, that are all authoritative for our domains and also resolver for our campus hosts. Most of our clients however use our AD/LDAP/DNS Microsoft servers as their resolver, which on their turn contact our 6 dns servers for further resolving. Could you simply run BIND resolvers for your clients and as far as possible avoid using the Microsoft services? Two, has anyone already seen this type of behavior on a Microsoft AD/LDAP/DNS server and has a clue what could cause this stalling ? Yes, I've seen it. I suspect dropped packets might be the cause, but I have no hard evidence. My solution was to use BIND instead. :) -- 73, Ged. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: limiting number of requests of a single hosts
On 15/06/12 16:37, Holemans Wim wrote: Wim Holemans Netwerkdienst Universiteit Antwerpen Network Services University of Antwerp One of the problems is that these firewalls are going to be replaced soon and we don't want to spend to much effort in trying to fix what seems an annoying side-effect of something caused by a DNS system. We actually captured dns traffic around our AD server and were we see an average of 500 dns packets/5s in/out in normal conditions, this drops to about 100 for 20 seconds and then rises to 2000 dns packets/5sec causing our resolving servers to send a multiple amount of requests to the outside world killing the firewall. One thing that comes to mind is: have you traced outside the firewall with e.g. wireshark and looked at what precedes the burst? I am thinking maybe the firewall makes a stop in the packet flow that will then trigger the flood? Possibly caused by some table in the firewall being overflowed, maybe even with unrelated traffic. In this case, only one solution is possible. We know changed the settings on the AD server to only use 2 of the resolving servers (which have a max recursive clients implemented) and checked the box, saying that the AD server could do his own lookups if the forwarders are not available. -- Best regards Sten Carlsen No improvements come from shouting: MALE BOVINE MANURE!!! ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
