Re: Ways to handle DNS amplification attacks with OpenBSD
Am Sonntag, den 10.06.2012, 00:37 + schrieb Stuart Henderson: On 2012-06-09, Kostas Zorbadelos kzo...@otenet.gr wrote: I am interested to hear possible solutions in other layers as well. http://fanf.livejournal.com/122111.html seems a nice approach... This seems to work nicely if the attacker spoofs random addresses or if the real target is not the DNS server but the endpoint receiving its replies (therefore the term amplification attack). In Kostas's case it appears the attacker spoofs legit client addresses, which means rate limiting would likely cut off these clients. Rudi
Re: Ways to handle DNS amplification attacks with OpenBSD
On 09/06/12 18:58, Kostas Zorbadelos wrote: Kostas Zorbadeloskzo...@otenet.gr writes: there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. Ohh, they do and that is the problem. We can't just cut the offending clients because they will have no Internet service :) Also we do not accept packets with sources in our network ranges from the outside in our border ACLs (I guess this is common antispoof practice). Hi, My understanding so far is that the queries hit your DNS servers from your ISP network/clients and are not spoofed. Also those queries hit the recursive/caching DNS servers (open only to ISP network) and they are of type ANY for a specific domain (example.com). If this is true then why not try to 'filter' on the application layer than on the network? As far as I know BIND cannot block specific queries. You might want to check unbound and local-data. Maybe you could try some kind of DNS-proxy to filter out the unwanted queries, since there is a pattern. Check http://thesprawl.org/projects/dnschef/ (haven't tested it). Alternatively you would want to implement that pf helper/'proxy' as Henning suggested which without doubt would be faster but you have to develop it. Last, run the iptables matching filter on the DNS's firewall itself (if they run linux) and not on the external firewalls. good luck, Giannis
Re: Ways to handle DNS amplification attacks with OpenBSD
On 2012-06-10, Rudolf Leitgeb rudolf.leit...@gmx.at wrote: Am Sonntag, den 10.06.2012, 00:37 + schrieb Stuart Henderson: On 2012-06-09, Kostas Zorbadelos kzo...@otenet.gr wrote: I am interested to hear possible solutions in other layers as well. http://fanf.livejournal.com/122111.html seems a nice approach... This seems to work nicely if the attacker spoofs random addresses or if the real target is not the DNS server but the endpoint receiving its replies (therefore the term amplification attack). Yes that's kind-of implied by amplification attack. Queries are sourced from a false address (the actual target of the attack), with the intention to direct larger DNS responses at the target. In Kostas's case it appears the attacker spoofs legit client addresses, which means rate limiting would likely cut off these clients. From the followup mail it sounds like these are queries being generated directly from client machines on the network i.e. DDOS rather than amplification. So what is causing the queries? Client PCs under the control of attackers? CPE routers with broken configuration allowing global access to DNS forwarders?
Re: Ways to handle DNS amplification attacks with OpenBSD
Kapetanakis Giannis bil...@edu.physics.uoc.gr writes: On 09/06/12 18:58, Kostas Zorbadelos wrote: Hi, Hi Giannis, My understanding so far is that the queries hit your DNS servers from your ISP network/clients Yes. and are not spoofed. I didn't say that. Also those queries hit the recursive/caching DNS servers (open only to ISP network) and they are of type ANY for a specific domain (example.com). Yes. If this is true then why not try to 'filter' on the application layer than on the network? I agree that this is the proper layer to address the issue. However this is not easy. As far as I know BIND cannot block specific queries. You might want to check unbound and local-data. You can't just reimplement a very busy commercial setup overnight, especially in such a critical service. Maybe you could try some kind of DNS-proxy to filter out the unwanted queries, since there is a pattern. Check http://thesprawl.org/projects/dnschef/ (haven't tested it). I will take a look at it. Alternatively you would want to implement that pf helper/'proxy' as Henning suggested which without doubt would be faster but you have to develop it. Yes. Last, run the iptables matching filter on the DNS's firewall itself (if they run linux) and not on the external firewalls. This is what we did and what we will do again. good luck, Giannis Kostas
Ways to handle DNS amplification attacks with OpenBSD
Hello all, there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 We used IPtables and the string module to match a specific signature of the problematic queries and it worked quite well (in our attack case the problematic queries had a very specific and simple pattern). The question is, if we had OpenBSD and PF as a firewall what could we do to address this? From searching the archives I saw this quite old post http://www.monkey.org/openbsd/archive/misc/0207/msg00743.html I haven't seen any string matching capability in PF for the packet payload. Unless I am missing something, what would your suggestions be in such a scenario? I am interested to hear possible solutions in other layers as well. Regards, Kostas
Re: Ways to handle DNS amplification attacks with OpenBSD
* Kostas Zorbadelos kzo...@otenet.gr [2012-06-09 13:12]: We used IPtables and the string module to match a specific signature of the problematic queries and it worked quite well (in our attack case the problematic queries had a very specific and simple pattern). The question is, if we had OpenBSD and PF as a firewall what could we do to address this? From searching the archives I saw this quite old post http://www.monkey.org/openbsd/archive/misc/0207/msg00743.html I haven't seen any string matching capability in PF for the packet payload. Unless I am missing something, what would your suggestions be in such a scenario? I am interested to hear possible solutions in other layers as well. string matching to more or less random packets' payload in the kernel? that is beyond insane. the proper solution is a small userland helper process, using divert-to and maybe socket splicing. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: Ways to handle DNS amplification attacks with OpenBSD
Kostas Zorbadelos kzo...@otenet.gr writes: Hello all, Hi there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. So you run resolvers for your clients. I will assume you're an ISP. In that case, you should be checking that the DNS queries that seem to come from your clients *actually* come from your clients, not out of nowhere, from spoofed IPs. This could be done very easily with PF, *if* your current architecture allows it (if you have a way to distinguish network flow coming from your clients from spoofed requests coming from the Internet). Of course, if you're not an ISP, then forget what I said. The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 Indeed, that involves authoritative nameservers flooded with requests that can come from anywhere. [...] -- Jérémie Courrèges-Anglas GPG fingerprint: 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494
Re: Ways to handle DNS amplification attacks with OpenBSD
Kostas Zorbadelos kzo...@otenet.gr writes: there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. My first impulse when reading the sans diary item was to rate-limit, possibly via the overload table mechanism, and if not blocking them outright perhaps put the DNS requests from the overloads in a minimal-bandwidth queue. That may or may not be appropriate to your context, and I suspect detection may be the main priority. While string matching in PF is not an option, I vaguely remember snort users coming up with patterns to match earlier DNS tomfoolery, so there's a chance you may be able to get useful info and possibly even a working snort setup to deal with this one. -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Ways to handle DNS amplification attacks with OpenBSD
On Sat, 09 Jun 2012 13:51:00 +0200, jca+o...@wxcvbn.org (Jérémie Courrèges-Anglas) wrote: Kostas Zorbadelos kzo...@otenet.gr writes: Hello all, Hi there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. So you run resolvers for your clients. I will assume you're an ISP. In that case, you should be checking that the DNS queries that seem to come from your clients *actually* come from your clients, not out of nowhere, from spoofed IPs. This could be done very easily with PF, *if* your current architecture allows it (if you have a way to distinguish network flow coming from your clients from spoofed requests coming from the Internet). Does it affect cashing name server only or the one with zones to i know its stupid question because the authoritative server have to be open for all to redistribute domain ( or not for example we do not want some regions to access our domain ?) Of course, if you're not an ISP, then forget what I said. The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 Indeed, that involves authoritative nameservers flooded with requests that can come from anywhere. [...] -- Jérémie Courrèges-Anglas GPG fingerprint: 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494
Re: Ways to handle DNS amplification attacks with OpenBSD
On Sat, 09 Jun 2012 14:08:58 +0200 Peter N. M. Hansteen wrote: While string matching in PF is not an option, I vaguely remember snort users coming up with patterns to match earlier DNS tomfoolery, so there's a chance you may be able to get useful info and possibly even a working snort setup to deal with this one. I've made custom rules scanning for user names with Snort and it was pretty easy. I had little performance concerns though so if possible, minimising the packet percentage handed to Snort or analysed would obviously be important.
Re: Ways to handle DNS amplification attacks with OpenBSD
Am Samstag, den 09.06.2012, 14:11 +0300 schrieb Kostas Zorbadelos: The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 We used IPtables and the string module to match a specific signature of the problematic queries and it worked quite well (in our attack case the problematic queries had a very specific and simple pattern). Mitigating this with snort looks much uglier than the beautiful and elegant iptables counter measure posted in this list. Not sure how it holds up under load, though. Since the attacker uses fixed patterns, he/she seems to be a script kiddy, and there is a good chance that the TTL can be used to identify his/her packets. My approach would be to check what TTLs the packets have vs. those from your clients and see whether you can filter based on that. Rudi
Re: Ways to handle DNS amplification attacks with OpenBSD
Hi, will try to comment to many posts at once :) Kostas Zorbadelos kzo...@otenet.gr writes: there is a need to restrict a specific type of DNS queries (ANY queries) in our nameservers. We faced a DDoS attack in our resolvers and the thing is that we could not simply cut access to DNS resolution to specific client IPs, the queries came from our own unsuspecting customers. pe...@bsdly.net (Peter N. M. Hansteen) writes: My first impulse when reading the sans diary item was to rate-limit, possibly via the overload table mechanism, and if not blocking them outright perhaps put the DNS requests from the overloads in a minimal-bandwidth queue. That may or may not be appropriate to your context, and I suspect detection may be the main priority. I also thought about this one, but unfortunately it would affect service for the customers and it would be unacceptable in this case. We would also need to automate detection as you mentioned. While string matching in PF is not an option, I vaguely remember snort users coming up with patterns to match earlier DNS tomfoolery, so there's a chance you may be able to get useful info and possibly even a working snort setup to deal with this one. From the little I know about snort, I am not sure what would the action be in case of detection. The desired outcome is to filter the problem traffic not allowing it to reach the nameservers. Perhaps a working setup could be achieved, rearranging however the network setup. jca+o...@wxcvbn.org (Jérémie Courrèges-Anglas) writes: So you run resolvers for your clients. I will assume you're an ISP. Yes. In that case, you should be checking that the DNS queries that seem to come from your clients *actually* come from your clients, not out of nowhere, from spoofed IPs. This could be done very easily with PF, *if* your current architecture allows it (if you have a way to distinguish network flow coming from your clients from spoofed requests coming from the Internet). Ohh, they do and that is the problem. We can't just cut the offending clients because they will have no Internet service :) Also we do not accept packets with sources in our network ranges from the outside in our border ACLs (I guess this is common antispoof practice). Of course, if you're not an ISP, then forget what I said. The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 Indeed, that involves authoritative nameservers flooded with requests that can come from anywhere. Exactly. Henning Brauer lists-open...@bsws.de writes: string matching to more or less random packets' payload in the kernel? that is beyond insane. I am interested to know if this has caused problems in IPtables' setups. It sounds dangerous, however Linux systems provide the capability. I guess they have thought about consequences and hopefully somehow addressed them. the proper solution is a small userland helper process, using divert-to and maybe socket splicing. I am not sure we are talking about the same thing (you must have an implementation clearly in your mind ;-) ), but my feeling for a proper way to address this problem is via a userland application in a proxy or intercepting mode. This could filter the offending traffic and give to the nameserver the rest to service. I think you also talk about this (correct me if I am wrong). The main problem with it is that it needs to be developed :) Perhaps relayd is a good match for this one. IPtables' string module however is here now and it provided a crude but working solution. Thanks for all the comments. Kostas
Re: Ways to handle DNS amplification attacks with OpenBSD
* Kostas Zorbadelos kzo...@otenet.gr [2012-06-09 18:02]: Henning Brauer lists-open...@bsws.de writes: string matching to more or less random packets' payload in the kernel? that is beyond insane. I am interested to know if this has caused problems in IPtables' setups. It sounds dangerous, however Linux systems provide the capability. I guess they have thought about consequences and hopefully somehow addressed them. your guess is wrong... they might have been lucky so far, or not, I don't follow all the itables bugs. the proper solution is a small userland helper process, using divert-to and maybe socket splicing. I am not sure we are talking about the same thing (you must have an implementation clearly in your mind ;-) ), but my feeling for a proper way to address this problem is via a userland application in a proxy or intercepting mode. This could filter the offending traffic and give to the nameserver the rest to service. that is pretty much what it comes down to, tho writing these proxies is very easy these days, using the techniques i mentioned above. I think you also talk about this (correct me if I am wrong). The main problem with it is that it needs to be developed :) right. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: Ways to handle DNS amplification attacks with OpenBSD
Rudolf Leitgeb rudolf.leit...@gmx.at writes: Am Samstag, den 09.06.2012, 14:11 +0300 schrieb Kostas Zorbadelos: The situation is similar but not the same as the one discribed here: https://isc.sans.edu/diary.html?storyid=13261 We used IPtables and the string module to match a specific signature of the problematic queries and it worked quite well (in our attack case the problematic queries had a very specific and simple pattern). Mitigating this with snort looks much uglier than the beautiful and elegant iptables counter measure posted in this list. Not sure how it holds up under load, though. In our case (nameservers handling thousands of queries per second) and during the time of attack multiple times that, it worked with negligible performance impact. The actual network traffic however was in the order of 40-50 Mbps per server. Since the attacker uses fixed patterns, he/she seems to be a script kiddy, and there is a good chance that the TTL can be used to identify his/her packets. My approach would be to check what TTLs the packets have vs. those from your clients and see whether you can filter based on that. What do you mean identify and filter based on TTL? In our case the attacker used a specific query for a single domain. Rudi Kostas
Re: Ways to handle DNS amplification attacks with OpenBSD
Am Samstag, den 09.06.2012, 19:17 +0300 schrieb Kostas Zorbadelos: What do you mean identify and filter based on TTL? In our case the attacker used a specific query for a single domain. I mean the TTL field from the IP header of these packets. While the attacker's packets spoof the sender address, they might not spoof the TTL, and probably being away more hops from your servers than your clients, their packets should have lower TTL values. A network traffic dump could show quickly whether this approach could possibly work. Cheers, Rudi PS: Obviously a skilled attacker can also crank up TTL values to compensate for their longer route, but fixed pattern indicates to me that you deal with a script kiddie here.
Re: Ways to handle DNS amplification attacks with OpenBSD
On 2012-06-09, Kostas Zorbadelos kzo...@otenet.gr wrote: I am interested to hear possible solutions in other layers as well. http://fanf.livejournal.com/122111.html seems a nice approach...
