Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
I have the same issue since some weeks. The problem is that the customer does not understand the problem. So if Netgear has solved the problem in a new firmware the customer should update it, but does he knows how to do this??? What can you do to limit this stupid traffic: - rate limit the queries per customer (not really a good idea) - rate limit this special kind of queries. (that's the best way at the moment) I haven't had the time to look into the packets to limit this queries. If they are all similiar you can set up a drop filter in the iptables like you should already have with the isc.org ANY requests. - Problem not really solved but you should be happy with this :-) the rule should be: $IPTABLES -I INPUT -p udp --dport 53 -m string --from 47 --algo bm --hex-string '|FF0001|' -m recent --set --name dnsanyquery $IPTABLES -I INPUT -p udp --dport 53 -m string --from 47 --algo bm --hex-string '|FF0001|' -m recent --name dnsanyquery --rcheck --seconds 600 --hitcount 3 -j DROP but what's the hex string for this kind of query. anybody got it? Freundliche Grüsse sasag Kabelkommunikation AG Michael Richter Professional Bachelor ODEC in Engineering mrich...@sasag.ch 052 633 01 71 Von: swinog-boun...@lists.swinog.ch [swinog-boun...@lists.swinog.ch]quot; im Auftrag von quot;Benoit Panizzon [benoit.paniz...@imp.ch] Gesendet: Freitag, 24. Mai 2013 12:03 An: swi...@swinog.ch Betreff: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record? Heyo! Any others who are being affected? It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers. Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight. I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-) Any others observing that behaviour of netgear products? Any know remedies? Mit freundlichen Grüssen Benoit Panizzon -- I m p r o W a r e A G- __ Zurlindenstrasse 29 Tel +41 61 826 93 07 CH-4133 PrattelnFax +41 61 826 93 02 Schweiz Web http://www.imp.ch __ ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hello Benoit On 24.05.2013 12:03, Benoit Panizzon wrote: It looks like our customers Netgear routers (known ones: WNR3500Lv2, WNDR4500) are asking our DNS Server for the A record of: time-g.netgear.com or time- a.netgear.com For me this looks like entries for timeservers (NTP). This two destination share the same IP address (so it is not a very good fail safe solution ;) : fabian@flashback:~ $ host time-g.netgear.com time-g.netgear.com is an alias for time-a.netgear.com. time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $ host time-a.netgear.com time-a.netgear.com has address 209.249.181.22 fabian@flashback:~ $ And the PTR also looks interesting (sorry for line wrapping): fabian@flashback:~ $ host 209.249.181.22 22.181.249.209.in-addr.arpa is an alias for 22.0-127.181.249.209.in-addr.arpa. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.on-networks.com. 22.0-127.181.249.209.in-addr.arpa domain name pointer time-a.netgear.com. fabian@flashback:~ $ This IP address does answer to ntp requests (sorry again for line wrapping): fabian@flashback:~ $ ntpdate -q 209.249.181.22 server 209.249.181.22, stratum 1, offset 0.004557, delay 0.19078 24 May 12:41:50 ntpdate[55957]: adjust time server 209.249.181.22 offset 0.004557 sec fabian@flashback:~ $ Instead of an A record reply, they get a CNAME as answer with additional information the A record of that CNAME. That is what netgear has published on their DNS Servers. It could be, that Netgear did change something in their DNS configuration (eg. moving time-g from A record to CNAME), which the used ntpd or sntp on this routers do not understand and so do re-request the DNS entry again because it could not sync the time. Those routers are not happy with that reply and just start sending several hundred requests per second for A time-g.netgear.com resulting in considerable load and traffic on our DNS caches. Some customers have already transfered 35GB of DNS traffic, only since today midnight. Are the high requests numbers only for time-g.netgear.com and not for time-a.netgear.com? If yes, this could prove the above idea of ntpd/sntp on this devices not properly working with a CNAME entry. Do you have configuration access to such routers? If yes, check the entries for NTP and probably change some of them e.g. to ch.pool.ntp.org and/or 1.ch.pool.ntp.org. I have contacted netgear technical support. The issue is yet unknown to them. They got my pcap files to analyze :-) It could eventually be a good idea to also point them to this DNS entries, eventually the time-g server died and the sysadmin added the CNAME without knowing the impact this could have. bye Fabian ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
On 2013-05-24 12:52 , Michael Richter wrote: [..] What can you do to limit this stupid traffic: - rate limit the queries per customer (not really a good idea) - rate limit this special kind of queries. (that's the best way at the moment) I haven't had the time to look into the packets to limit this queries. If they are all similiar you can set up a drop filter in the iptables like you should already have with the isc.org ANY requests. - Problem not really solved but you should be happy with this :-) [..] but what's the hex string for this kind of query. anybody got it? You want to deploy RRL. iptables is not the right location for doing this kind of stuff as you will have false positives. Please see http://www.redbarn.org/dns/ratelimits Greets, Jeroen ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
[swinog] WG: DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hmm I thought it is better you'll do the rate limiting on a lower layer. It's the same fix. you give the customer x queries in y time. But with RRL I think every query is counted. With iptables you can say, just count the ANY queries. So it's more specific Freundliche Grüsse sasag Kabelkommunikation AG Michael Richter Professional Bachelor ODEC in Engineering mrich...@sasag.ch 052 633 01 71 Von: Jeroen Massar [jer...@massar.ch] Gesendet: Freitag, 24. Mai 2013 13:43 An: Michael Richter Cc: Benoit Panizzon; swi...@swinog.ch Betreff: Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record? On 2013-05-24 12:52 , Michael Richter wrote: [..] What can you do to limit this stupid traffic: - rate limit the queries per customer (not really a good idea) - rate limit this special kind of queries. (that's the best way at the moment) I haven't had the time to look into the packets to limit this queries. If they are all similiar you can set up a drop filter in the iptables like you should already have with the isc.org ANY requests. - Problem not really solved but you should be happy with this :-) [..] but what's the hex string for this kind of query. anybody got it? You want to deploy RRL. iptables is not the right location for doing this kind of stuff as you will have false positives. Please see http://www.redbarn.org/dns/ratelimits Greets, Jeroen ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hey All If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;) Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm... but what's the hex string for this kind of query. anybody got it? Had there somebody fun with Stefans presentation of yesterday...? ;) -- Best regards, Roman Hochuli Operations Manager nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent Imagination is the one weapon in the war against reality. -- Jules de Gaultier ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
You want to deploy RRL. iptables is not the right location for doing this kind of stuff as you will have false positives. Please see http://www.redbarn.org/dns/ratelimits I agree that iptables might not be the perfect solution for that, however, as we have also been confronted with that problem some months ago with a lot of affected devices (each with 1000pps of those queries) we have limited those queries for some time as it is easy to deploy quickly. At that point of time time-g.netgear.com had no entry at all so the clients did not stop with the folding. Today it looks a little different, as there is at least a cname for that entry. We have used the u32 module for matching, we check name=time-g.netgear.com and type=A within the query. The matching line looks like: iptables -A INPUT -p udp --dport 53 -m u32 --u32 0x00x160x3c@0x14=0x674696d0x00x160x3c@0x18=0x652d67070x00x160x3c@0x1c=0x6e6574670x00x160x3c@0x20=0x656172030x00x160x3c@0x24=0x636f6d000x00x160x3c@0x280x=0x1 -j YOUR_CHAIN_OR_WHATEVER You can then use the limit module for example. Just as a thought, maybe it would change something to send the clients a ntp server in the dhcp response, as it is obviously looking for an ntp server. Has someone maybe already tried that? Cheers, Gregor ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hey all We had the same problem, at last a few weeks ago. We reported it to Netgear in Dec 12 for the first Time - no result We tried to overwrite these records with another - just for testing. The routers were still asking k-times a second. I think it's not a DNS-problem, cause it doesn't matter what's the answer on a request is, the router is still asking. Only a reboot of device stops the 'attack'. Best Regards Beat -Ursprüngliche Nachricht- Von: swinog-boun...@lists.swinog.ch [mailto:swinog-boun...@lists.swinog.ch] Im Auftrag von Roman Hochuli Gesendet: Freitag, 24. Mai 2013 14:33 An: swi...@swinog.ch Betreff: Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record? Hey All If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;) Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm... but what's the hex string for this kind of query. anybody got it? Had there somebody fun with Stefans presentation of yesterday...? ;) -- Best regards, Roman Hochuli Operations Manager nexellent ag Saegereistrasse 33 CH-8152 Glattbrugg Phone: +41 44 872 20 00 Fax: +41 44 872 20 01 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent Imagination is the one weapon in the war against reality. -- Jules de Gaultier ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Hi Jeroen You want to deploy RRL. Please see http://www.redbarn.org/dns/ratelimits Excellent, thank you. Didn't know that bind feature. Mit freundlichen Grüssen Benoit Panizzon -- I m p r o W a r e A G- __ Zurlindenstrasse 29 Tel +41 61 826 93 07 CH-4133 PrattelnFax +41 61 826 93 02 Schweiz Web http://www.imp.ch __ ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
Thanks Gregor!! that was exactly what I was looking for. have a nice weekend --- You want to deploy RRL. iptables is not the right location for doing this kind of stuff as you will have false positives. Please see http://www.redbarn.org/dns/ratelimits I agree that iptables might not be the perfect solution for that, however, as we have also been confronted with that problem some months ago with a lot of affected devices (each with 1000pps of those queries) we have limited those queries for some time as it is easy to deploy quickly. At that point of time time-g.netgear.com had no entry at all so the clients did not stop with the folding. Today it looks a little different, as there is at least a cname for that entry. We have used the u32 module for matching, we check name=time-g.netgear.com and type=A within the query. The matching line looks like: iptables -A INPUT -p udp --dport 53 -m u32 --u32 0x00x160x3c@0x14=0x674696d0x00x160x3c@0x18=0x652d67070x00x160x3c@0x1c=0x6e6574670x00x160x3c@0x20=0x656172030x00x160x3c@0x24=0x636f6d000x00x160x3c@0x280x=0x1 -j YOUR_CHAIN_OR_WHATEVER You can then use the limit module for example. Just as a thought, maybe it would change something to send the clients a ntp server in the dhcp response, as it is obviously looking for an ntp server. Has someone maybe already tried that? Cheers, Gregor ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
On 2013-05-24 14:33 , Roman Hochuli wrote: Hey All If it is really hurting you big time you may choose to run a very mean hack: temporarily setup a netgear.com-Zone on your dns-servers and point these records to a useful NTP server. Adding an A-record for their website would probably a good idea as well. ;) Yes, it is an EXTREMELY UGLY HACK. But as stated above: it might be easier to cut yourself a hand off than loosing the whole arm... If you are doing that, which is a workable solution for you and your customers do it only for time-g.netgear.com by defining a zone for that and using '@' to get the record defined, that way you don't cause colateral damage to the many other records that might exist in netgear.com. Of course the best avenue is to contact netgear and really hammer on them to fix it; though it seems to be a device issue, not a DNS problem. but what's the hex string for this kind of query. anybody got it? Had there somebody fun with Stefans presentation of yesterday...? ;) Tranalyzer only analyzes as far as I recall and the slides do not indicate differently... Greets, Jeroen ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
Re: [swinog] DDOS DNS Attack by Netgear Products caused by CNAME instead of A record?
netgear tech support has confirmed there is a problem in one of the models and released a fixed firmware. Great! How do I tell the customers? Mit redirect the http traffic for those customer to an webpage which explain and offer a download link on that page maybe combine with a dns ratelimit for a while. my 5 cents ... ___ swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
