Re: register.com down sev0?
On Sat, Oct 28, 2006 at 12:39:31AM -0500, Chris Owen wrote: The spam I got was directly from register.com. It came with a register.com return email address, pointed to a register.com web site and came from an IP address the resolved to *.register.com (I will admit I didn't confirm the netblock belonged to them). I've never done any business with them and the spam was for a domain name renewal for a domain registered elsewhere. In other words, it was a classic whois scrapped spam. Some clarification: the information is probably not being scraped via WHOIS. You're not allowed to scrape via WHOIS. Deceptive companies who want to get around this simply buy the WHOIS records (I should be more precise: the data that would appear in a WHOIS lookup) from the registrar directly. I can point you to an Email thread discussing this find, which includes couple statements from OpenSRS's Product Manager (who in a roundabout way admitted that anyone can buy their WHOIS database), if you'd like. This doesn't explain the spam, but it I really do not see any purpose to buying a registrar's copy of customer WHOIS records other than for mass-marketing. This is bad business in general. As I've previously said, this isn't like its some sort of borderline case where someone in one part of the company is doing something that someone else doesn't know about. These guys are pretty hard core. I'd say I get 20-30 emails a year from them for various domain names I'm a contact on. I've also received USPS spam which is another story but no less unethical since they are all these BS renewal type letters. They might not be Domain Registry of America but they are hardly innocent. I've mentioned this on NANOG before. See the thread about why I refuse to put legitimate contact information (Email contact information is always valid; just not the address or phone number) in our domain WHOIS records. The DROA is half of the reason; the other half is what I described above. The entire situation is depressing, solely because ICANN is doing absolutely nothing to try and stop this sort-of behaviour (both what the DROA does, and registrars selling their customers' WHOIS records to whoever bids the most for it). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
ICANN Registrar Policy [Was: Re: register.com down sev0?]
On a semi-related note, I feel compelled to add that it seems to be getting worse with regards to due diligence paid by domain registrars in how domains are being issued, as well: http://www.f-secure.com/weblog/#1008 - ferg -- Jeremy Chadwick [EMAIL PROTECTED] wrote: [snip] The entire situation is depressing, solely because ICANN is doing absolutely nothing to try and stop this sort-of behaviour (both what the DROA does, and registrars selling their customers' WHOIS records to whoever bids the most for it). [snip] -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: register.com down sev0?
I submitted both spams to spamcop and the appropriate abuse addresses would have been notified in both cases. I got no response from either of my submissions. As for a reason for ignoring my complaint I really couldn't say since, well they ignored me. Did you ever send a complaint to [EMAIL PROTECTED] and [EMAIL PROTECTED] personally (so that you could actually verify it was sent and delivered)? I've never dealt with a company that didn't at least acknowledge receipt of a complaint. -Don
Re: register.com down sev0?
On Oct 28, 2006, at 10:52 AM, Donald Stahl wrote: I submitted both spams to spamcop and the appropriate abuse addresses would have been notified in both cases. I got no response from either of my submissions. As for a reason for ignoring my complaint I really couldn't say since, well they ignored me. Did you ever send a complaint to [EMAIL PROTECTED] and [EMAIL PROTECTED] personally (so that you could actually verify it was sent and delivered)? I've never dealt with a company that didn't at least acknowledge receipt of a complaint. Then you must not deal with very many companies. (Not a comment on Register.com, 'cause I don't, and will never, know if they respond since I block their mail to avoid bogus renewal notices.) -- TTFN, patrick
Re: register.com down sev0?
Donald Stahl wrote: I submitted both spams to spamcop and the appropriate abuse addresses would have been notified in both cases. I got no response from either of my submissions. As for a reason for ignoring my complaint I really couldn't say since, well they ignored me. Did you ever send a complaint to [EMAIL PROTECTED] and [EMAIL PROTECTED] personally (so that you could actually verify it was sent and delivered)? I've never dealt with a company that didn't at least acknowledge receipt of a complaint. -Don I send out several hundred complaints monthly and have acknowledgments from only 20%. Most of the acknowledgments are automated. Either I or the abuse@ system is broken - perhaps both. Randy
Re: register.com down sev0?
On Fri, 27 Oct 2006, Joseph S D Yao wrote: On Wed, Oct 25, 2006 at 10:10:05PM -0400, [EMAIL PROTECTED] wrote: ... As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. ... ns1.register.com. 600 IN A 216.21.234.96 ns2.register.com. 600 IN A 216.21.226.96 ns3.register.com. 600 IN A 216.21.234.97 ns4.register.com. 600 IN A 216.21.226.97 I am not saying that register.com IS doing this, just that you can't say that they're NOT just from this evidence. I think Alex could have included a few lines of traceroute to these hosts showing that they all end behind: 7 tbr1-p014001.wswdc.ip.att.net (12.123.8.98) 9.754 ms 9.685 ms 9.608 ms 8 tbr1-cl4.sl9mo.ip.att.net (12.122.10.30) 29.708 ms 29.593 ms 33.498 ms 9 12.122.85.178 (12.122.85.178) 36.300 ms 28.558 ms 28.521 ms So... it sorta looks like both /24's are behind something in StLouis, Missouri ( to me atleast ).
Re: register.com down sev0?
Once upon a time, Chris L. Morrow [EMAIL PROTECTED] said: I think Alex could have included a few lines of traceroute to these hosts showing that they all end behind: 7 tbr1-p014001.wswdc.ip.att.net (12.123.8.98) 9.754 ms 9.685 ms 9.608 ms 8 tbr1-cl4.sl9mo.ip.att.net (12.122.10.30) 29.708 ms 29.593 ms 33.498 ms 9 12.122.85.178 (12.122.85.178) 36.300 ms 28.558 ms 28.521 ms Also, it looks like anyone filtering on ARIN boundaries won't even see that. Register.com has 216.21.224.0/20 assigned, but announces 7 /24s and 2 /22s out of it. -- Chris Adams [EMAIL PROTECTED] Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: register.com down sev0?
On Sat, 2006-10-28 at 17:36 +, Chris L. Morrow wrote: So... it sorta looks like both /24's are behind something in StLouis, Missouri ( to me atleast ). My tests from 2 years ago showed the same thing, both /24s were behind the same system in Exodus' NYC DC in Manhattan (IIRC). That is what prompted me to move everything to the rcom partner side which uses eNom. -Jim P.
Re: register.com down sev0?
My tests from 2 years ago showed the same thing, both /24s were behind the same system in Exodus' NYC DC in Manhattan (IIRC). That is what prompted me to move everything to the rcom partner side which uses eNom. I don't know about a partner side but their premium service was always run by Register.com themselves. The servers were in a number of locations across the world. Whether any of this remains true today I have no idea. Register.com may have also resold eNom services but I doubt that had anything to do with their premium service. -Don
RE: register.com down sev0?
It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Yes, I realize that the latter approach is more of a managerial hassle, but for those of you who feel that your silicon is running a tad too warm, you may wish to consider this as a possible performance improvement technique. YMMV. Your former router vendor, Tony
different flavours of uRPF [RE: register.com down sev0?]
On Thu, 26 Oct 2006, Tony Li wrote: It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Isn't that only a problem if you want to run a loose mode uRPF? Given that loose mode uRPF isn't very useful in most places where you'd like to do ingress filtering, this doesn't seem like a big issue.. BTW, I still keep wondering why Cisco hasn't implemented something like Juniper's feasible-path strict uRPF. Works quite well with multihomed and asymmetric routing as well -- no need to fiddle with communities, BGP weights etc. to ensure symmetry. -- Pekka Savola You each name yourselves king, yet the Netcore Oykingdom bleeds. Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Re: register.com down sev0?
but i am not foolish enough to believe that religious ranting on mailing lists is gonna change anyone from doing what makes business sense for their network. Indeed! And it is not going to change the minds of the majority of network operations folks who are not on the NANOG list nor the majority of telecoms executives who are also not on the NANOG list. Back in the old days, the NANOG list did hold the majority of Internet operations folks so new ideas like flap dampening were able to spread quickly. But those days are long gone. NANOG still has an important educational role but it is no longer based on being part of the old boys club and knowing the secret handshake. In other words, there is no cohesive society of network operators which can be swayed by attempts at social engineering like shaming or cajoling. BCP 38 has had its day. Nowadays, it is more important to look at how to mitigate current DDoS techniques and to describe the larger problem and look for larger solutions. However, any attempt at larger solutions require a large amount of humility because nobody can say for sure, what will work and what won't. The fact remains that there is not a good technical method for mitigating large scale distributed DDoS that results in LARGE TRAFFIC FLOWS ENTERING A NETWORK FROM ALL PEERED ASES SIMULTANEOUSLY. Perhaps if we could find a way to allow the attacked AS to set ACLs automatically in all the source AS networks, that would help mitigate these attacks. For instance, consider a set of ASes which all install an ACL-setter box. These boxes all trust each other to send-receive ACL setting requests through a trusted channel. The owner of a box sets some limits on the ACLs that can be set, for instance n ACLs per AS, max ACL lifetime, etc. And the box owner also decides the subset of their routers which will accept an ACL for a given address range. Then when an attack comes in, the victim AS uses some tool to identify large sources, i.e. a CIDR block that covers some significant percentage of the source addresses in one AS. They then issue an ACL request to that AS to block the flow and the ACL takes effect almost instantaneously with no human intervention. Yes, this can result in some IP addresses being blocked unfairly, but the DDoS traffic levels often have the same impact. In any case, the AS holding the destination address is the one doing the blocking even though the mechanism is an ACL inside the source AS network. On the technical side, it is not a complex problem to put such a system in place. The complexity is largely in getting network operators to come to an agreement on the terms under which operator A will allow operator B to set ACLs in operator A's network. Until network operators see DDoS as a significant business problem, this will not happen. Note that a business problem does not refer solely to the direct costs of mitigating a DDoS attack. It also includes the indirect fallout which is harder to measure such as loss of goodwill, missed opportunities, etc. --Michael Dillon
RE: register.com down sev0?
On Thu, 26 Oct 2006, Tony Li wrote: It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Yes, I realize that the latter approach is more of a managerial hassle, but for those of you who feel that your silicon is running a tad too warm, you may wish to consider this as a possible performance improvement technique. YMMV. Your former router vendor, Tony Erm, most ISP's I talk to (since I became aware of this not too long ago) believe this is a perfect replacement for BCP38. And yet, spoofing is possible from their space. Gadi.
Re: different flavours of uRPF [RE: register.com down sev0?]
Pekka Savola wrote: On Thu, 26 Oct 2006, Tony Li wrote: It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Isn't that only a problem if you want to run a loose mode uRPF? Given that loose mode uRPF isn't very useful in most places where you'd like to do ingress filtering, this doesn't seem like a big issue.. Strict mode uRPF is likely to be implemented by performing a full forwarding table lookup and then comparing the packet's incoming interface to the interface from the forwarding table result. Tony
Re: register.com down sev0?
Hi Vadim! Vadim Antonov wrote: On Thu, 26 Oct 2006, Tony Li wrote: Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes... That would be, well, about 6 memory reads. Radix trees are great. They are indeed. If a radix trie is indeed used, you would expect to see about log2(200,000) + 1 = 19 reads on average. or against a handful of prefixes that you've statically configured in an ACL. Which will take much longer with line-by-line sequential matching. Fortunately, modern ACL implementations frequently use TCAMs (1 read) or tree based structures (log2(handful) + 1) as well. As always, the details of a particular implementation are everything. YMMV. Tony
Re: different flavours of uRPF [RE: register.com down sev0?]
On Fri, 27 Oct 2006, Tony Li wrote: Pekka Savola wrote: On Thu, 26 Oct 2006, Tony Li wrote: It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Isn't that only a problem if you want to run a loose mode uRPF? Given that loose mode uRPF isn't very useful in most places where you'd like to do ingress filtering, this doesn't seem like a big issue.. Strict mode uRPF is likely to be implemented by performing a full forwarding table lookup and then comparing the packet's incoming interface to the interface from the forwarding table result. Pekka might have meant wouldn't you build a seperate 'urpf table' per interface perhaps? (just guessing at his intent) though there is only one 'urpf table' which is the fib, right?
RE: different flavours of uRPF [RE: register.com down sev0?]
It was possible to implement BCP38 before the router vendors came up with uRPF. Further, uRPF is frequently a very inefficient means of implementing BCP 38. Consider that you're going to either compare the source address against a table of 200,000 routes or against a handful of prefixes that you've statically configured in an ACL. Isn't that only a problem if you want to run a loose mode uRPF? Given that loose mode uRPF isn't very useful in most places where you'd like to do ingress filtering, this doesn't seem like a big issue.. Loose mode is a RTBH Reaction tool - not BCP 38. Don't use a screw driver to hammer a nail. BTW, I still keep wondering why Cisco hasn't implemented something like Juniper's feasible-path strict uRPF. Works quite well with multihomed and asymmetric routing as well -- no need to fiddle with communities, BGP weights etc. to ensure symmetry. Wow - I'm going to need to dust off the tutorial materials on how uRPF and using the FIB as a policy enforcement tool works. Does uRPF need to scan through the entire FIB? Saying this is saying routers look through the entire FIB table to find the next hop? What ever happened to TRIE techniques? uRPF's look up is at the same speed as the forwarding look up. In fact, in many implementations, the forwarding lookup gets the source and destination address values from the FIB. Now, are there other ways of doing BCP38 - yes lots: - ACLs - Radius loaded ACLs - uRPF Strict-Feasible-VRF modes - IP Source Verify - DHCP Lease Query - NAT on the home CPE Why hasn't Cisco done uRPF Feasible path? Cause until recently, our CEF structures would not allow for feasible/alternate paths. If the FIB is your policy table, then _what_ you are limited to the capabilities of that FIB when using it to police the packet. Cisco has that now, so feasible path is just a matter of time to work through the coding queues. What I'm shaking my head over with this whole dialog is the 1990's thinking. BCP38 is out of date. Anyone who works, mitigates, analysis, and studies attack vectors on network systems know that checking the IP source address is one of many Anti-Spoof checks you need to do on the packet. With Ethernet and Cable, you need to do a MAC check. With all mediums you need to check the Prec/DSCP value (porn at Prec 6 does wonders for the routing protocols when there is congestion in the path). Then there is TTL values, Fragments, and other values which need to be policed on the edge. This is why uRPF - while helpful - is not the primary BCP38 tool people should be considering on the edge.
Re: register.com down sev0?
Paul, As of right now I'm not prepared to comment on our recent outage in this forum. That said, I do want to discuss your assertion that Register.com is a source of spam. Spam mail is something we take very seriously. As a business we do not send spam email and we have procedures in place to address spam sent by our customers. If you're seeing spam involving us, and haven't gotten any traction from our abuse desk ([EMAIL PROTECTED]), I'd like to know about it. I've privately emailed you my phone number, please give me a call, so we can discuss this further. -- Charles Knipe Manager - Infrastructure Services Register.com, Inc.
Re: register.com down sev0?
Charles J. Knipe wrote: Paul, As of right now I'm not prepared to comment on our recent outage in this forum. That said, I do want to discuss your assertion that Register.com is a source of spam. It's pretty well-known that register.com has been a source of spam, and that complaints to them have been ineffective. If you're here to tell us that the problem has recently been fixed, or that you're working on fixing it, people will be happy to hear that. If you're here to tell us that there never was a problem and that we're all just imagining it... you'll need these: http://www.spectorracing.com/catalog/category_477_UNDERWEAR_SParco_Racing_Underwear_page_1.html Carmyth fabric has a higher flame resistance than any previous material
RE: register.com down sev0?
Nah. You assume branching factor of 2 (and not radix tree but rather a form of binary tree, i.e. AVL, r/b or Patricia - they have that O(log2(num_entries)) behaviour, while radix trees are traversed in O(key_length/branching_factor)). I assumed a binary radix trie (not tree) because that's the normal cannonical version used by computer science students. Yes, as you outlined, there are many games you can play, if you're willing to make space/time tradeoffs. Regardless of the details, the point remains: if your data structures are largely constant, then you are more efficient searching a small data set vs. searching a large one. Tony
Re: register.com down sev0?
It's pretty well-known that register.com has been a source of spam, and that complaints to them have been ineffective. Albert, I don't know about Register.com's opinion but I dare say the statement above isn't very helpful to me as an admin. When you say has been a source of spam is there a time frame involved? Was this in the last week? Month? Year? When you say register.com has been the source do you mean a) their netblocks b) their mail servers or c) partners acting on their behalf? You also state that complaints have been ineffective. Again is there a time frame? Did anyone get back to you? Did they investigate? Did they give you a reason for ignoring or doing nothing about your complaint? I ask this not because I want to know but because if someone from the company came here to address the issue then perhaps we should give them as much information as possible (After all- you have a contact now) Simply saying that it's pretty well-known doesn't really help. I frankly doubt they would bother posting here with let us know if they had no intention of looking into it- this isn't exactly a group likely to be pacified by empty promises. (It's also possible that in the past the right people never found out- or that there are new people there who take the issue more seriously). will be happy to hear that. If you're here to tell us that there never was a problem and that we're all just imagining it... you'll need these: I don't think they are going to claim there was never a problem- unfortunately sometimes the marketing folks don't consult or listen to their technical folks- it's happened at a lot of companies. That said- I haven't had spam from a register.com netblock in a long time. Then again maybe I've just been lucky. -Don
Re: register.com down sev0?
On Wed, Oct 25, 2006 at 10:10:05PM -0400, [EMAIL PROTECTED] wrote: ... As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. ... ns1.register.com. 600 IN A 216.21.234.96 ns2.register.com. 600 IN A 216.21.226.96 ns3.register.com. 600 IN A 216.21.234.97 ns4.register.com. 600 IN A 216.21.226.97 This is two pairs, each pair in a single /24 (or /26), and there are ways in which each of these hosts could be in a widely different spot from the other three, or in several different spots. Why am I saying this? Most of the folks here know this and how to do this even better than I do. I am not saying that register.com IS doing this, just that you can't say that they're NOT just from this evidence. And by now it's moot anyway. -- Joe Yao --- This message is not an official statement of OSIS Center policies.
Re: register.com down sev0?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2006, at 7:48 PM, Donald Stahl wrote: It's pretty well-known that register.com has been a source of spam, and that complaints to them have been ineffective. I don't know about Register.com's opinion but I dare say the statement above isn't very helpful to me as an admin. When you say has been a source of spam is there a time frame involved? Was this in the last week? Month? Year? I've received spam from them in the past month (actually I got two). When this thread started I went back to see if I could find them but unfortunately I no longer had copy. When you say register.com has been the source do you mean a) their netblocks b) their mail servers or c) partners acting on their behalf? The spam I got was directly from register.com. It came with a register.com return email address, pointed to a register.com web site and came from an IP address the resolved to *.register.com (I will admit I didn't confirm the netblock belonged to them). I've never done any business with them and the spam was for a domain name renewal for a domain registered elsewhere. In other words, it was a classic whois scrapped spam. You also state that complaints have been ineffective. Again is there a time frame? Did anyone get back to you? Did they investigate? Did they give you a reason for ignoring or doing nothing about your complaint? I submitted both spams to spamcop and the appropriate abuse addresses would have been notified in both cases. I got no response from either of my submissions. As for a reason for ignoring my complaint I really couldn't say since, well they ignored me. I ask this not because I want to know but because if someone from the company came here to address the issue then perhaps we should give them as much information as possible (After all- you have a contact now) Simply saying that it's pretty well-known doesn't really help. As I've previously said, this isn't like its some sort of borderline case where someone in one part of the company is doing something that someone else doesn't know about. These guys are pretty hard core. I'd say I get 20-30 emails a year from them for various domain names I'm a contact on. I've also received USPS spam which is another story but no less unethical since they are all these BS renewal type letters. They might not be Domain Registry of America but they are hardly innocent. I frankly doubt they would bother posting here with let us know if they had no intention of looking into it- this isn't exactly a group likely to be pacified by empty promises. (It's also possible that in the past the right people never found out- or that there are new people there who take the issue more seriously). Well maybe this guys is serious about addressing the problem but if they are serious as a company the least they could do is respond to complaints that come via spamcop. Hell it think most spamcop complaints we get are mostly BS but I at least bother to respond to them. will be happy to hear that. If you're here to tell us that there never was a problem and that we're all just imagining it... you'll need these: I don't think they are going to claim there was never a problem- unfortunately sometimes the marketing folks don't consult or listen to their technical folks- it's happened at a lot of companies. That said- I haven't had spam from a register.com netblock in a long time. Then again maybe I've just been lucky. I'd go with lucky then. Chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFQu0TElUlCLUT2d0RAj0DAKCR1pSj/xEqYcTZAv86NRjuVO2DzACfXKVc eQ30FesWFzLWNWlwGFW6tA4= =CIB0 -END PGP SIGNATURE-
Re: register.com down sev0?
On Fri, 27 Oct 2006, Albert Meyer wrote: Charles J. Knipe wrote: Paul, As of right now I'm not prepared to comment on our recent outage in this forum. That said, I do want to discuss your assertion that Register.com is a source of spam. It's pretty well-known that register.com has been a source of spam, and that complaints to them have been ineffective. If you're here to tell us that the problem has recently been fixed, or that you're working on fixing it, people will be happy to hear that. If you're here to tell us that there never was a problem and that we're all just imagining it... you'll need these: http://www.spectorracing.com/catalog/category_477_UNDERWEAR_SParco_Racing_Underwear_page_1.html Carmyth fabric has a higher flame resistance than any previous material Interpreting someone else and therefore wrong, he told you that if you get no help, contact him directly. I think that's pretty cool, and you will be able to tell if it works or not. Let's try and not kill people who try and help, today. Gadi.
Re: register.com down sev0?
I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? randy
Re: register.com down sev0?
Randy, I don't think I implied anything of the sort. I did, however, pipe up when a BCP is mentioned that I endorse, and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. Having said that, botnets don't need to spoof addresses -- the sheer dispersion of geographic and AS infection base renders the whole point of spoofing almost moot. And having said that, it doesn't make BCP 38 any less valid. - ferg -- Randy Bush [EMAIL PROTECTED] wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? randy -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
Actually, I misspoke earlier, but not quite. ;-) Rob Beverly has an ongoing project which I have wholly endorsed, but it has gotten relatively little attention: http://spoofer.csail.mit.edu/ I would highly recommend that folks how choose to so, please participate. :-) - ferg p.s. Statistics available: http://spoofer.csail.mit.edu/summary.php -- Sean Donelan [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? Do you have any data concerning the actual consistent deployment of BCP38++ in different parts of the world? -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
The only data I have is from the MIT anti-spoofing test project which has been pretty consistent for a long time. About 75%-80% of the nets, addressses, ASNs tests couldn't spoof, and about 20%-25% could. The geo-location maps don't show much difference between parts of the world. RIPE countries don't seem to be better or worse than ARIN countries or APNIC countries or so on. ISPs on every continent seem to be about the same. http://spoofer.csail.mit.edu/summary.php If someone finds the silver bullet that will change the remaining 25% or so of networks, I think ISPs on every continent would be interested. On Thu, 26 Oct 2006, Fergie wrote: No. I think that is indicative of the problem. Don't you? -- Sean Donelan [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? Do you have any data concerning the actual consistent deployment of BCP38++ in different parts of the world?
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
This would appear, on its face, to be an easy exercise in educating the IPSs in the foodchain. Is there reasonable enough interest with NANOG to do that? If so, I volunteer to workshop at the next NANOG. But only if there is reasonable consensus to that effect. Or someone else could do it, too. :-) The point I'm trying to make is that if the community thinks it is valuable, then the path is clear. If not, then... - ferg -- Sean Donelan [EMAIL PROTECTED] wrote: The only data I have is from the MIT anti-spoofing test project which has been pretty consistent for a long time. About 75%-80% of the nets, addressses, ASNs tests couldn't spoof, and about 20%-25% could. The geo-location maps don't show much difference between parts of the world. RIPE countries don't seem to be better or worse than ARIN countries or APNIC countries or so on. ISPs on every continent seem to be about the same. http://spoofer.csail.mit.edu/summary.php If someone finds the silver bullet that will change the remaining 25% or so of networks, I think ISPs on every continent would be interested. On Thu, 26 Oct 2006, Fergie wrote: No. I think that is indicative of the problem. Don't you? -- Sean Donelan [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? Do you have any data concerning the actual consistent deployment of BCP38++ in different parts of the world? -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Fergie wrote: The point I'm trying to make is that if the community thinks it is valuable, then the path is clear. What is the biggest problem to solve? Would it be enough for ISPs to make sure that they will not send out packets which didn't belong within their PA blocks, or is it that one user shouldn't be able to spoof at all (even IPs adjacant to their own)? Would the global problem go away if global spoofing stopped working? I of course realise that it's best if user cannot spoof at all, but it might be easier for ISPs to filter based on their PA blocks than to (in some cases) purchase new equipment to replace their current equipment that cannot do IP spoof filtering. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: 10,352 active botnets (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Fergie wrote: Jose's numbers are conservative. Given some mathematical acrobatics, I'd suggest examining some of the (shocking) number sin Microsoft's Security Intelligence Report (Google it) -- these are reflective: Of the 4 million computers cleaned by the company's MSRT (malicious software removal tool), about 50 percent (2 million) contained at least one backdoor Trojan. While this is a high percentage, Microsoft notes that this is a decrease from the second half of 2005. During that period, the MSRT data showed that 68 percent of machines cleaned by the tool contained a backdoor Trojan. Ref: http://www.eweek.com/article2/0,1759,2036439,00.asp If you're wondering why DDoS attacks are so effective, look no further than your backyard. - ferg Jose may be a bit conservative with numbers, but he has good data and shares it, which is more than I can say for some people. Jose is definitely someone who knows what he is talking about when it comes to botnets. These numbers are not really relevant in my opinion, but they help get the message across. Gadi.
Re: register.com down sev0?
On Thu, Oct 26, 2006 at 06:03:54AM +, Fergie wrote: Randy, I don't think I implied anything of the sort. I did, however, pipe up when a BCP is mentioned that I endorse, and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. The challenge is that the router vendors still haven't done The Right Thing. I have one device that 1) halves its forwarding table space by enabling u-rpf 2) can only do either strict or loose mode rpf *GLOBALLY* so I can not strict rpf-check a static customer AND loose rpf someone larger for unrouted space. because of the above (#1 isn't that bad, but #2 is) I can't enable u-rpf on the device as a policy. Changing one interface from loose - strict silently changes all other u-rpf interfaces and then customers gripe about dropped packets. obviously moving these checks closer to the edge is ideal, such as always doing rpf on the ethernet lan interface for your customer CPE. Having said that, botnets don't need to spoof addresses -- the sheer dispersion of geographic and AS infection base renders the whole point of spoofing almost moot. yup, it's an evolving threat, even if some solution to the botnet problem is discovered, it will take years to fix. Think of the smurf amplifiers that are still out there[1]. - jared 1 - http://www.powertech.no/smurf/ -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 2006-10-26 at 02:20 -0400, Sean Donelan wrote: http://spoofer.csail.mit.edu/summary.php If someone finds the silver bullet that will change the remaining 25% or so of networks, I think ISPs on every continent would be interested. Financial incentive is the key. If there is none, those with the most to gain (backbone operators) also have to power to create such incentive. It wouldn't be fundamentally different from the basic network policing that happened on the academic networks which formed the Internet backbone in the 80s and early 90s. Keywords: = Work with OS and CPE vendors to include probes with equipment/software. Create lists of badly behaved prefixes. Drop offending prefixes from the DFZ. -- Result: BPC compliance or go the scenic route (bust). Problem solved .. move on. Problems: = Politics. Ill-informed politicians can come up with the most incredible excuses to protect offenders. Decide who define the criteria used to identify offending networks, and administer the filtering recommendation. Has to tolerate some collateral damage. Widespread misconception of an untouchable public internet. Such a thing doesn't exist. The net still consist of interconnected privately owned networks within which the owner/operator is free to implement and enforce whatever policies they want. Some countries may require that customers/users are informed about the existence and consequences of such restrictions, but that shouldn't be much of a problem either. I'd be more than happy to tell anyone who object to BCP38 to look elsewhere for network connectivity. -- Per Heldal - http://heldal.eml.cc/
Re: register.com down sev0?
On Thu, 2006-10-26 at 06:03 +, Fergie wrote: Having said that, botnets don't need to spoof addresses -- the sheer dispersion of geographic and AS infection base renders the whole point of spoofing almost moot. A lot of new possibilities arise if spoofing can be eliminated with near 100% certainty. Some examples: Automated filtering. Automated notification to providers. Cut off host X or... Expose compromised systems and hold their owners financially responsible for damages. Severe punishment of large number of users may cause outrage, basis for regress, class-action lawsuits, and maybe finally turn the attention to the real source of the problem; software vendors whose products are of such a dismal quality that they'd be banned worldwide from just about any market other than that for computer software. -- Per Heldal - http://heldal.eml.cc/
Re: register.com down sev0?
On Thu, Oct 26, 2006 at 12:14:43AM -0400, [EMAIL PROTECTED] wrote: On 26 Oct 2006, Paul Vixie wrote: i wonder if that's due to the spam they've been sending out? Paul, this isn't nanae. Let's not sling accusations like that wildly. There's nothing wild about it -- Paul is one of the most sober, reasoned observers of the spam problem, and if he told me that my servers were sending spam, then I'd darn well go investigate. Right now. Besides -- it's not like this isn't common knowledge in the anti-spam world. I'm sure I'm not the only one who's had unsatisfying correspondence with register.com wherein they refuse to lift a finger to stop the abuse from/facilitated by their operation. ---Rsk
Re: register.com down sev0? - More information
As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. Register.com offered several models for DNS service including distributed anycast based services. Considering what I've heard about the scale of the attack I'm glad they chose not host their own domain name on the anycast networks- it simply would have taken more people down. Some facts: 1. I've spoken with some ATT engineers about what was going on. According to them this was (as mentioned earlier) a multi gigabit attack that came in through every peer on the ATT network. Anycasting would not have fixed this problem- the attack was too large and too diverse. (I guess if they had 10 gige pipes and pops all over the planet- maybe. But that's not exactly a valid business model.) 2. These were not spoofed source addresses. This looks like a rather large botnet sending real traffic. 3. The attack was large enough to affect many other customers in the same data center- one with a lot of bandwidth off ATT's backbone. 4. DNS is a tiny protocol. It's possible to send a LOT of small, but perfectly valid, DNS packets. The fact that the attack was multi gigabit per second is bad enough. Couple that with the packets all being really tiny and you have a recipe for routing disaster. 5. ATT (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter :) Think about that for a second. To stop an attack Register.com would need to call ATT and request a filter/null route. Since ATT operations is based in Singapore (again this was last time I dealt with them) I'm sure getting those filters/routes in probably doesn't happen nearly fast enough. I have heard that ATT is currently in the process of setting up communities- maybe someone who knows more could comment. The truth is that none of us has all the facts about what happened. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) Register.com is not public (If I recall correctly they were bought out a couple of years ago by a private firm). Furthermore if they were public I would think their stockholders might have something to say about spending large sums of money to prevent a DDoS which probably would not work anyway. -Don
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 26 Oct 2006 02:20:48 -0400 (EDT), Sean Donelan [EMAIL PROTECTED] wrote: The only data I have is from the MIT anti-spoofing test project which has been pretty consistent for a long time. About 75%-80% of the nets, addressses, ASNs tests couldn't spoof, and about 20%-25% could. The geo-location maps don't show much difference between parts of the world. RIPE countries don't seem to be better or worse than ARIN countries or APNIC countries or so on. ISPs on every continent seem to be about the same. http://spoofer.csail.mit.edu/summary.php If someone finds the silver bullet that will change the remaining 25% or so of networks, I think ISPs on every continent would be interested. That would be nice -- but I wonder how much operational impact that would have. As you note, the 20-25% figure (of addresses) has been pretty constant for quite a while. Assuming that subverted machines are uniformly distributed (a big assumption) and assuming that their methodology is valid (another big assumption), that means we've already knocked out the 75-80% of the sources of spoofed IP address attacks. Has anyone seen a commensurate reduction in DDoS attacks? I sure haven't heard of that. Are people saying that the problem would be several times worse if anti-spoofing weren't in place? As best I can tell, the limiting factor on attack rates isn't the lack of sources but the lack of a profit motive for launching the attacks. Put another way, anti-spoofing does three things: it makes reflector attacks harder, it makes it easier to use ACLs to block sources, and it helps people track down the bot and notify the admin. Are people actually successfully doing either of the latter two? I'd be surprised if there were much of either. That leaves reflector attacks. Are those that large a portion of the attacks people are seeing? I agree that anti-spoofing is a good idea, and I've said so for a long time. I was one of the people who insisted that ATT do it, way back when. But I'm not convinced it's a major factor here. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Re: register.com down sev0? - More information
Once upon a time, Don [EMAIL PROTECTED] said: Some facts: 3. The attack was large enough to affect many other customers in the same data center- one with a lot of bandwidth off ATT's backbone. Is this what got Red Hat over the last couple of days as well? I think they have a lot of their stuff on ATT's network. -- Chris Adams [EMAIL PROTECTED] Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: 10,352 active botnets (was Re: register.com down sev0?)
On Thursday 26 Oct 2006 13:45, you wrote: Is there a similar statistic available for Mac OS X ? Now now. Of the 4 million computers cleaned by the company's MSRT (malicious software removal tool), about 50 percent (2 million) contained at least one backdoor Trojan. While this is a high percentage, Microsoft notes that this is a decrease from the second half of 2005. During that period, the MSRT data showed that 68 percent of machines cleaned by the tool contained a backdoor Trojan. A lot depends on the definition. I've removed some malware trying to exploit an old Microsoft JRE bug. This stuff gets everywhere (well anywhere IE goes). These get downloaded to some cached program folder for Java, and because the exploit hasn't worked for years, sit there till some antivirus software comes along and removes them, doing nowt but consuming disk space. If you are the Microsoft malicious software removal tool marketing department, that is a trojan removed. To the average person on the street, it is another bit of meaningless fluff their PC will lose when they reinstall. So yes, Microsoft is big enough to have bits who have a vested interest in making the other bits look bad (if only incidentally). Thus is the way of big companies.
DNS DDoS [was: register.com down sev0?]
On Oct 26, 2006, at 1:31 AM, [EMAIL PROTECTED] wrote: It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. Yeah - I know it hard-to-impossible to do that, and it is a tug-of-war between worm writers (to generate queries indistinguishable from real client-resolver-generated queries) and trying-to-detect-malformed- queries (such as duplicated qid, or from IP space that shouldn't be hitting this specific node). You probably dealt with more ddos than rest of us combined, so I bow to your superior knowledge. First, thanx for the nod, but there are some here who have dealt with more than I have. But I think I've seen enough to know something about it. You can try things like filter IP addresses which should not be going to node X, but what happens if the DDoS changes the network topology enough that you can't be certain users are going where you did not? If the DDoS is large, this is pretty much guaranteed. Worse, suppose the topology changes for reasons unrelated to a DDoS. You could end up DoS'ing end users without an attack! (You could theoretically only put the filters in place when an attack is happening, but that has other problems - which may or may not be worse.) Filtering on things like duplicated query IDs is not possible on router hardware doing 10s of Gbps or millions of PPS. And doing it on the server is not useful if there are more bits / pps than the router can process. Remember, servers can't answer packets that are dropped before they get to the servers. Etc., etc., etc. Overall, we are losing the war. What good providers, like the roots, Ultra, etc., do is to minimize the effect of any attack. If a miscreant fires the DDoS of biblical proportions and only 5% of users are affected, I consider that a success. Unfortunately, those 5% don't think so, but one can only do what one can do. Besides, if it truly is an attack of biblical proportion, those 5% are probably having much larger problems than name resolution. Couple other comments: From all indications I've seen (and most are not authoritative, but it's all the info I have), this was not a DDoS of biblical proportions. There were no whole networks to go offline, there were no massive swaths of address space flapping, there were no entire peering points being congested, etc. A few Gbps does not count as biblical any more. Whether this attack used spoof-source or not, BCP38 is _VITAL_, IMHO, to helping curb these things. It guarantees, at the very least, that you know where the attack is sourced. Filtering become much easier. Reaching the right operators to help with the problem becomes orders of magnitude easier. And if the miscreants just start using BotNets with real IP address, GOOD. It's not the End All Be All answer, but it is a _huge_ step in the right direction. Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the ip bcp38 config option. :) -- TTFN, patrick
Re: register.com down sev0?
I don't think I implied anything of the sort. ahhh, but you did. I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? if the register.com botnet attack was not from spoofed addresses, then bcp 38 would not have helped. the case for which we know bcp 38 is useful, is the dns reflector attack. so far, botnets seem to have no need to spoof, they just overwhelm you with zombies from real space. randy
10,352 active botnets (was Re: register.com down sev0?
On Thu, 26 Oct 2006 05:11:14 -, Fergie said: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? The same people I mentioned the other day as not having enough clue to do DNS correctly don't have enough clue to do BCP38 correctly either. As one person mentioned, if stuff still requires pioneer-level skillsets to use, the pioneers have more work to do. The problem is that the following wave seems to be made up mostly of chimpanzees, and nobody's figured out how to make routers and network services that can be run by chimps... Maybe the new slogan needs to be Save the Internet! Train the chimps! pgpFsZMkxDfPo.pgp Description: PGP signature
Re: register.com down sev0?
On Thu, 26 Oct 2006, Randy Bush wrote: I don't think I implied anything of the sort. ahhh, but you did. I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? if the register.com botnet attack was not from spoofed addresses, then bcp 38 would not have helped. the case for which we know bcp 38 is useful, is the dns reflector attack. so far, botnets seem to have no need to spoof, they just overwhelm you with zombies from real space. And yet they do anyway. Before the reflector attacks run at the beginning of this year, you stated you do not see the need to deal with spoofing, as it is not something being exploited. It is being exploited, let's deal with it. Gadi. randy
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Oct 26, 2006, at 9:33 AM, Steven M. Bellovin wrote: Put another way, anti-spoofing does three things: it makes reflector attacks harder, it makes it easier to use ACLs to block sources, and it helps people track down the bot and notify the admin. Are people actually successfully doing either of the latter two? I'd be surprised if there were much of either. That leaves reflector attacks. Are those that large a portion of the attacks people are seeing? I disagree. As someone who has been attacked by spoof-source packets, and not-spoof-source packed, I can say, from personal experience, that the former is much, much easier to mitigate. And, as I posted before, even if all universal adoption of BCP38 means is that DDoS attacks move to botnets with 100% real source IP addresses, that would still be a Very Good Thing, IMHO. But perhaps others feel differently. Or perhaps they just haven't been attacked enough. :) -- TTFN, patrick
Re: register.com down sev0?
On Oct 26, 2006, at 11:24 AM, Randy Bush wrote: the case for which we know bcp 38 is useful, is the dns reflector attack. so far, botnets seem to have no need to spoof, they just overwhelm you with zombies from real space. Incorrect. While that is one mode of attack from a botnet, it is not the only mode. And there are reasons for even botnets to spoof source addresses. And reasons that the attack-ee would prefer they did not. Randy, are you REALLY arguing -against- BCP38? Or just yanking Fergie's chain 'cause it wouldn't have helped in this particular instance? -- TTFN, patrick
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
Put another way, anti-spoofing does three things: it makes reflector attacks harder, it makes it easier to use ACLs to block sources, and it helps people track down the bot and notify the admin. Are people actually successfully doing either of the latter two? I think it's a time constraint- looking up, sorting and notifying admins about 10,000 attack sources isn't practical. I'd love to do it- but I don't have time. That said- if someone notifies me of a compromised host I immediately investigate- and I suspect so would everyone else on this list. Has anyone put together a centralized system where you can send in a list of attacking bots, let it automatically sort by allocation, and then let it notify the appropriate admin with a list of [potentially] compromised hosts? Then again: Considering how many admins don't care, how many end users don't care/know, and how quickly many of thee systems would get re-infected maybe it's all a bit pointless. I'd be surprised if there were much of either. That leaves reflector attacks. Are those that large a portion of the attacks people are seeing? Everything I have seen of late has been legitimate traffic originating from across the globe. With tens of thousands of compromised hosts that's all it takes. -Don
Re: DNS DDoS [was: register.com down sev0?]
At 11:21 AM 10/26/2006, you wrote: Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the ip bcp38 config option. :) Even better would be the option: no ip bcp38 Make it so a conscious action is needed to disable it, but PLEASE put that in the release notes so when the config doesn't change we know that something really did change... :) R Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: register.com down sev0? - More information
5. ATT (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter :) Lest anyone take me too seriously on that last point- ATT hosting does have community strings for certain features- unfortunately not for null routing. -Don (My apologies for the earlier lack of a full email name)
Re: register.com down sev0?
the case for which we know bcp 38 is useful, is the dns reflector attack. so far, botnets seem to have no need to spoof, they just overwhelm you with zombies from real space. Incorrect. While that is one mode of attack from a botnet, it is not the only mode. And there are reasons for even botnets to spoof source addresses. And reasons that the attack-ee would prefer they did not. Randy, are you REALLY arguing -against- BCP38? Or just yanking Fergie's chain 'cause it wouldn't have helped in this particular instance? i merely said that using this particular attack to launch yet another bcp38 religious dos against the nanog list was bogus. have we learned one new thing from the last day's oratory? personally, i long ago implemented spoofing blocking in all places i have been able to do so. but i am not foolish enough to believe that religious ranting on mailing lists is gonna change anyone from doing what makes business sense for their network. and, as spoofed attacks other than the dns reflector seem to have been rare, that perceived interest in anti-spoofing blocks is low when compared to other priorities in these hard times. i think we have converted those who were convertable and the rest watch the religious zealotry and scratch their heads. randy
Re: DNS DDoS [was: register.com down sev0?]
The network hardware vendors do need to include the feature to support BCP-38. It'll help us out on a number of fronts especially with some of the recent cyber attacks. We're in process of reaching out to many of the companies and many providers to encourage the implementation of BCP-38. We've gotten a lot of great feedback from many of you and its greatly appreciated. You know who you are :) Especially some of the feedback related to the hardware OS issues. -Jerry [EMAIL PROTECTED] or [EMAIL PROTECTED] Sent via BlackBerry from Cingular Wireless -Original Message- From: Robert Boyle [EMAIL PROTECTED] Date: Thu, 26 Oct 2006 12:04:03 To:Patrick W. Gilmore [EMAIL PROTECTED], nanog@merit.edu Subject: Re: DNS DDoS [was: register.com down sev0?] At 11:21 AM 10/26/2006, you wrote: Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the ip bcp38 config option. :) Even better would be the option: no ip bcp38 Make it so a conscious action is needed to disable it, but PLEASE put that in the release notes so when the config doesn't change we know that something really did change... :) R Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Don wrote: Has anyone put together a centralized system where you can send in a list of attacking bots, let it automatically sort by allocation, and then let it notify the appropriate admin with a list of [potentially] compromised hosts? mynetwatchman [1] comes to mind and so does dshield [2] [1] http://www.mynetwatchman.com [2] http://www.dshield.org -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: register.com down sev0?
At 07:25 AM 10/26/2006, Jared Mauch wrote: On Thu, Oct 26, 2006 at 06:03:54AM +, Fergie wrote: Randy, I don't think I implied anything of the sort. I did, however, pipe up when a BCP is mentioned that I endorse, and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. The challenge is that the router vendors still haven't done The Right Thing. I have one device that 1) halves its forwarding table space by enabling u-rpf 2) can only do either strict or loose mode rpf *GLOBALLY* so I can not strict rpf-check a static customer AND loose rpf someone larger for unrouted space. It was possible to implement BCP38 before the router vendors came up with uRPF. because of the above (#1 isn't that bad, but #2 is) I can't enable u-rpf on the device as a policy. Changing one interface from loose - strict silently changes all other u-rpf interfaces and then customers gripe about dropped packets. obviously moving these checks closer to the edge is ideal, such as always doing rpf on the ethernet lan interface for your customer CPE. Yes, it is. And does not require uRPF. I know you're looking to do the right thing. It's important though that this not be put entirely on the router vendors. How many managed T1 services out there have routers controlled by the ISP providing them? How many of those routers are configured with a single line ACL that would implement BCP38 sufficiently? How many aggregation routers for incoming T1s are not configured with a single line ACL per T-1 to ensure the packets coming in are from assigned, not-multihomed space? If scripts are being used to auto-configure routers to ship out to T-1 customers, then appropriate ACLs should be written by such scripts at the same time. Scripts that configure aggregation switches should similarly be reviewed for ACL inclusion. It's certainly helpful to have implementations such as uRPF to help make it easier to deploy BCP38, but deployment of BCP38 is not dependent on the existence of uRPF. Having said that, botnets don't need to spoof addresses -- the sheer dispersion of geographic and AS infection base renders the whole point of spoofing almost moot. yup, it's an evolving threat, even if some solution to the botnet problem is discovered, it will take years to fix. Think of the smurf amplifiers that are still out there[1]. Dan (the other co-author of the BCP in question)
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
- Original Message - From: william(at)elan.net [EMAIL PROTECTED] To: Don [EMAIL PROTECTED] Cc: nanog@merit.edu Sent: Thursday, October 26, 2006 8:17 AM Subject: Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?) On Thu, 26 Oct 2006, Don wrote: Has anyone put together a centralized system where you can send in a list of attacking bots, let it automatically sort by allocation, and then let it notify the appropriate admin with a list of [potentially] compromised hosts? mynetwatchman [1] comes to mind and so does dshield [2] [1] http://www.mynetwatchman.com [2] http://www.dshield.org -- William Leibzon Elan Networks [EMAIL PROTECTED] Anyone familiar with these folks? http://www.simplicita.com/Simplicita_Research_Data_Partner_Program.html --Michael
Re: 10,352 active botnets (was Re: register.com down sev0?
Maybe the new slogan needs to be Save the Internet! Train the chimps! Shouldnt 'ip verify unicast source reachable-by rx' be a default setting on all interfaces? Only to be removed by trained chimps? -Matt -- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
Re: register.com down sev0?
On Wed, 25 Oct 2006, Randy Bush wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed source attacks' more often than I'd think is reasonable. I've not got 'hard numbers' but almost every time the attack is determined to be 'botnet' it's not spoofed. Odd... (not that I'm against bcp38, I just think the distraction in conversation from 'bcp38 is good' to 'we must stop bots' is not helpful)
Re: register.com down sev0?
On Thu, 26 Oct 2006, Fergie wrote: and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. you could google answers from other folks but in shor: 1) it doesn't always work as advertised 2) people don't always tell you the routes the hold 3) equipment vendors don't alway splan properly for 'features' Not everyone is as smart as you (both) and can manage that problem as they scale...
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Mikael Abrahamsson wrote: On Thu, 26 Oct 2006, Fergie wrote: The point I'm trying to make is that if the community thinks it is valuable, then the path is clear. I of course realise that it's best if user cannot spoof at all, but it might be easier for ISPs to filter based on their PA blocks than to (in do your customers: 1) not bring their own ip space? 2) always advertise to you their ip space?
Re: register.com down sev0?
what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed source attacks' more often than I'd think is reasonable. I've not got 'hard numbers' but almost every time the attack is determined to be 'botnet' it's not spoofed. Odd... (not that I'm against bcp38, I just think the distraction in conversation from 'bcp38 is good' to 'we must stop bots' is not helpful) bingo! when you have religion about a hammer, everything looks like a nail. randy
Re: register.com down sev0?
Chris, W.R.T. #2 below: Be for real: No one ever suggested that backbone service providers attempt to ingress filter traffic -- this is an edge function. Cheers, - ferg -- Chris L. Morrow [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. you could google answers from other folks but in shor: 1) it doesn't always work as advertised 2) people don't always tell you the routes the hold 3) equipment vendors don't alway splan properly for 'features' Not everyone is as smart as you (both) and can manage that problem as they scale... -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: register.com down sev0?
We all have our opinions, Randy. Hammers and nails being what they are... - ferg -- Randy Bush [EMAIL PROTECTED] wrote: what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed source attacks' more often than I'd think is reasonable. I've not got 'hard numbers' but almost every time the attack is determined to be 'botnet' it's not spoofed. Odd... (not that I'm against bcp38, I just think the distraction in conversation from 'bcp38 is good' to 'we must stop bots' is not helpful) bingo! when you have religion about a hammer, everything looks like a nail. randy -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: 10,352 active botnets (was Re: register.com down sev0?
Matthew Crocker wrote: Maybe the new slogan needs to be Save the Internet! Train the chimps! Shouldnt 'ip verify unicast source reachable-by rx' be a default setting on all interfaces? Only to be removed by trained chimps? Only if you wish to break existing configurations during IOS upgrades. I could see ip verify unicast source reachable-by any (less breakage), but rx will kill all types of good asymmetric routing. The largest breakage I have seen caused by rx is the link IP breakage caused by the router responding out multiple interfaces. It's also a problem when customers are straddling the fence, purposefully using asymmetric routing. It would be nicer to have router support where a packet is acceptable if it's network is acceptable in the BGP (or IGP) policy/filter (ie, network may not be there, but it is allowed) as well as the link addresses associated with the BGP (or IGP) peer. -Jack
[Fwd: Re: DNS DDoS [was: register.com down sev0?]]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 We ran into similar attacks (couple days back) coming from non-spoofed address range (being initiated from valid prefixes). In working (w/ a co-worker of mine) on a network attack situation (trace process) for a 30,000 user location (serving 60 other school districts) running BCP38 rate-limit which got ddos'd w/ about 8mpps. It appears that these attacks were coming from the inside which not only saturated devices along its way but also got amplified into several other networks also causing significant flaps to its peered connection (OC-xx). Besides being distracted with this incredible among of traffic flow our goal number one goal was to prevent this bleeding, thanks to the distributed monitoring sensors (maybe we got lucky) we were able to identify and sink-hole (null route) certain blocks (vlans) while we worked with the network/desktop team to isolate the infected machines. This was certainly a hair-pulling experience. The point that I'm trying to make here is, you can have data coming from a herd of comprised hosts (bots, self-propagating worms, spam-relays,fake http get request, backdoors, etc) that can attack against a well-protected system(s) so any kind of defense mechanism can/will get defeated. Then again, it doesn't mean one wouldn't want to follow well practiced prevention methods. Just curious, any ddos vendors want to share their success stories :-) regards, /virendra - Original Message Subject: Re: DNS DDoS [was: register.com down sev0?] Date: Thu, 26 Oct 2006 17:32:56 + From: [EMAIL PROTECTED] Reply-To: [EMAIL PROTECTED] To: Robert Boyle [EMAIL PROTECTED], [EMAIL PROTECTED],Patrick W. Gilmore [EMAIL PROTECTED], Nanog nanog@merit.edu References: [EMAIL PROTECTED][EMAIL PROTECTED] [EMAIL PROTECTED] The network hardware vendors do need to include the feature to support BCP-38. It'll help us out on a number of fronts especially with some of the recent cyber attacks. We're in process of reaching out to many of the companies and many providers to encourage the implementation of BCP-38. We've gotten a lot of great feedback from many of you and its greatly appreciated. You know who you are :) Especially some of the feedback related to the hardware OS issues. - -Jerry [EMAIL PROTECTED] or [EMAIL PROTECTED] Sent via BlackBerry from Cingular Wireless - -Original Message- From: Robert Boyle [EMAIL PROTECTED] Date: Thu, 26 Oct 2006 12:04:03 To:Patrick W. Gilmore [EMAIL PROTECTED], nanog@merit.edu Subject: Re: DNS DDoS [was: register.com down sev0?] At 11:21 AM 10/26/2006, you wrote: Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the ip bcp38 config option. :) Even better would be the option: no ip bcp38 Make it so a conscious action is needed to disable it, but PLEASE put that in the release notes so when the config doesn't change we know that something really did change... :) R Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFQS8zpbZvCIJx1bcRAn93AKCSF2JcGTbB/bX/NcxxWdOwBXRDagCbBkY4 OBRqFdIvWojOwTK+K6Mlp2U= =LumS -END PGP SIGNATURE-
Re: register.com down sev0?
At 05:26 PM 10/26/2006, Fergie wrote: Chris, W.R.T. #2 below: Be for real: No one ever suggested that backbone service providers attempt to ingress filter traffic -- this is an edge function. I guess I'd add some clarification, though it should be obvious without. Backbone service providers who also sell edge circuits (e.g. dedicated T-1's to non-multihomed customers) ARE providing the edge function. A provider who claims we're a backbone, so we should do no ingress filtering at all is being disingenuous, at least for many of the largest networks today. I'm not accusing anyone of actually making such statements at all. I agree with Paul that this is an edge function, but that edge is a part of nearly every provider at some point in their businesses. -- Chris L. Morrow [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. you could google answers from other folks but in shor: 1) it doesn't always work as advertised 2) people don't always tell you the routes the hold 3) equipment vendors don't alway splan properly for 'features' Not everyone is as smart as you (both) and can manage that problem as they scale... -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: register.com down sev0?
On Thu, 26 Oct 2006, Fergie wrote: Chris, W.R.T. #2 below: Be for real: No one ever suggested that backbone service providers attempt to ingress filter traffic -- this is an edge function. ah, cause I thought 'everyone should do bcp38' mean 'everyone'... I agree that it's a great thing, I think 'everyone' should do it, I even thing we should where possible. I think LOTS of this would go away if people filtered their lan segments... tey have the horsey's there to do it without the compromises that must be taken at the 'core' (or 'more central portions of 'the net') And I was sorta yanking your chain some :) Cheers, - ferg -- Chris L. Morrow [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: and co-authored -- and likewise, cannot figure out for life of me, why there is such push-back from the Ops community on doing The Right Thing. you could google answers from other folks but in shor: 1) it doesn't always work as advertised 2) people don't always tell you the routes the hold 3) equipment vendors don't alway splan properly for 'features' Not everyone is as smart as you (both) and can manage that problem as they scale... -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: register.com down sev0? - More information
5. ATT (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter :) Think about that for a second. To stop an attack Register.com would need to call ATT and request a filter/null route. Since ATT operations is based in Singapore (again this was last time I dealt with them) I'm sure getting those filters/routes in probably doesn't happen nearly fast enough. I have heard that ATT is currently in the process of setting up communities- maybe someone who knows more could comment. Well, this is not exactly true.ATT does support BGP communities, although their communities aren't all that powerful, IMO. To my knowledge, you are correct when you say that they do not support any null-routing capabilities. I would love to find out the procedure and string required to request/implement null routing via a community. For those who would like to see ATT's official guide, it can be found at: http://www.onesc.net/communities/as7018 charles
Re: 10,352 active botnets (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Gadi Evron wrote: Jose may be a bit conservative with numbers, but he has good data and shares it, which is more than I can say for some people. http://www.asu.edu/security/aware/2005/lippard.htm
Re: register.com down sev0?
On Thu, 26 Oct 2006, Chris L. Morrow wrote: On Wed, 25 Oct 2006, Randy Bush wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. oh? you have knowledge that this botnet attack used spoofed source addresses? what's curious, to me atleat, is that folks equate 'botnet' and 'spoofed source attacks' more often than I'd think is reasonable. I've not got 'hard numbers' but almost every time the attack is determined to be 'botnet' it's not spoofed. Odd... (not that I'm against bcp38, I just think the distraction in conversation from 'bcp38 is good' to 'we must stop bots' is not helpful) SAT time. Almost all spoofed attacks are run by botnets. Almost all attacks are run by botnets Almost all spoofed attacked are bigger by a large factor Almost all botnet attacks are spoofed attacks? Not quite. That's about it.
Re: [Fwd: Re: DNS DDoS [was: register.com down sev0?]]
On Thu, 26 Oct 2006, virendra rode // wrote: Just curious, any ddos vendors want to share their success stories :-) If you access Cisco as a customer: http://www.cisco.com/en/US/customer/products/ps5887/products_case_study0900aecd80120478.shtml Rackspace Managed Hosting - Customer Success Story -Hank Nussbacher http://www.interall.co.il
register.com down sev0?
I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up. Would be interesting to see the RFO for that one, including the why we didn't have any DNS servers offsite or used anycast to at least limit amount of damage. -alex
Re: register.com down sev0?
On Wed, 25 Oct 2006 [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up. Would be interesting to see the RFO for that one, including the why we didn't have any DNS servers offsite or used anycast to at least limit amount of damage. just guessing but: 1) it's 'hard' 2) there is very little 'sla' on anything register.com does ? (no idea, not a customer) 3) it's 'hard' 4) why? this is the 100yr flood scenario for them? (perhaps) cost/benefit ... analyze :) It's possible that that 'analyze' part may change if there is significant fall-out from this event though.
Re: register.com down sev0?
On Wednesday 25 Oct 2006 15:59, you wrote: just guessing but: 1) it's 'hard' rant The reason the public facing DNS is poorly set up at the majority of institutions is the IT guy says lets bring it in house to give us more control, how hard can it be?. When if they had left it with their ISP it would be done right (along with the thousands of others that the ISP does right). I've seen it done dozens of times when consulting. I have data from a personal survey that confirms this is the leading cause of poor DNS configuration and lack of redundancy in my part of the UK. I even have a few domains we slave to servers across several continents, and otherwise clueful IT people pick SOA settings that still cause their domains to expire too quickly when, had they left it to us, it would just work. (okay I could override those settings, but if I do that why bother letting them master it in the first place?! we delegated control to you, and then overrode all your settings because they were stupid?!). So don't let the IT guy be a hidden master either, just leave it to the ISP. How I reach the zillions of IT guys out there to say don't do DNS inhouse, you'll only mess up is the remaining question; slashdot? /rant
Re: register.com down sev0?
On Wed, 25 Oct 2006, Simon Waters wrote: On Wednesday 25 Oct 2006 15:59, you wrote: just guessing but: 1) it's 'hard' rant How I reach the zillions of IT guys out there to say don't do DNS inhouse, you'll only mess up is the remaining question; slashdot? /rant wanna present all this rant and the proper solution to rant at the next nanog? :)
Re: register.com down sev0?
Chris L. Morrow wrote: On Wed, 25 Oct 2006, Simon Waters wrote: On Wednesday 25 Oct 2006 15:59, you wrote: just guessing but: 1) it's 'hard' rant How I reach the zillions of IT guys out there to say don't do DNS inhouse, you'll only mess up is the remaining question; slashdot? /rant wanna present all this rant and the proper solution to rant at the next nanog? :) Perhaps we should be celebrating the upcoming 10th anniversary of bcp 17. -- Joel Jaeggli Unix Consulting [EMAIL PROTECTED] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2
Re: register.com down sev0?
On Wed, 25 Oct 2006 18:59:05 -, Chris L. Morrow said: On Wed, 25 Oct 2006, Simon Waters wrote: How I reach the zillions of IT guys out there to say don't do DNS inhouse, you'll only mess up is the remaining question; slashdot? /rant wanna present all this rant and the proper solution to rant at the next nanog? :) Preaching to the choir, I suspect. Or are the people who are known offenders likely to actually be reached by the presentation? This is a common thread I keep encountering - the sites with enough clue to send somebody to NANOG aren't usually the sites I need to send a cluegram to. If anybody has a viable suggestion for that... pgpddPaHDBgd3.pgp Description: PGP signature
Re: register.com down sev0?
On Wed, 25 Oct 2006, [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up. I'll take your word on exhaustively checking every possible address. BTW, do you mean nameservers down, webservers down, or something else? Did the Internet break? Would be interesting to see the RFO for that one, including the why we didn't have any DNS servers offsite They colo in more than a half-dozen facilities around the world. or used anycast to at least limit amount of damage. I also have information from a pretty good source that they actually do quite a bit of anycast. matto [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Re: register.com down sev0?
On Wed, 25 Oct 2006, [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. They are apparently under a multi-gbps ddos of biblical proportions. [EMAIL PROTECTED]darwin Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Re: register.com down sev0?
On Wed, 25 Oct 2006, Matt Ghali wrote: On Wed, 25 Oct 2006, [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up. I'll take your word on exhaustively checking every possible address. BTW, do you mean nameservers down, webservers down, or something else? Did the Internet break? *.register.com means nameservers, webservers, whois servers, etc. Of course, Internet does not break, but we've received quite a number of calls about internet is down - given that register.com serves a large number of domains, yes, this is operationally affecting. Would be interesting to see the RFO for that one, including the why we didn't have any DNS servers offsite They colo in more than a half-dozen facilities around the world. or used anycast to at least limit amount of damage. I also have information from a pretty good source that they actually do quite a bit of anycast. Not that I can see - possibly that depends on a specific domain's webservers. The glue servers for register.com themselves: Name: ns1.register.com Address: 216.21.234.96 Name: ns2.register.com Address: 216.21.226.96 Name: ns3.register.com Address: 216.21.234.97 Name: ns4.register.com Address: 216.21.226.97 (note just two different /24s) Both of those /24s were down/down about 30 minutes ago, and are flapping/flapping now. route-views.oregon-ix.netshow ip bgp 216.21.234.73 ... BGP routing table entry for 216.21.234.0/24, version 5214460 701 7018 4264 13910, (suppressed due to dampening) 157.130.10.233 from 157.130.10.233 (137.39.3.60) Origin IGP, localpref 100, valid, external Dampinfo: penalty 898, flapped 5 times in 00:35:15, reuse in 00:03:50 route-views.oregon-ix.netshow ip bgp 216.21.226.97 BGP routing table entry for 216.21.226.0/24, version 5214460 ... 701 7018 4264 13910, (suppressed due to dampening) 157.130.10.233 (inaccessible) from 157.130.10.233 (137.39.3.60) Origin IGP, localpref 100, valid, external Dampinfo: penalty 861, flapped 5 times in 00:36:13, reuse in 00:03:00 From various vantage points, both /24s are routed exactly the same (7018 in NYC). -alex
Re: register.com down sev0?
They are apparently under a multi-gbps ddos of biblical proportions. locusts? racks which face backwards being turned into pillars of salt? death of the primaries? floods? sorry. been a hard day. randy
Re: register.com down sev0?
On Wed, 25 Oct 2006, Matt Ghali wrote: On Wed, 25 Oct 2006, [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. They are apparently under a multi-gbps ddos of biblical proportions. As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) Compliance of icann-accredited gtld-registrars with rfc2182 might be a good subject for research (again, thanks to rs for idea) -alex
Re: register.com down sev0?
I'm seeing *.register.com down (including ns*) from everywhere. They are apparently under a multi-gbps ddos of biblical proportions. i wonder if that's due to the spam they've been sending out? As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - no. really, not. such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. there is no zone anywhere, including COM, the root zone, or any other, that is immune from worst-case DDoS. anycast all you want. diversify. build a name service infrastructure larger than the earth's moon. none of that will matter as long as OPNs (the scourge of internet robustness) still exist. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) that's an easy but catty criticism, and baseless. i'm sure that some way could be found to improve register.com's infrastructure, and i don't just mean by stopping the spamming they've been doing. but it's not trivial and in the face of well-tuned worst-case DDoS, nothing will help. Compliance of icann-accredited gtld-registrars with rfc2182 might be a good subject for research (again, thanks to rs for idea) i've been wondering if ICANN's accredidation could be revoked for spammers, and register.com has indeed been spamming. and it may also be that they are out of compliance with RFC 2182. but that would be like catching al capone for income tax evasion just because you couldn't pin murder on him. (OPNs = Other People's Networks) -- Paul Vixie
Re: register.com down sev0?
On 26 Oct 2006, Paul Vixie wrote: I'm seeing *.register.com down (including ns*) from everywhere. They are apparently under a multi-gbps ddos of biblical proportions. i wonder if that's due to the spam they've been sending out? Paul, this isn't nanae. Let's not sling accusations like that wildly. As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - no. really, not. such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. there is no zone anywhere, including COM, the root zone, or any other, that is immune from worst-case DDoS. anycast all you want. diversify. build a name service infrastructure larger than the earth's moon. none of that will matter as long as OPNs (the scourge of internet robustness) still exist. This isn't 2001, and, I will argue that it *is*, in fact, possible to be protected from a worst case ddos, and not at obscene price. However, even if you argue that point, there's no excuse for not being prepared at all, and not following the BCP. While we all may be guilty of not having topologically/geographically diverse DNS - for someone whose core business is DNS, that's unexcusable. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) that's an easy but catty criticism, and baseless. i'm sure that some way could be found to improve register.com's infrastructure, and i don't just mean by stopping the spamming they've been doing. but it's not trivial and in the face of well-tuned worst-case DDoS, nothing will help. Well, let's talk about worst-case ddos. Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company. I know that the above was just rough back-of-the-envelope, and things are far more complicated than that, but this discussion does not really belong to nanog-l. Compliance of icann-accredited gtld-registrars with rfc2182 might be a good subject for research (again, thanks to rs for idea) i've been wondering if ICANN's accredidation could be revoked for spammers, and register.com has indeed been spamming. and it may also be that they are out of compliance with RFC 2182. but that would be like catching al capone for income tax evasion just because you couldn't pin murder on him. Things like that, and accusations like that, I don't think really belong to nanog-l. (speaking for myself only)
Re: register.com down sev0?
On Wed, 2006-10-25 at 18:41 -0700, Matt Ghali wrote: On Wed, 25 Oct 2006, [EMAIL PROTECTED] wrote: I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up. I'll take your word on exhaustively checking every possible address. BTW, do you mean nameservers down, webservers down, or something else? Did the Internet break? Would be interesting to see the RFO for that one, including the why we didn't have any DNS servers offsite They colo in more than a half-dozen facilities around the world. or used anycast to at least limit amount of damage. I also have information from a pretty good source that they actually do quite a bit of anycast. There are two sides to rcom, the mompop side (aka register.com) and the partner side (Rconnection, for folks with ~25+ domains registered). On the mompop side they don't have (as far as I am concerned) a highly redundant and distributed DNS system. That opinion is based on a few hours of research abt 2 years ago. Over on the partner side they outsource the DNS systems for their customers to eNom, which does use a highly redundant and distributed anycast setup. I haven't seen any problems wrt DNS for my systems today (eNom via rcom), so I can only presume the OP was referring to the mompop side of rcom. -Jim P.
Re: register.com down sev0?
On Oct 25, 2006, at 11:14 PM, [EMAIL PROTECTED] wrote:On 26 Oct 2006, Paul Vixie wrote:I'm seeing *.register.com down (including ns*) from everywhere.They are apparently under a multi-gbps ddos of "biblicalproportions".i wonder if that's due to the spam they've been sending out?Paul, this isn't nanae. Let's not sling accusations like that wildly. Good god. It isn't like they are some borderline case or anything.Chris PGP.sig Description: This is a digitally signed message part
Re: register.com down sev0?
On Oct 26, 2006, at 12:14 AM, [EMAIL PROTECTED] wrote: On 26 Oct 2006, Paul Vixie wrote: i wonder if that's due to the spam they've been sending out? Paul, this isn't nanae. Let's not sling accusations like that wildly. Accusations and objective facts are two separate things. there is no zone anywhere, including COM, the root zone, or any other, that is immune from worst-case DDoS. anycast all you want. diversify. build a name service infrastructure larger than the earth's moon. none of that will matter as long as OPNs (the scourge of internet robustness) still exist. This isn't 2001, and, I will argue that it *is*, in fact, possible to be protected from a worst case ddos, and not at obscene price. You are mistaken. However, even if you argue that point, there's no excuse for not being prepared at all, and not following the BCP. While we all may be guilty of not having topologically/geographically diverse DNS - for someone whose core business is DNS, that's unexcusable. We agree. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) that's an easy but catty criticism, and baseless. i'm sure that some way could be found to improve register.com's infrastructure, and i don't just mean by stopping the spamming they've been doing. but it's not trivial and in the face of well-tuned worst-case DDoS, nothing will help. Well, let's talk about worst-case ddos. Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company. Even assuming your numbers, which I do not grant, you are still mistaken. There is no single appropriately[sic] place which can absorb 50Mpps. If you meant appropriately placed (as in topologically dispersed locations), a well crafted attack could still guarantee _at least_ a partial DoS from an end user PoV. It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. This is not speculation, this is fact. All a good provider can do, even with 1000s of server, is minimize the impact of any DoS. Oh, and putting 2K servers into the right places is not a trivial expense, even for a large company. Last time I checked, 10GE pipes were not handed out for free. And you can't just rack these things in mom-and-pop colo saying well, it has a GigE on the motherboard when the colo has an OC3 to the 'Net. The Cap- and Op-Ex involved in doing what you suggest properly is large enough to probably be prohibitively expensive for a company like register.com. I know that the above was just rough back-of-the-envelope, and things are far more complicated than that, but this discussion does not really belong to nanog-l. We disagree. Keeping large name servers running is _absolutely_ a network operations topic. Not only is the defense mostly network based (since the network is the most likely thing to break), network operators are the people who get the phone calls when DNS does break. -- TTFN, patrick
Re: register.com down sev0?
I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? - ferg -- Patrick W. Gilmore [EMAIL PROTECTED] wrote: [snip] There is no single appropriately[sic] place which can absorb 50Mpps. If you meant appropriately placed (as in topologically dispersed locations), a well crafted attack could still guarantee _at least_ a partial DoS from an end user PoV. It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. This is not speculation, this is fact. All a good provider can do, even with 1000s of server, is minimize the impact of any DoS. [snip] -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
10,352 active botnets (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, [EMAIL PROTECTED] wrote: Well, let's talk about worst-case ddos. Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company. Botnets were the topic at today's Info Security conference in New York City. http://www.infosecurityevent.com Coincidences? Or just as random as your iPod shuffle? Jose Nazario estimated that there were 10,352 botnets active on the Internet earlier this year. You will probably always be outnumbered on the public Internet.
BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
On Thu, 26 Oct 2006, Fergie wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? Do you have any data concerning the actual consistent deployment of BCP38++ in different parts of the world?
Re: register.com down sev0?
On Thu, 26 Oct 2006, Patrick W. Gilmore wrote: There is no single appropriately[sic] place which can absorb 50Mpps. If you meant appropriately placed (as in topologically dispersed locations), a well crafted attack could still guarantee _at least_ a partial DoS from an end user PoV. It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. Yeah - I know it hard-to-impossible to do that, and it is a tug-of-war between worm writers (to generate queries indistinguishable from real client-resolver-generated queries) and trying-to-detect-malformed-queries (such as duplicated qid, or from IP space that shouldn't be hitting this specific node). You probably dealt with more ddos than rest of us combined, so I bow to your superior knowledge. I know that the above was just rough back-of-the-envelope, and things are far more complicated than that, but this discussion does not really belong to nanog-l. We disagree. Keeping large name servers running is _absolutely_ a network operations topic. Not only is the defense mostly network based (since the network is the most likely thing to break), network operators are the people who get the phone calls when DNS does break. Sorry - I meant that discussion whether or not register.com is spamming isn't somewhat offtopic. Of course, DNS operations (and particularly dealing with biblical scale ddos) is very much on-topic. -alex
Re: 10,352 active botnets (was Re: register.com down sev0?)
Jose's numbers are conservative. Given some mathematical acrobatics, I'd suggest examining some of the (shocking) number sin Microsoft's Security Intelligence Report (Google it) -- these are reflective: Of the 4 million computers cleaned by the company's MSRT (malicious software removal tool), about 50 percent (2 million) contained at least one backdoor Trojan. While this is a high percentage, Microsoft notes that this is a decrease from the second half of 2005. During that period, the MSRT data showed that 68 percent of machines cleaned by the tool contained a backdoor Trojan. Ref: http://www.eweek.com/article2/0,1759,2036439,00.asp If you're wondering why DDoS attacks are so effective, look no further than your backyard. - ferg -- Sean Donelan [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, [EMAIL PROTECTED] wrote: Well, let's talk about worst-case ddos. Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company. Botnets were the topic at today's Info Security conference in New York City. http://www.infosecurityevent.com Coincidences? Or just as random as your iPod shuffle? Jose Nazario estimated that there were 10,352 botnets active on the Internet earlier this year. You will probably always be outnumbered on the public Internet. -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: BCP38 thread 93,871,738,435 (was Re: register.com down sev0?)
No. I think that is indicative of the problem. Don't you? - ferg -- Sean Donelan [EMAIL PROTECTED] wrote: On Thu, 26 Oct 2006, Fergie wrote: I don't want to detract from the heat of this discussion, as important as it is, but it (the discussion) illustrates a point that RIPE has recognized -- and is actively perusing -- yet, ISPs on this continent seem consistently to ignore: The consistent implementation of BCP 38. It is nothing less than irresponsible, IMO... Why _is_ that? Do you have any data concerning the actual consistent deployment of BCP38++ in different parts of the world? -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
