Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-22 Thread jamie rishaw
Data on June 20 : .COM. : 108,985,894 unique domains + the tld. - 234,479 NSEC3/RRSIG records, - 2,253,400 nameserver entries on 831,088 unique IP addresses. .. ish. -jamie On Fri, Jun 21, 2013 at 5:23 PM, Barry Shein b...@world.std.com wrote: I think we need a better measure than

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Jimmy Hess
On 6/20/13, Hal Murray hmur...@megapathdsl.net wrote: Perhaps we should setup a distributed system for checking things rather than another SPOF. That's distributed both geographically and administratively and using several code-bases. [snip] I would be in favor of being able to pay two

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Jimmy Hess
On 6/20/13, valdis.kletni...@vt.edu valdis.kletni...@vt.edu wrote: It's relatively small when you consider there's something like 140M .com's Yeah... I'm in agreement about that's probably what is going on... It's relatively small, but absolutely large, and absolute numbers matter. 5 domains

RE: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Kain, Rebecca (.)
Golodner Cc: nanog@nanog.org Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) At 17:12 20/06/2013 -0500, Richard Golodner wrote: I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Valdis . Kletnieks
On Thu, 20 Jun 2013 23:42:24 -0400, shawn wilson said: I think Netsol should be fined. Maybe even a class action suite filed against them for lost business. And that's it. So your contract with NetSol has an SLA guarantee in it, and you can demonstrate that (a) said SLA has been violated and

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Nicolai
On Thu, Jun 20, 2013 at 05:28:17PM -0400, valdis.kletni...@vt.edu wrote: It's relatively small when you consider there's something like 140M .com's Just FWIW, the current size of .com is roughly 109M domains. Someday it will reach 140M but not today. Nicolai

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread David Walker
https://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ Why are they infinitely looping a script on their web server to check for a cookie? Are these people insane?

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread John Levine
Registrar Primary and Registrar Auditor There are certainly registrars who are more security oriented than Netsol. If you haven't followed all of the corporate buying and selling, Netsol is now part of web.com, so their business is more to support web hosting than to be a registrar. I expect

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Barry Shein
I think we need a better measure than number of domains (in this case .COM), particularly vs total domains. If it was 100 domains it might seem small, unless that list began with facebook.com, amazon.com, google.com and g*d forbid theworld.com. -- -Barry Shein The World |

RE: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread John Souvestre
Operators Group Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) I think ICANN would have to add a delay in where a request was sent out to make sure everyone was on the same page and then what happens the couple thousand (more) times a day that someone isn't updated

RE: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread Frank Bulk
...@chocolatine.org] Sent: Friday, June 21, 2013 11:16 AM To: nanog@nanog.org Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) On Thu, Jun 20, 2013 at 05:28:17PM -0400, valdis.kletni...@vt.edu wrote: It's relatively small when you consider there's something like 140M .com's

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread John Levine
In article 001a01ce6ef9$bf74d4a0$3e5e7de0$@iname.com you write: It's 120M if you add the .COM and the .NET's together, both of which NetSol is responsible for. http://www.verisigninc.com/en_US/products-and-services/domain-name-services/ registry-products/tld-zone-access/index.xhtml In late

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-21 Thread George Herbert
I know how we got here, but perhaps we can take corporate parentage and how big .com is now to -discuss? What happened with the registry data that caused the outage and what can / should be done about it / to prevent it happening again still seem to me to be operational topics. George

This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread jamie rishaw
This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Jared Mauch
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread jamie rishaw
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of localhost.. Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch ja...@puck.nether.net wrote: It

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread George Herbert
Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes. I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info. Is there anything about ztomy.com that has been seen that's supicious as in they might be

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread jamie rishaw
It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. As far as the logic of the DNS, it is functioning as designed (What's up, Vix!) - There's another aspect of this that caused this situation. Any Alexa or similar people on this list (Goog

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Andrew Fried
Not so easy and straightforward to do. You'll find that a lot of the big names out there frequently tweak DNS, which will result in a non-stop stream of alerts. Andy Andrew Fried andrew.fr...@gmail.com On 6/20/13 3:57 PM, Jared Mauch wrote: It seems there may be a need for some sort of

Fwd: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread jamie rishaw
' to other nameservers) that show no updates in `whois` records. Curiouser and curiouser. Paul? -- Forwarded message -- From: jamie rishaw j...@arpa.com Date: Thu, Jun 20, 2013 at 3:21 PM Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) To: George Herbert

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Jimmy Hess
On 6/20/13, jamie rishaw j...@arpa.com wrote: It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. https://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ -- -JH

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Jeff Shultz
On 6/20/2013 1:46 PM, Jimmy Hess wrote: On 6/20/13, jamie rishaw j...@arpa.com wrote: It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s.

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Carsten Bormann
Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived

RE: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Gabor Tokaji
PM To: NANOG list Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Valdis . Kletnieks
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number.

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread RijilV
On 20 June 2013 14:28, valdis.kletni...@vt.edu wrote: On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Randy Bush
So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? luckily, none of the rest of us make mistakes

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Ryan - Lists
I don't think he was saying that at all. Just stating that from a pure numbers standpoint 50k/140mil is a small percentage. OTOH, I agree to your point - Network Solutions definitely downplayed this in their release. Curiously so. Sent from my iPhone On Jun 20, 2013, at 5:42 PM, RijilV

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Bryan Irvine
On Thu, Jun 20, 2013 at 2:49 PM, Randy Bush ra...@psg.com wrote: So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? luckily, none of the rest of us make mistakes Ages ago I responded on a Cisco list where the topic was biggest screwup you've

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Richard Golodner
On Thu, 2013-06-20 at 14:42 -0700, RijilV wrote: On 20 June 2013 14:28, valdis.kletni...@vt.edu wrote: On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Randy Bush
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread George Herbert
At the DNS Servers or service provider level, one can (and I often do) have redundant providers. At the registrar level? ... Not with our current infrastructure, as far as I know how. The Internet: Discovering new SPOF since 1969! George William Herbert Sent from my iPhone On Jun 20,

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Phil Fagan
at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread jamie rishaw
No. The ztomy nameservers appeared in this morning's master .COM zonefile as /authoritative/ for the number of domains I mentioned. It is a clear change from just a couple of days ago, when the listed nameservers were nowhere to be seen. I have solid data to back this up, straight from Verisign

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Fred Reimer
I, for one, would not be in favor of an authoritarian rule over DNS, or any other Internet system, to ensure that the state of [the] service[s] is running as it should. I suppose one could view such an authoritarian rule over (sub) systems to be a good thing, as in there is someone to complain to

Fwd: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Timothy Morizot
On Jun 20, 2013 5:31 PM, Randy Bush ra...@psg.com wrote: and dnssec did not save us. is there anything which could have? Hmmm. DNSSEC wouldn't have prevented an outage. But from everything I've seen reported, had the zones been signed, validating recursive resolvers (comcast, google, much of

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Jimmy Hess
On 6/20/13, Randy Bush ra...@psg.com wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything

Re: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Rubens Kuhl
On Thu, Jun 20, 2013 at 8:41 PM, Timothy Morizot tmori...@gmail.com wrote: On Jun 20, 2013 5:31 PM, Randy Bush ra...@psg.com wrote: and dnssec did not save us. is there anything which could have? Hmmm. DNSSEC wouldn't have prevented an outage. But from everything I've seen reported, had

Re: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Timothy Morizot
On Jun 20, 2013 7:30 PM, Rubens Kuhl rube...@gmail.com wrote: In this case of registrar compromise, DS record could have been changed alongside NS records, so DNSSEC would only have been a early warning, because uncoordinated DS change disrupts service. As soon as previous timeouts played out,

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Hank Nussbacher
At 07:28 21/06/2013 +0900, Randy Bush wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. They are too busy adding new revenue:

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Hank Nussbacher
At 17:12 20/06/2013 -0500, Richard Golodner wrote: I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner sarcasm and

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Hal Murray
at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread shawn wilson
I think ICANN would have to add a delay in where a request was sent out to make sure everyone was on the same page and then what happens the couple thousand (more) times a day that someone isn't updated or is misconfigured? I think Netsol should be fined. Maybe even a class action suite filed

Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)

2013-06-20 Thread Valdis . Kletnieks
On Thu, 20 Jun 2013 20:25:24 -0700, Hal Murray said: How would you check/verify that the communication path from the monitoring agency to the right people in your NOC was working correctly? Remember to consider the possible impact of a false-positive report over an unauthenticated channel.