Re: Need help in flushing DNS
The only apparent link is registration thru network solutions On Wed, Jun 19, 2013 at 10:49 PM, Alex Buie alex.b...@frozenfeline.netwrote: Anyone have news/explanation about what's happening/happened? On Wed, Jun 19, 2013 at 10:34 PM, Paul Ferguson fergdawgs...@gmail.com wrote: Sure enough: ; DiG 9.7.3 @localhost yelp.com A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 53267 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;yelp.com. IN A ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 ;; Query time: 143 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 07:33:13 2013 ;; MSG SIZE rcvd: 42 NetRange: 204.11.56.0 - 204.11.59.255 CIDR: 204.11.56.0/22 OriginAS: AS40034 NetName: CONFLUENCE-NETWORKS--TX3 NetHandle: NET-204-11-56-0-1 Parent: NET-204-0-0-0-0 NetType: Direct Allocation Comment: Hosted in Austin TX. Comment: Abuse : Comment: ab...@confluence-networks.com Comment: +1-917-386-6118 RegDate: 2012-09-24 Updated: 2012-09-24 Ref: http://whois.arin.net/rest/net/NET-204-11-56-0-1 OrgName: Confluence Networks Inc OrgId: CN Address: 3rd Floor, Omar Hodge Building, Wickhams Address: Cay I, P.O. Box 362 City: Road Town StateProv: Tortola PostalCode: VG1110 Country: VG RegDate: 2011-04-07 Updated: 2011-07-05 Ref: http://whois.arin.net/rest/org/CN OrgAbuseHandle: ABUSE3065-ARIN OrgAbuseName: Abuse Admin OrgAbusePhone: +1-917-386-6118 OrgAbuseEmail: ab...@confluence-networks.com OrgAbuseRef: http://whois.arin.net/rest/poc/ABUSE3065-ARIN OrgNOCHandle: NOCAD51-ARIN OrgNOCName: NOC Admin OrgNOCPhone: +1-415-462-7734 OrgNOCEmail: n...@confluence-networks.com OrgNOCRef: http://whois.arin.net/rest/poc/NOCAD51-ARIN OrgTechHandle: TECHA29-ARIN OrgTechName: Tech Admin OrgTechPhone: +1-415-358-0858 OrgTechEmail: ipad...@confluence-networks.com OrgTechRef: http://whois.arin.net/rest/poc/TECHA29-ARIN # # ARIN WHOIS data and services are subject to the Terms of Use # available at: https://www.arin.net/whois_tou.html # - ferg On Wed, Jun 19, 2013 at 10:30 PM, Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected On Wed, Jun 19, 2013 at 10:19 PM, John Levine jo...@iecc.com wrote: Reaching out to DNS operators around the globe. Linkedin.com has had some issues with DNS and would like DNS operators to flush their DNS. If you see www.linkedin.com resolving NS to ns1617.ztomy.com or ns2617.ztomy.com then please flush your DNS. Any other info please reach out to me off-list. While you're at it, www.usps.com, www.fidelity.com, and other well known sites have had DNS poisoning problems. When I restarted my cache, they look OK. -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Re: Need help in flushing DNS
Hanlon's razor? Misconfiguration. Perhaps not done in malice, but I have no idea where the poison leaked in, or why. :-) - ferg On Wed, Jun 19, 2013 at 10:49 PM, Alex Buie alex.b...@frozenfeline.net wrote: Anyone have news/explanation about what's happening/happened? On Wed, Jun 19, 2013 at 10:34 PM, Paul Ferguson fergdawgs...@gmail.comwrote: Sure enough: ; DiG 9.7.3 @localhost yelp.com A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 53267 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;yelp.com. IN A ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 ;; Query time: 143 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 07:33:13 2013 ;; MSG SIZE rcvd: 42 NetRange: 204.11.56.0 - 204.11.59.255 CIDR: 204.11.56.0/22 OriginAS: AS40034 NetName: CONFLUENCE-NETWORKS--TX3 NetHandle: NET-204-11-56-0-1 Parent: NET-204-0-0-0-0 NetType: Direct Allocation Comment: Hosted in Austin TX. Comment: Abuse : Comment: ab...@confluence-networks.com Comment: +1-917-386-6118 RegDate: 2012-09-24 Updated: 2012-09-24 Ref: http://whois.arin.net/rest/net/NET-204-11-56-0-1 OrgName: Confluence Networks Inc OrgId: CN Address: 3rd Floor, Omar Hodge Building, Wickhams Address: Cay I, P.O. Box 362 City: Road Town StateProv: Tortola PostalCode: VG1110 Country: VG RegDate: 2011-04-07 Updated: 2011-07-05 Ref: http://whois.arin.net/rest/org/CN OrgAbuseHandle: ABUSE3065-ARIN OrgAbuseName: Abuse Admin OrgAbusePhone: +1-917-386-6118 OrgAbuseEmail: ab...@confluence-networks.com OrgAbuseRef: http://whois.arin.net/rest/poc/ABUSE3065-ARIN OrgNOCHandle: NOCAD51-ARIN OrgNOCName: NOC Admin OrgNOCPhone: +1-415-462-7734 OrgNOCEmail: n...@confluence-networks.com OrgNOCRef: http://whois.arin.net/rest/poc/NOCAD51-ARIN OrgTechHandle: TECHA29-ARIN OrgTechName: Tech Admin OrgTechPhone: +1-415-358-0858 OrgTechEmail: ipad...@confluence-networks.com OrgTechRef: http://whois.arin.net/rest/poc/TECHA29-ARIN # # ARIN WHOIS data and services are subject to the Terms of Use # available at: https://www.arin.net/whois_tou.html # - ferg On Wed, Jun 19, 2013 at 10:30 PM, Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected On Wed, Jun 19, 2013 at 10:19 PM, John Levine jo...@iecc.com wrote: Reaching out to DNS operators around the globe. Linkedin.com has had some issues with DNS and would like DNS operators to flush their DNS. If you see www.linkedin.com resolving NS to ns1617.ztomy.com or ns2617.ztomy.com then please flush your DNS. Any other info please reach out to me off-list. While you're at it, www.usps.com, www.fidelity.com, and other well known sites have had DNS poisoning problems. When I restarted my cache, they look OK. -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Re: Need help in flushing DNS
On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH
Re: Need help in flushing DNS
On Jun 19, 2013, at 11:23 PM, Jimmy Hess mysi...@gmail.com wrote: On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; IIRC, Confluence Networks/ztomy pounce on expired domains to sell ads or somesuch. I seem to recall them grabbing the parent domain of name servers for ben.edu last year... Regards, -drc
Re: Need help in flushing DNS
.-- My secret spy satellite informs me that at 2013-06-19 10:34 PM Paul Ferguson wrote: ; DiG 9.7.3 @localhost yelp.com A SNIP ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 Interesting to see that traffic to this IP addresses is going through prolexic... I guess they're considering this as a DOS. andree@bofh:~/src$ traceroute 204.11.57.20 traceroute to 204.11.57.20 (204.11.57.20), 64 hops max, 52 byte packets 1 10.200.200.200 (10.200.200.200) 17.089 ms 13.144 ms 13.552 ms 2 67.215.89.1 (67.215.89.1) 20.963 ms 15.371 ms 17.026 ms 3 67.215.93.14 (67.215.93.14) 20.486 ms 14.458 ms 16.917 ms 4 ge-0-7-0-5.r06.snjsca04.us.bb.gin.ntt.net (128.241.219.145) 19.449 ms 19.375 ms 15.274 ms 5 ae-2.prolexic.snjsca04.us.bb.gin.ntt.net (128.241.219.242) 17.107 ms 23.272 ms 16.019 ms 6 209.200.184.34 (209.200.184.34) 14.878 ms 19.062 ms 15.776 ms 7 unknown.prolexic.com (72.52.30.126) 67.871 ms 64.376 ms 66.988 ms 8 domain.not.configured (204.11.57.20) 71.729 ms 65.830 ms 67.823 ms Reflection attacks are so yesterday... Cheers, Andree
Re: Need help in flushing DNS
I have no knowledge of any DDoS -related activity involving Yelp! and Prolexic. Even if there is one, the fact that their DNS records have been poisoned has not direct relationship to any current DDoS (there isn't one that I am aware of). - ferg On Thu, Jun 20, 2013 at 12:31 AM, Andree Toonk andree+na...@toonk.nl wrote: .-- My secret spy satellite informs me that at 2013-06-19 10:34 PM Paul Ferguson wrote: ; DiG 9.7.3 @localhost yelp.com A SNIP ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 Interesting to see that traffic to this IP addresses is going through prolexic... I guess they're considering this as a DOS. andree@bofh:~/src$ traceroute 204.11.57.20 traceroute to 204.11.57.20 (204.11.57.20), 64 hops max, 52 byte packets 1 10.200.200.200 (10.200.200.200) 17.089 ms 13.144 ms 13.552 ms 2 67.215.89.1 (67.215.89.1) 20.963 ms 15.371 ms 17.026 ms 3 67.215.93.14 (67.215.93.14) 20.486 ms 14.458 ms 16.917 ms 4 ge-0-7-0-5.r06.snjsca04.us.bb.gin.ntt.net (128.241.219.145) 19.449 ms 19.375 ms 15.274 ms 5 ae-2.prolexic.snjsca04.us.bb.gin.ntt.net (128.241.219.242) 17.107 ms 23.272 ms 16.019 ms 6 209.200.184.34 (209.200.184.34) 14.878 ms 19.062 ms 15.776 ms 7 unknown.prolexic.com (72.52.30.126) 67.871 ms 64.376 ms 66.988 ms 8 domain.not.configured (204.11.57.20) 71.729 ms 65.830 ms 67.823 ms Reflection attacks are so yesterday... Cheers, Andree -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Re: Need help in flushing DNS
.-- My secret spy satellite informs me that at 2013-06-20 12:31 AM Andree Toonk wrote: .-- My secret spy satellite informs me that at 2013-06-19 10:34 PM Paul Ferguson wrote: ; DiG 9.7.3 @localhost yelp.com A SNIP ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 Interesting to see that traffic to this IP addresses is going through prolexic... I guess they're considering this as a DOS. andree@bofh:~/src$ traceroute 204.11.57.20 traceroute to 204.11.57.20 (204.11.57.20), 64 hops max, 52 byte packets 1 10.200.200.200 (10.200.200.200) 17.089 ms 13.144 ms 13.552 ms 2 67.215.89.1 (67.215.89.1) 20.963 ms 15.371 ms 17.026 ms 3 67.215.93.14 (67.215.93.14) 20.486 ms 14.458 ms 16.917 ms 4 ge-0-7-0-5.r06.snjsca04.us.bb.gin.ntt.net (128.241.219.145) 19.449 ms 19.375 ms 15.274 ms 5 ae-2.prolexic.snjsca04.us.bb.gin.ntt.net (128.241.219.242) 17.107 ms 23.272 ms 16.019 ms 6 209.200.184.34 (209.200.184.34) 14.878 ms 19.062 ms 15.776 ms 7 unknown.prolexic.com (72.52.30.126) 67.871 ms 64.376 ms 66.988 ms 8 domain.not.configured (204.11.57.20) 71.729 ms 65.830 ms 67.823 ms Slight correction for the archives, the trace above was going to 204.11.57.20 (not 204.11.56.20) which is the IP of the NS server (ns1620.ztomy.com), which also goes through prolexic (see above) andree@bofh:~/src$ dig @a.gtld-servers.net www.craigslist.com ns ; DiG 9.8.3-P1 @a.gtld-servers.net www.craigslist.com ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 52520 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 2 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;www.craigslist.com.IN NS ;; AUTHORITY SECTION: craigslist.com. 172800 IN NS ns1620.ztomy.com. craigslist.com. 172800 IN NS ns2620.ztomy.com. ;; ADDITIONAL SECTION: ns1620.ztomy.com. 172800 IN A 204.11.56.20 ns2620.ztomy.com. 172800 IN A 204.11.57.20 ;; Query time: 120 msec ;; SERVER: 192.5.6.30#53(192.5.6.30) ;; WHEN: Thu Jun 20 00:50:49 2013 ;; MSG SIZE rcvd: 116 This is the trace to 204.11.56.20 also via prolexic andree@bofh:~/src$ sudo tcptraceroute 204.11.56.20 80 Tracing the path to 204.11.56.20 on TCP port 80 (http), 30 hops max 1 10.200.200.200 14.840 ms 21.474 ms 13.641 ms 2 67.215.89.1 19.265 ms 13.646 ms 14.769 ms 3 67.215.93.14 15.000 ms 15.161 ms 15.159 ms 4 ge-0-7-0-5.r06.snjsca04.us.bb.gin.ntt.net (128.241.219.145) 15.358 ms 14.852 ms 16.432 ms 5 ae-2.prolexic.snjsca04.us.bb.gin.ntt.net (128.241.219.242) 13.735 ms 16.149 ms 17.957 ms 6 204.11.56.20 [open] 15.447 ms 16.897 ms 15.821 ms Btw, one more interesting detail these used to be announced as one /23. As of this week that's two /24's currently 204.11.56.0/24 (june 17) and 204.11.57.0/24 (june 19) Andree
Re: Need help in flushing DNS
Hi, .-- My secret spy satellite informs me that at 2013-06-20 12:38 AM Paul Ferguson wrote: I have no knowledge of any DDoS -related activity involving Yelp! and Prolexic. Even if there is one, the fact that their DNS records have been poisoned has not direct relationship to any current DDoS (there isn't one that I am aware of). That's not what I was trying to say. The domains like yelp, linkedin, craigslist all incorrectly have (or had) NS record like: ns1620.ztomy.com. 172800 IN A 204.11.56.20 ns2620.ztomy.com. 172800 IN A 204.11.57.20 Traffic to these IP's is going through Prolexic (see previous mail). Thought that was interesting... Andree
Re: Need help in flushing DNS
I have domains that are *not* expired, which are being affected by this. Domains are hosted via Dynect, and are resolving into this 204.11.56.0/24 range across the globe. Dynect management portal was down until minutes ago as well. - Charles On Jun 20, 2013, at 12:45 AM, David Conrad d...@virtualized.org wrote: On Jun 19, 2013, at 11:23 PM, Jimmy Hess mysi...@gmail.com wrote: On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; IIRC, Confluence Networks/ztomy pounce on expired domains to sell ads or somesuch. I seem to recall them grabbing the parent domain of name servers for ben.edu last year... Regards, -drc
Re: net neutrality and peering wars continue
On Jun 19, 2013, at 7:21 PM, Benson Schliesser bens...@queuefull.net wrote: The sending peer (or their customer) has more control over cost. I'll assume that, by sending peer, you mean the content network. If so, I disagree. The content network has no control whatsoever over the location of the eyeball customer. The eyeball customer has sole control over his or her own location, while the content network has sole control over the location from which they reply to requests. Therefore, control is shared between the two sides. And both are incentivized to minimize costs. If both minimize their costs, overall costs are minimized. That's why this system works. -Bill
Re: Need help in flushing DNS
Smileyface aside, I'm disappointed to see operators simply flushing caches and not performing at the least a dumpdb for possible future forensic analysis. This is what I call the Windows solution, - 'Oh, just reboot, and it'll work'. We're better than that. (Aren't we?) On Thu, Jun 20, 2013 at 1:02 AM, Paul Ferguson fergdawgs...@gmail.comwrote: Hanlon's razor? Misconfiguration. Perhaps not done in malice, but I have no idea where the poison leaked in, or why. :-) - ferg On Wed, Jun 19, 2013 at 10:49 PM, Alex Buie alex.b...@frozenfeline.net wrote: Anyone have news/explanation about what's happening/happened? On Wed, Jun 19, 2013 at 10:34 PM, Paul Ferguson fergdawgs...@gmail.com wrote: Sure enough: ; DiG 9.7.3 @localhost yelp.com A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 53267 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;yelp.com. IN A ;; ANSWER SECTION: yelp.com. 300 IN A 204.11.56.20 ;; Query time: 143 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 07:33:13 2013 ;; MSG SIZE rcvd: 42 NetRange: 204.11.56.0 - 204.11.59.255 CIDR: 204.11.56.0/22 OriginAS: AS40034 NetName: CONFLUENCE-NETWORKS--TX3 NetHandle: NET-204-11-56-0-1 Parent: NET-204-0-0-0-0 NetType: Direct Allocation Comment: Hosted in Austin TX. Comment: Abuse : Comment: ab...@confluence-networks.com Comment: +1-917-386-6118 RegDate: 2012-09-24 Updated: 2012-09-24 Ref: http://whois.arin.net/rest/net/NET-204-11-56-0-1 OrgName: Confluence Networks Inc OrgId: CN Address: 3rd Floor, Omar Hodge Building, Wickhams Address: Cay I, P.O. Box 362 City: Road Town StateProv: Tortola PostalCode: VG1110 Country: VG RegDate: 2011-04-07 Updated: 2011-07-05 Ref: http://whois.arin.net/rest/org/CN OrgAbuseHandle: ABUSE3065-ARIN OrgAbuseName: Abuse Admin OrgAbusePhone: +1-917-386-6118 OrgAbuseEmail: ab...@confluence-networks.com OrgAbuseRef: http://whois.arin.net/rest/poc/ABUSE3065-ARIN OrgNOCHandle: NOCAD51-ARIN OrgNOCName: NOC Admin OrgNOCPhone: +1-415-462-7734 OrgNOCEmail: n...@confluence-networks.com OrgNOCRef: http://whois.arin.net/rest/poc/NOCAD51-ARIN OrgTechHandle: TECHA29-ARIN OrgTechName: Tech Admin OrgTechPhone: +1-415-358-0858 OrgTechEmail: ipad...@confluence-networks.com OrgTechRef: http://whois.arin.net/rest/poc/TECHA29-ARIN # # ARIN WHOIS data and services are subject to the Terms of Use # available at: https://www.arin.net/whois_tou.html # - ferg On Wed, Jun 19, 2013 at 10:30 PM, Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected On Wed, Jun 19, 2013 at 10:19 PM, John Levine jo...@iecc.com wrote: Reaching out to DNS operators around the globe. Linkedin.com has had some issues with DNS and would like DNS operators to flush their DNS. If you see www.linkedin.com resolving NS to ns1617.ztomy.com or ns2617.ztomy.com then please flush your DNS. Any other info please reach out to me off-list. While you're at it, www.usps.com, www.fidelity.com, and other well known sites have had DNS poisoning problems. When I restarted my cache, they look OK. -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: Need help in flushing DNS
I am not speaking officially, but the evidence so far is that this was not DNS poisoning, but domain name hijacking. My colleagues will have more to say later today. On Thu, Jun 20, 2013 at 1:19 AM, John Levine jo...@iecc.com wrote: Reaching out to DNS operators around the globe. Linkedin.com has had some issues with DNS and would like DNS operators to flush their DNS. If you see www.linkedin.com resolving NS to ns1617.ztomy.com or ns2617.ztomy.com then please flush your DNS. Any other info please reach out to me off-list. While you're at it, www.usps.com, www.fidelity.com, and other well known sites have had DNS poisoning problems. When I restarted my cache, they look OK.
Re: net neutrality and peering wars continue
On 20 June 2013 13:07, Bill Woodcock wo...@pch.net wrote: On Jun 19, 2013, at 7:21 PM, Benson Schliesser bens...@queuefull.net wrote: The sending peer (or their customer) has more control over cost. I'll assume that, by sending peer, you mean the content network. If so, I disagree. The content network has no control whatsoever over the location of the eyeball customer. The eyeball customer has sole control over his or her own location, while the content network has sole control over the location from which they reply to requests. Therefore, control is shared between the two sides. And both are incentivized to minimize costs. If both minimize their costs, overall costs are minimized. That's why this system works. I think his point was that the receiving side can massage their BGP announcements all they like but the sending network has more instantaneous control over how the traffic will flow. This is before analysis, communication, application of policies / contractual arrangements, de-peering etc.etc. kick in. cheers Marty
Re: net neutrality and peering wars continue
On Jun 20, 2013, at 8:09, Martin Barry ma...@supine.com wrote: On 20 June 2013 13:07, Bill Woodcock wo...@pch.net wrote: On Jun 19, 2013, at 7:21 PM, Benson Schliesser bens...@queuefull.net wrote: The sending peer (or their customer) has more control over cost. I'll assume that, by sending peer, you mean the content network. If so, I disagree. The content network has no control whatsoever over the location of the eyeball customer. ... I think his point was that the receiving side can massage their BGP announcements all they like but the sending network has more instantaneous control over how the traffic will flow. This is before analysis, communication, application of policies / contractual arrangements, de-peering etc.etc. kick in. Right. By sending peer I meant the network transmitting a packet, unidirectional flow, or other aggregate of traffic into another network. I'm not assuming anything about whether they are offering content or something else - I think it would be better to talk about peering fairness at the network layer, rather than the business / service layer. Cheers, -Benson
Re: net neutrality and peering wars continue
On Jun 19, 2013, at 23:41, Siegel, David david.sie...@level3.com wrote: Well, with net flow Analytics, it's not really the case that we don't have a way of evaluating the relative burdens. Every major net flow Analytics vendor is implementing some type of distance measurement capability so that each party can calculate not only how much traffic they carry for each peer, but how far. Admittedly, it's been a few years since I looked at such tools... So please help me understand: does the tool evaluate distance (and therefore burden) as it extends into the peer's network, or just into the local network? And in either case, is this kind of data normalized and shared between peers? It seems like there could be a mechanism here to evaluate fairness of burdens, but I'm skeptical that these tools are used in such a way. I'd be glad to be incorrect. ;) Cheers, -Benson
Re: Wiki for people doing IPv6-only testing
Thus spake Jason Fesler (jfes...@gigo.com) on Wed, Jun 19, 2013 at 04:55:01PM -0700: On a recent IPv6 providers call, there was a desire for participants to share information with each other on what works and what breaks in an IPv6-only environment. I offered to set that up. It was further suggested I should share this with more than just that small community; to anyone who might be doing work to test out IPv6-only scenarios. http://wiki.test-ipv6.com You may also want to check out the work Ron Broersma has done at DREN. Dale
RE: net neutrality and peering wars continue
The tools cannot estimate burden into the peers network very well, particularly when longest-exit routing is implement to balance the mileage burden, so each party shares their information with each other and compares data in order to make decisions. It's not common, but there are a handful of peers that share this information with each other. Dave -Original Message- From: Benson Schliesser [mailto:bens...@queuefull.net] Sent: Thursday, June 20, 2013 6:45 AM To: Siegel, David Cc: North American Network Operators' Group Subject: Re: net neutrality and peering wars continue On Jun 19, 2013, at 23:41, Siegel, David david.sie...@level3.com wrote: Well, with net flow Analytics, it's not really the case that we don't have a way of evaluating the relative burdens. Every major net flow Analytics vendor is implementing some type of distance measurement capability so that each party can calculate not only how much traffic they carry for each peer, but how far. Admittedly, it's been a few years since I looked at such tools... So please help me understand: does the tool evaluate distance (and therefore burden) as it extends into the peer's network, or just into the local network? And in either case, is this kind of data normalized and shared between peers? It seems like there could be a mechanism here to evaluate fairness of burdens, but I'm skeptical that these tools are used in such a way. I'd be glad to be incorrect. ;) Cheers, -Benson
Re: net neutrality and peering wars continue
On Jun 20, 2013, at 5:37 AM, Benson Schliesser bens...@queuefull.net wrote: Right. By sending peer I meant the network transmitting a packet, unidirectional flow, or other aggregate of traffic into another network. I'm not assuming anything about whether they are offering content or something else - I think it would be better to talk about peering fairness at the network layer, rather than the business / service layer. In that case, it's essentially never an issue, since essentially every packet in one direction is balanced by a packet in the other direction, so rotational symmetry takes care of the fairness. I think you may be taking your argument too far, though, since by this logic, the sending and receiving networks also have control over what they choose to transit and receive, and I think that discounts too far the reality that it is in fact the _customers_ that are making all of these decisions, and the networks are, in the aggregate, inflexible in their need to service customers. What a customer will pay to do, a service provider will take money to perform. It's not really service providers (in aggregate) making these decisions. It's customers. -Bill
RE: Need help in flushing DNS
Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH
Re: Need help in flushing DNS
Is there an organization that coordinates outages like this amongst the industry? On Thu, Jun 20, 2013 at 9:36 AM, Frank Bulk frnk...@iname.com wrote: Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacking/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH -- Phil Fagan Denver, CO 970-480-7618
Re: Need help in flushing DNS
I'm sure that folks in the ICANN SSAC will be talking about this subject well in to the future once a postmortem is completed. Also, perhaps even the DNS-OARC community. Coordination? This is the Internet! :-) - ferg On Thu, Jun 20, 2013 at 8:49 AM, Phil Fagan philfa...@gmail.com wrote: Is there an organization that coordinates outages like this amongst the industry? On Thu, Jun 20, 2013 at 9:36 AM, Frank Bulk frnk...@iname.com wrote: Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacking/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Re: Need help in flushing DNS
Hah..knew it On Thu, Jun 20, 2013 at 9:53 AM, Paul Ferguson fergdawgs...@gmail.comwrote: I'm sure that folks in the ICANN SSAC will be talking about this subject well in to the future once a postmortem is completed. Also, perhaps even the DNS-OARC community. Coordination? This is the Internet! :-) - ferg On Thu, Jun 20, 2013 at 8:49 AM, Phil Fagan philfa...@gmail.com wrote: Is there an organization that coordinates outages like this amongst the industry? On Thu, Jun 20, 2013 at 9:36 AM, Frank Bulk frnk...@iname.com wrote: Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/ http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacking/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618
Re: Need help in flushing DNS
I don't think there's one recognized authority. However, https://isc.sans.edu/ is pretty up to date. --chip On Thu, Jun 20, 2013 at 11:53 AM, Paul Ferguson fergdawgs...@gmail.comwrote: I'm sure that folks in the ICANN SSAC will be talking about this subject well in to the future once a postmortem is completed. Also, perhaps even the DNS-OARC community. Coordination? This is the Internet! :-) - ferg On Thu, Jun 20, 2013 at 8:49 AM, Phil Fagan philfa...@gmail.com wrote: Is there an organization that coordinates outages like this amongst the industry? On Thu, Jun 20, 2013 at 9:36 AM, Frank Bulk frnk...@iname.com wrote: Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/ http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacking/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Just my $.02, your mileage may vary, batteries not included, etc
Re: Need help in flushing DNS
Is there a need for such authority or coordination center? On Thu, Jun 20, 2013 at 9:59 AM, chip chip.g...@gmail.com wrote: I don't think there's one recognized authority. However, https://isc.sans.edu/ is pretty up to date. --chip On Thu, Jun 20, 2013 at 11:53 AM, Paul Ferguson fergdawgs...@gmail.comwrote: I'm sure that folks in the ICANN SSAC will be talking about this subject well in to the future once a postmortem is completed. Also, perhaps even the DNS-OARC community. Coordination? This is the Internet! :-) - ferg On Thu, Jun 20, 2013 at 8:49 AM, Phil Fagan philfa...@gmail.com wrote: Is there an organization that coordinates outages like this amongst the industry? On Thu, Jun 20, 2013 at 9:36 AM, Frank Bulk frnk...@iname.com wrote: Some news coverage here with pretty pictures of LinkedIn access: http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacki ng/ http://techcrunch.com/2013/06/19/linkedin-outage-due-to-possible-dns-hijacking/ Frank -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Thursday, June 20, 2013 1:23 AM To: Paul Ferguson Cc: NANOG list Subject: Re: Need help in flushing DNS On 6/20/13, Paul Ferguson fergdawgs...@gmail.com wrote: On Wed, Jun 19, 2013 at 10:44 PM, Tom Paseka t...@cloudflare.com wrote: On Wed, Jun 19, 2013 at 10:32 PM, Patrick W. Gilmore I think ztomy.com smells really bad for some reason, looks like 100% advertising; sure doesn't appear to be a DNS hosting provider, I sure can't imagine two major domains entering incorrect authoritative nameserver list changes on the same day... http://www.dailychanges.com/ztomy.com/#transferred-in The domain ztomy.com was registered on November 22, 2007, and we have nameserver history going back to December 9, 2007. It is listed as a nameserver for 182,174 domains Currently displaying 50 of 1,602 domain names transferred into ztomy.com on June 19, 2013. patr...@ianai.netwrote: On Jun 20, 2013, at 01:30 , Grant Ridder shortdudey...@gmail.com wrote: Yelp is evidently also affected Not from here. Patrick: $ dig NS yelp.com @8.8.8.8 +short ns1620.ztomy.com. ns2620.ztomy.com. -- -JH -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Just my $.02, your mileage may vary, batteries not included, etc -- Phil Fagan Denver, CO 970-480-7618
Re: Need help in flushing DNS
* philfa...@gmail.com (Phil Fagan) [Thu 20 Jun 2013, 17:50 CEST]: Is there an organization that coordinates outages like this amongst the industry? No; all outages on the Internet happen independently from each other and are not coordinated to (not) coincide in any way. -- Niels. -- It's amazing what people will do to get their name on the internet, which is odd, because all you really need is a Blogspot account. -- roy edroso, alicublog.blogspot.com
Re: Need help in flushing DNS
http://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ - Jared On Jun 19, 2013, at 11:42 PM, Zaid Ali Kahn z...@zaidali.com wrote: Reaching out to DNS operators around the globe. Linkedin.com has had some issues with DNS and would like DNS operators to flush their DNS. If you see www.linkedin.com resolving NS to ns1617.ztomy.com or ns2617.ztomy.com then please flush your DNS. Any other info please reach out to me off-list. Zaid
Re: Need help in flushing DNS
Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon
Re: Need help in flushing DNS
I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618
Re: Need help in flushing DNS
I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Re: Need help in flushing DNS
Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.comwrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618
This is a coordinated hacking. (Was Re: Need help in flushing DNS)
This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of localhost.. Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch ja...@puck.nether.net wrote: It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: net neutrality and peering wars continue
The tools cannot estimate burden into the peers network very well, particularly when longest-exit routing is implement to balance the mileage burden, so each party shares their information with each other and compares data in order to make decisions. It's not common, but there are a handful of peers that share this information with each other. i have not been able to find it easily, but some years back rexford and others published on a crypto method for peers to negotiate traffic adjustment between multiple peering points with minimal disclosure. it was a cool paper. randy
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes. I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info. Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity... On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw j...@arpa.com wrote: I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of localhost.. Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch ja...@puck.nether.net wrote: It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs -- -george william herbert george.herb...@gmail.com
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. As far as the logic of the DNS, it is functioning as designed (What's up, Vix!) - There's another aspect of this that caused this situation. Any Alexa or similar people on this list (Goog PR, etc)? I'd love to bulk submit a domain list for some analytics. Contact me off list. On Thu, Jun 20, 2013 at 3:14 PM, George Herbert george.herb...@gmail.comwrote: Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes. I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info. Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity... On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw j...@arpa.com wrote: I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of localhost.. Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch ja...@puck.nether.net wrote: It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs -- -george william herbert george.herb...@gmail.com -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
Not so easy and straightforward to do. You'll find that a lot of the big names out there frequently tweak DNS, which will result in a non-stop stream of alerts. Andy Andrew Fried andrew.fr...@gmail.com On 6/20/13 3:57 PM, Jared Mauch wrote: It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Fwd: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
Wait, wait. whois doesnt jive with dns. .. Conspiracy Theory Hat On : - Did someone gain access to the COM dispersion zone, or parts thereof? - Did someone figure out how to [ insert theory here ] ? I'm looking at domains that were solidly pointing at ztomy at 2:30AM (that are 'recovered' to other nameservers) that show no updates in `whois` records. Curiouser and curiouser. Paul? -- Forwarded message -- From: jamie rishaw j...@arpa.com Date: Thu, Jun 20, 2013 at 3:21 PM Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) To: George Herbert george.herb...@gmail.com Cc: Jared Mauch ja...@puck.nether.net, NANOG nanog@nanog.org It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. As far as the logic of the DNS, it is functioning as designed (What's up, Vix!) - There's another aspect of this that caused this situation. Any Alexa or similar people on this list (Goog PR, etc)? I'd love to bulk submit a domain list for some analytics. Contact me off list. On Thu, Jun 20, 2013 at 3:14 PM, George Herbert george.herb...@gmail.comwrote: Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes. I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info. Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity... On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw j...@arpa.com wrote: I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of localhost.. Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch ja...@puck.nether.net wrote: It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw j...@arpa.com wrote: This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., I know you or know of you,) wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan philfa...@gmail.com wrote: Agree'd in these smaller scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide services are effected. Perhaps the result of a State led campaign topic for another day. On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson fergdawgs...@gmail.com wrote: I am betting that Netsol doesn't need any more coordination at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS: ; DiG 9.7.3 @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;parsonstech.com.INNS ;; ANSWER SECTION: parsonstech.com.172800INNSns2617.ztomy.com. parsonstech.com.172800INNSns1617.ztomy.com. ;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81 - ferg On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan philfa...@gmail.com wrote: I should caveat.coordinate the recovery of. On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth bran...@rd.bbc.co.ukwrote: Is there an organization that coordinates outages like this amongst the industry? No, usually they are surprise outages though Anonymous have tried coordinating a few brandon -- Phil Fagan Denver, CO 970-480-7618 -- Fergie, a.k.a. Paul Ferguson fergdawgster(at)gmail.com -- Phil Fagan Denver, CO 970-480-7618 -- -george william herbert george.herb...@gmail.com
Re: net neutrality and peering wars continue
* wo...@pch.net (Bill Woodcock) [Thu 20 Jun 2013, 16:59 CEST]: On Jun 20, 2013, at 5:37 AM, Benson Schliesser bens...@queuefull.net wrote: Right. By sending peer I meant the network transmitting a packet, unidirectional flow, or other aggregate of traffic into another network. I'm not assuming anything about whether they are offering content or something else - I think it would be better to talk about peering fairness at the network layer, rather than the business / service layer. In that case, it's essentially never an issue, since essentially every packet in one direction is balanced by a packet in the other direction, so rotational symmetry takes care of the fairness. You're mistaken if you think that CDNs have equal number of packets going in and out. I think you may be taking your argument too far, though, since by this logic, the sending and receiving networks also have control over what they choose to transit and receive, and I think that discounts too far the reality that it is in fact the _customers_ that are making all of these decisions, and the networks are, in the aggregate, inflexible in their need to service customers. What a customer will pay to do, a service provider will take money to perform. It's not really service providers (in aggregate) making these decisions. It's customers. I think the point is here that networks are nudging these decisions by making certain services suck more than others by way of preferential network access. -- Niels.
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On 6/20/13, jamie rishaw j...@arpa.com wrote: It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. https://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ -- -JH
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On 6/20/2013 1:46 PM, Jimmy Hess wrote: On 6/20/13, jamie rishaw j...@arpa.com wrote: It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. https://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ -- -JH small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number. -- Jeff Shultz
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries... I have no data to base this speculation on. Grüße, Carsten
RE: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
Hello everyone, I'm new here. +1 to this theory. I've been watching what's happening since 3am Eastern, because a domain of mine (of the many at NetSol) was a victim of this event. -Gabor -Original Message- From: Carsten Bormann [mailto:c...@tzi.org] Sent: Thursday, June 20, 2013 5:11 PM To: NANOG list Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries... I have no data to base this speculation on. Grüße, Carsten
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number. It's relatively small when you consider there's something like 140M .com's pgpA4dQRKUb7v.pgp Description: PGP signature
Re: net neutrality and peering wars continue
On Thu, 20 Jun 2013 22:39:56 +0200, Niels Bakker said: You're mistaken if you think that CDNs have equal number of packets going in and out. And even if the number of packets match, there's the whole 1500 bytes of data, 64 bytes of ACK thing to factor in... pgp0aUntNCndk.pgp Description: PGP signature
Re: net neutrality and peering wars continue
On Jun 20, 2013, at 10:39 PM, Niels Bakker niels=na...@bakker.net wrote: * wo...@pch.net (Bill Woodcock) [Thu 20 Jun 2013, 16:59 CEST]: On Jun 20, 2013, at 5:37 AM, Benson Schliesser bens...@queuefull.net wrote: Right. By sending peer I meant the network transmitting a packet, unidirectional flow, or other aggregate of traffic into another network. I'm not assuming anything about whether they are offering content or something else - I think it would be better to talk about peering fairness at the network layer, rather than the business / service layer. In that case, it's essentially never an issue, since essentially every packet in one direction is balanced by a packet in the other direction, so rotational symmetry takes care of the fairness. You're mistaken if you think that CDNs have equal number of packets going in and out. They are roughly equal (modulo delayed acks, etc.). However, the number of octets is very different from the number of packets. There is much greater asymmetry in number of octets than in number of packets. To the best of my knowledge, most (if not all) of the peering agreements that discuss traffic ratios do so in terms of data transferred, not number of datagrams. Owen
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On 20 June 2013 14:28, valdis.kletni...@vt.edu wrote: On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number. It's relatively small when you consider there's something like 140M .com's So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only the websites given how many other things folks are likely to be using DNS for... .r'
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? luckily, none of the rest of us make mistakes
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
I don't think he was saying that at all. Just stating that from a pure numbers standpoint 50k/140mil is a small percentage. OTOH, I agree to your point - Network Solutions definitely downplayed this in their release. Curiously so. Sent from my iPhone On Jun 20, 2013, at 5:42 PM, RijilV rij...@riji.lv wrote: On 20 June 2013 14:28, valdis.kletni...@vt.edu wrote: On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number. It's relatively small when you consider there's something like 140M .com's So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only the websites given how many other things folks are likely to be using DNS for... .r'
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Thu, Jun 20, 2013 at 2:49 PM, Randy Bush ra...@psg.com wrote: So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? luckily, none of the rest of us make mistakes Ages ago I responded on a Cisco list where the topic was biggest screwup you've made. I posted that I once forgot the implicit deny in an ACL and accidentally blocked all traffic between 4 locations in 2 states for a company I was working for. Downtime was a very brutal 60 seconds. Someone very insightful responded with anyone who hasn't done similar is lying about the 10 years on their resume. So the real question would be, why wasn't there someone who has already done this in the past working on this zone? ;) -B
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Thu, 2013-06-20 at 14:42 -0700, RijilV wrote: On 20 June 2013 14:28, valdis.kletni...@vt.edu wrote: On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said: small number of Network Solutions customers They must be staffed with physicists, astronomers, or economists I don't know anyone else that would consider nearly fifty thousand (from a previous post by Phil Fagan) to be a small number. It's relatively small when you consider there's something like 140M .com's So it's okay to screw over nearly fifty thousand customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only the websites given how many other things folks are likely to be using DNS for... .r' I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner
Re: net neutrality and peering wars continue
* o...@delong.com (Owen DeLong) [Thu 20 Jun 2013, 23:38 CEST]: On Jun 20, 2013, at 10:39 PM, Niels Bakker niels=na...@bakker.net wrote: * wo...@pch.net (Bill Woodcock) [Thu 20 Jun 2013, 16:59 CEST]: On Jun 20, 2013, at 5:37 AM, Benson Schliesser bens...@queuefull.net wrote: Right. By sending peer I meant the network transmitting a packet [...] every packet in one direction is balanced by a packet in the other direction You're mistaken if you think that CDNs have equal number of packets going in and out. They are roughly equal (modulo delayed acks, etc.). However, the number of octets is very different from the number of packets. There is much greater asymmetry in number of octets than in number of packets. Thank you, Captain Obvious. Also, if you don't have data, best to keep your opinion to yourself, because you might well be wrong. -- Niels.
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
At the DNS Servers or service provider level, one can (and I often do) have redundant providers. At the registrar level? ... Not with our current infrastructure, as far as I know how. The Internet: Discovering new SPOF since 1969! George William Herbert Sent from my iPhone On Jun 20, 2013, at 3:28 PM, Randy Bush ra...@psg.com wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it shouldwho's the clearing house here. On Thu, Jun 20, 2013 at 4:28 PM, Randy Bush ra...@psg.com wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy -- Phil Fagan Denver, CO 970-480-7618
Re: net neutrality and peering wars continue
Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee This leaves the operational model of WAN backbone transit networks unchanged: fights about traffic balance and settlement fees can continue in perpetuity. Those big sources who fall afoul of balance can opt to deliver traffic directly to the last-mile network(s) in given markets. Transfers WAN networking cost-burden to the content originator (through their agents: CDN operators or transit providers) Reduces financial burden on last-mile operator (demand is reduced on their company operated backbone and/or transit capacity that they purchase) RESULTS Customers get to receive content they are requesting: technical and political impediments are removed. Last-mile operator only has to improve in-region network facilities: to deliver the data that their own customers have requested
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
No. The ztomy nameservers appeared in this morning's master .COM zonefile as /authoritative/ for the number of domains I mentioned. It is a clear change from just a couple of days ago, when the listed nameservers were nowhere to be seen. I have solid data to back this up, straight from Verisign GRS (Verisign), the authoritative registry for .COM, .NET and others. j On Thu, Jun 20, 2013 at 4:10 PM, Carsten Bormann c...@tzi.org wrote: Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries... I have no data to base this speculation on. Grüße, Carsten -- Jamie Rishaw // .com.arpa@j - reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Re: net neutrality and peering wars continue
On Jun 20, 2013, at 5:47 PM, Robert M. Enger na...@enger.us wrote: Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee C) Buck up and carry the traffic their customers are paying them to carry. Least I just sound like a complainer, I actually think this makes rational business sense. The concept of peering was always equal benefit, not equal cost. No one ever compares the price of building last mile transport to the cost of building huge data centers all over with content close to the users. The whole bit-mile thing represents an insignificant portion of the cost, long haul (in large quantities) is dirt cheap compared to last mile or data center build costs. If you think of a pure content play peering with a pure eyeball play there is equal benefit, in fact symbiosis, neither could exist without the other. The traffic flow will be highly asymmetric. Eyeball networks also artificially cap their own ratios with their products. Cable and DSL are both 3x-10x down, x up products. Their TOS policies prohibit running servers. Any eyeball network with a asymmetric edge technology and no-server TOS need only look in the mirror to see why their aggregate ratio is hosed. Lastly, simple economics. Let's theorize about a large eyeball network with say 20M subscribers, and a large content network with say 100G of peering traffic to go to those subscribers. * Choice A would be to squeeze the peer for bad ratio in the hope of getting them to pay for, or be behind some other transit customer. Let's be generous and say $3/meg/month, so the 100G of traffic might generate $300,000/month of revenue. Let's even say you can squeeze 5 CDN's for that amount, $1.5M/month total. * Choice B would be to squeeze the subscribers for more revenue to carry the 100G of imbalanced traffic. Perhaps an extra $0.10/sub/month. That would be $2M/month in extra revenue. Now, consider the customer satisfaction issue? Would your broadband customers pay an extra $0.10 per month if Netflix and Amazon streaming never went out in the middle of a movie? Would they move up to a higher tier of service? A smart end user ISP would find a way to get uncongested paths to the content their users want, and make it rock solid reliable. The good service will more than support not only cost recovery, but higher revenue levels than squeezing peers. Of course we have evidence that most end user ISP's are not smart, they squeeze peers and have some of the lowest customer satisfaction rankings of not just ISP's, but all service providers! They want to claim consumers don't want Gigabit fiber, but then congest peers so badly there's no reason for a consumer to pay for more than the slowest speed. Squeezing peers is a prime case of cutting off your nose to spite your face. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
I, for one, would not be in favor of an authoritarian rule over DNS, or any other Internet system, to ensure that the state of [the] service[s] is running as it should. I suppose one could view such an authoritarian rule over (sub) systems to be a good thing, as in there is someone to complain to when things don't work, but recent events show that it is also easily abused. I much rather prefer the current cooperative administration of the Internet. Thanks, Fred Reimer On 6/20/13 6:39 PM, Phil Fagan philfa...@gmail.com wrote: at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it shouldwho's the clearing house here. On Thu, Jun 20, 2013 at 4:28 PM, Randy Bush ra...@psg.com wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy -- Phil Fagan Denver, CO 970-480-7618
Fwd: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Jun 20, 2013 5:31 PM, Randy Bush ra...@psg.com wrote: and dnssec did not save us. is there anything which could have? Hmmm. DNSSEC wouldn't have prevented an outage. But from everything I've seen reported, had the zones been signed, validating recursive resolvers (comcast, google, much of federal government, mine) would have returned servfail and would not have cached the bad nameservers in their good cache. Users would have simply failed to connect instead of being sent to the wrong page and recovery would have been quicker and easier. From my perspective as someone responsible for DNS at a fairly large enterprise, that would have been preferable. But then, the zones for which I'm responsible are signed. YMMV, Scott
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On 6/20/13, Randy Bush ra...@psg.com wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? What's puzzling is the How the heck did they do that? The registrar doesn't maintain the .COM database that contains the list of nameservers they had to submit changes to all those records. So, why weren't there security controls to make sure that the registrar could not submit changes without appropriate authorization from the Administrative/Tech contact? randy -- -JH
Re: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Thu, Jun 20, 2013 at 8:41 PM, Timothy Morizot tmori...@gmail.com wrote: On Jun 20, 2013 5:31 PM, Randy Bush ra...@psg.com wrote: and dnssec did not save us. is there anything which could have? Hmmm. DNSSEC wouldn't have prevented an outage. But from everything I've seen reported, had the zones been signed, validating recursive resolvers (comcast, google, much of federal government, mine) would have returned servfail and would not have cached the bad nameservers in their good cache. Users would have simply failed to connect instead of being sent to the wrong page and recovery would have been quicker and easier. From my perspective as someone responsible for DNS at a fairly large enterprise, that would have been preferable. But then, the zones for which I'm responsible are signed. In this case of registrar compromise, DS record could have been changed alongside NS records, so DNSSEC would only have been a early warning, because uncoordinated DS change disrupts service. As soon as previous timeouts played out, new DS/NS pairs would be considered as trustworthy as the old ones. Rubens
Re: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Jun 20, 2013 7:30 PM, Rubens Kuhl rube...@gmail.com wrote: In this case of registrar compromise, DS record could have been changed alongside NS records, so DNSSEC would only have been a early warning, because uncoordinated DS change disrupts service. As soon as previous timeouts played out, new DS/NS pairs would be considered as trustworthy as the old ones. Since DS records typically have a ttl of 24 hours, that protection should not be underestimated even in the case of registrar compromise. However, everything released so far indicates this was a netsol error and not a compromise. And it was an error corrected fairly quickly from what I can tell. The impact was prolonged because the bad nameservers were cached in resolvers across the Internet. Of course, very few details have actually been released, so that construction could be wrong. But even in the worst case DNSSEC would have provided some mitigation for a time.
Network diagnostics for the end user
Are there any tools out there that we could give to our end users to help diagnose network problems? We get a lot of the Internet is slow support calls and it would be helpful if we had something that would run on the end user's computer and help characterize the problem. We have central monitoring system of course but that doesn't always give a complete picture, as the problem could always be on the end user's computer - slow hard drive, not enough memory, wrong name servers, etc.
Re: net neutrality and peering wars continue
It's only cutting off your nose to spite your face if you look at the internet BU in a vacuum. The issue comes when they can get far more money from their existing product line, than what they get being a dumb bandwidth pipe to their customers. They don't want reasonable or even unreasonable pricing per meg, they want content to pay for access to their customers in the same range of cost that they currently get from their other arm's subscribers or to sit down and shut up and stop competing with their much more profitable broadcast arm. Because they can't just charge a premium on the internet access itself, as their customers would leave due to competition from providers that *are* just dumb pipes to transit based content. -Blake On Thu, Jun 20, 2013 at 6:18 PM, Leo Bicknell bickn...@ufp.org wrote: On Jun 20, 2013, at 5:47 PM, Robert M. Enger na...@enger.us wrote: Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee C) Buck up and carry the traffic their customers are paying them to carry. Least I just sound like a complainer, I actually think this makes rational business sense. The concept of peering was always equal benefit, not equal cost. No one ever compares the price of building last mile transport to the cost of building huge data centers all over with content close to the users. The whole bit-mile thing represents an insignificant portion of the cost, long haul (in large quantities) is dirt cheap compared to last mile or data center build costs. If you think of a pure content play peering with a pure eyeball play there is equal benefit, in fact symbiosis, neither could exist without the other. The traffic flow will be highly asymmetric. Eyeball networks also artificially cap their own ratios with their products. Cable and DSL are both 3x-10x down, x up products. Their TOS policies prohibit running servers. Any eyeball network with a asymmetric edge technology and no-server TOS need only look in the mirror to see why their aggregate ratio is hosed. Lastly, simple economics. Let's theorize about a large eyeball network with say 20M subscribers, and a large content network with say 100G of peering traffic to go to those subscribers. * Choice A would be to squeeze the peer for bad ratio in the hope of getting them to pay for, or be behind some other transit customer. Let's be generous and say $3/meg/month, so the 100G of traffic might generate $300,000/month of revenue. Let's even say you can squeeze 5 CDN's for that amount, $1.5M/month total. * Choice B would be to squeeze the subscribers for more revenue to carry the 100G of imbalanced traffic. Perhaps an extra $0.10/sub/month. That would be $2M/month in extra revenue. Now, consider the customer satisfaction issue? Would your broadband customers pay an extra $0.10 per month if Netflix and Amazon streaming never went out in the middle of a movie? Would they move up to a higher tier of service? A smart end user ISP would find a way to get uncongested paths to the content their users want, and make it rock solid reliable. The good service will more than support not only cost recovery, but higher revenue levels than squeezing peers. Of course we have evidence that most end user ISP's are not smart, they squeeze peers and have some of the lowest customer satisfaction rankings of not just ISP's, but all service providers! They want to claim consumers don't want Gigabit fiber, but then congest peers so badly there's no reason for a consumer to pay for more than the slowest speed. Squeezing peers is a prime case of cutting off your nose to spite your face. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: net neutrality and peering wars continue
Maybe someone could enlighten my ignorance on this issue. Why is there a variable charge for bandwidth anyways? In a very simplistic setup, if I have a router that costs $X and I run a $5 CAT6 cable to someone elses router which cost them $Y, plus a bit of maintenance time to set up the connections, tweak ACLs, etc... So now there's an interconnect between two providers at 1 gigabit, and the only issue I see is the routers needing to be replaced within Z years when it dies or when it needs to handle a 10 gigabit connection. So it seems I should be able to say Here's a 1 gigabit connection. It will cost $Q over Z years or you can pay $Q/Z yearly, etc... And wouldn't the costs go down if I had a bunch of dialup/DSL/cable/fiber users as they are paying to lower the costs of interconnects so they get content with less latency and fewer bottlenecks? -A On Thu, Jun 20, 2013 at 4:18 PM, Leo Bicknell bickn...@ufp.org wrote: On Jun 20, 2013, at 5:47 PM, Robert M. Enger na...@enger.us wrote: Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee C) Buck up and carry the traffic their customers are paying them to carry. Least I just sound like a complainer, I actually think this makes rational business sense. The concept of peering was always equal benefit, not equal cost. No one ever compares the price of building last mile transport to the cost of building huge data centers all over with content close to the users. The whole bit-mile thing represents an insignificant portion of the cost, long haul (in large quantities) is dirt cheap compared to last mile or data center build costs. If you think of a pure content play peering with a pure eyeball play there is equal benefit, in fact symbiosis, neither could exist without the other. The traffic flow will be highly asymmetric. Eyeball networks also artificially cap their own ratios with their products. Cable and DSL are both 3x-10x down, x up products. Their TOS policies prohibit running servers. Any eyeball network with a asymmetric edge technology and no-server TOS need only look in the mirror to see why their aggregate ratio is hosed. Lastly, simple economics. Let's theorize about a large eyeball network with say 20M subscribers, and a large content network with say 100G of peering traffic to go to those subscribers. * Choice A would be to squeeze the peer for bad ratio in the hope of getting them to pay for, or be behind some other transit customer. Let's be generous and say $3/meg/month, so the 100G of traffic might generate $300,000/month of revenue. Let's even say you can squeeze 5 CDN's for that amount, $1.5M/month total. * Choice B would be to squeeze the subscribers for more revenue to carry the 100G of imbalanced traffic. Perhaps an extra $0.10/sub/month. That would be $2M/month in extra revenue. Now, consider the customer satisfaction issue? Would your broadband customers pay an extra $0.10 per month if Netflix and Amazon streaming never went out in the middle of a movie? Would they move up to a higher tier of service? A smart end user ISP would find a way to get uncongested paths to the content their users want, and make it rock solid reliable. The good service will more than support not only cost recovery, but higher revenue levels than squeezing peers. Of course we have evidence that most end user ISP's are not smart, they squeeze peers and have some of the lowest customer satisfaction rankings of not just ISP's, but all service providers! They want to claim consumers don't want Gigabit fiber, but then congest peers so badly there's no reason for a consumer to pay for more than the slowest speed. Squeezing peers is a prime case of cutting off your nose to spite your face. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: net neutrality and peering wars continue
On Jun 20, 2013, at 9:10 PM, Aaron C. de Bruyn aa...@heyaaron.com wrote: Why is there a variable charge for bandwidth anyways? In a very simplistic setup, if I have a router that costs $X and I run a $5 CAT6 cable to someone elses router which cost them $Y, plus a bit of maintenance time to set up the connections, tweak ACLs, etc... So now there's an interconnect between two providers at 1 gigabit, and the only issue I see is the routers needing to be replaced within Z years when it dies or when it needs to handle a 10 gigabit connection. Many things aren't as obvious as you state above. Take for example routing table growth. There's going to be a big boom in selling routers (or turning off full routes) when folks devices melt at 512k routes in the coming years. Operating a router takes a lot of things, including power, space, people to rack it, swap failing or failed hardware, OPEX to the vendor to cover support contract (assuming you have one), fiber cleaning kits, new patch cables, optics, etc. These costs are variable per city and location as space/power can be different. This doesn't include telecom costs, which may be up/down depending on if you are using leased/dark/IRU or other services. Building fiber, data centers, can be quite capital expensive. Fiber, expect 50-100k per mile (for example). It can be even more depending on the market and situation. Much of that cost is in the labor to the technicians as well as local permits as opposed to what the fiber actually costs. Many people have fiber they built 10 years ago, or even older. Folks like ATT have been breathing life into their copper plant that was built over the past 100 years. Having that existing right-of-way makes permit costs lower, or allows you to get a blanket permit for entire cities/counties in cases. Some cable company has a presentation out there (maybe it was at a cable labs conference, or otherwise) I saw about average breaks per year. This costs splicing crews that you either have to pay to be on call or outsource to a contract company for emergency restoration. http://www.southern-telecom.com/AFL%20Reliability.pdf has some details about these. So it seems I should be able to say Here's a 1 gigabit connection. It will cost $Q over Z years or you can pay $Q/Z yearly, etc... And wouldn't the costs go down if I had a bunch of dialup/DSL/cable/fiber users as they are paying to lower the costs of interconnects so they get content with less latency and fewer bottlenecks? There was a presentation by Vijay about the costs of customer support. Many states have minimum wages higher than the federal minimum wage, but even that being said, you need to pay someone, train them, give them a computer, manager, phone and other guidance to provide support for billing, customer retention and sales. I recall Vijay saying that if a customer phoned for support it wiped out the entire profit from the customer for the lifetime of them being a customer. That may not still be the case, but there are costs each time you provide a staff person to answer that phone. Sometimes it's due to outage, sometimes it's PBKAC, sometimes you don't know and have to further research the issue. Your overhead costs may be much higher due to the type of other costs you bear (pension, union contracts, etc..) vs a competitor that doesn't have that same structure. This is often seen in the airline industry. I for one would like to see more competition in the last mile in the US, but I think the only people that will do it will be folks like sonic.net, google and other smaller independent telcos. Take someone like Allband Communications in Michigan. They brought POTS service (just recently) to locations that Verizon/ATT were unwilling to build. The person who wanted the phone service ended up having to start a telco to get POTS service there. They just went triple-play since it was the same cost to trench fiber as to put in the copper. - Jared
Re: net neutrality and peering wars continue
On 6/20/2013 10:26 PM, Jared Mauch wrote: Many things aren't as obvious as you state above. Take for example routing table growth. There's going to be a big boom in selling routers (or turning off full routes) when folks devices melt at 512k routes in the coming years. Indeed. We're running PFC3CXL's and had already reallocated FIB TCAM to 768K IPv4s in anticipation. We also had maximum-prefix 50 with a warning at 90%, and today it triggered (or at least first time I noticed it)... we ran 450K prefixes from 3 providers about 1:30 EDT today and got the warnings. The end is near :) If you haven't made provisions, please do so now :) Jeff
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
At 07:28 21/06/2013 +0900, Randy Bush wrote: netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. They are too busy adding new revenue: http://www.streetinsider.com/Corporate+News/NetSol+%28NTWK%29+Enters+$10M+Agreement+for+Financial+Suite+Implementation/8434663.html -Hank
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
At 17:12 20/06/2013 -0500, Richard Golodner wrote: I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner sarcasm and Netsol agrees with you: http://www.networksolutions.com/blog/2013/06/important-update-for-network-solutions-customers-experiencing-website-issues/ a small number of Network Solutions customers were inadvertently affected for up to several hours. /sarcasm -Hank
Re: net neutrality and peering wars continue
On Fri, Jun 21, 2013 at 12:26:01AM +0200, Niels Bakker wrote: [snip] Also, if you don't have data, best to keep your opinion to yourself, because you might well be wrong. The deuce you say! Replacing uninformed conjecture and conspiracy theories with actual data? Next thing you know there will be actual engineering discussions instead ... -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE / NANOG
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it shouldwho's the clearing house here. The Internet: Discovering new SPOF since 1969! :) Thanks. Perhaps we should setup a distributed system for checking things rather than another SPOF. That's distributed both geographically and administratively and using several code-bases. In this context, I'd expect lots of false alarms due to people changing their DNS servers but forgetting to inform their monitoring setup (either internal or outsourced). How would you check/verify that the communication path from the monitoring agency to the right people in your NOC was working correctly? -- These are my opinions. I hate spam.
Re: net neutrality and peering wars continue
On Thu, 20 Jun 2013, Jeff Kell wrote: On 6/20/2013 10:26 PM, Jared Mauch wrote: Many things aren't as obvious as you state above. Take for example routing table growth. There's going to be a big boom in selling routers (or turning off full routes) when folks devices melt at 512k routes in the coming years. Indeed. We're running PFC3CXL's and had already reallocated FIB TCAM to 768K IPv4s in anticipation. We also had maximum-prefix 50 with a warning at 90%, and today it triggered (or at least first time I noticed it)... we ran 450K prefixes from 3 providers about 1:30 EDT today and got the warnings. The end is near :) If you haven't made provisions, please do so now :) It's like 2008 all over again, but worse. In 2008, the Sup2 was nearing the end of its ability to hold full v4 routes. The good news back then was that you could upgrade to Sup720-3bxls for a little more than (IIRC) about $10k per unit. This time, at least as of today, Cisco hasn't provided an upgrade path that'll keep the 6500 family usable for a full-table router when the 1 Million route slots aren't enough to hold your 768k v4 routes and 128k v6 routes. At this rate, if they do produce a PFC that takes the 6500 to several million routes, it's probably going to be too late for those to be available in any real quantity on the secondary market. Maybe that's the plan. -- Jon Lewis, MCP :) | I route | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
I think ICANN would have to add a delay in where a request was sent out to make sure everyone was on the same page and then what happens the couple thousand (more) times a day that someone isn't updated or is misconfigured? I think Netsol should be fined. Maybe even a class action suite filed against them for lost business. And that's it. On Jun 20, 2013 11:28 PM, Hal Murray hmur...@megapathdsl.net wrote: at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it shouldwho's the clearing house here. The Internet: Discovering new SPOF since 1969! :) Thanks. Perhaps we should setup a distributed system for checking things rather than another SPOF. That's distributed both geographically and administratively and using several code-bases. In this context, I'd expect lots of false alarms due to people changing their DNS servers but forgetting to inform their monitoring setup (either internal or outsourced). How would you check/verify that the communication path from the monitoring agency to the right people in your NOC was working correctly? -- These are my opinions. I hate spam.
Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS)
On Thu, 20 Jun 2013 20:25:24 -0700, Hal Murray said: How would you check/verify that the communication path from the monitoring agency to the right people in your NOC was working correctly? Remember to consider the possible impact of a false-positive report over an unauthenticated channel. Because if it's possible, somebody will try it, just because they just want to watch stuff burn. :) pgpvQasT4FmSG.pgp Description: PGP signature
Re: Network diagnostics for the end user
On 20/06/13 17:45, Jeffrey Ollie wrote: Are there any tools out there that we could give to our end users to help diagnose network problems? We get a lot of the Internet is slow support calls and it would be helpful if we had something that would run on the end user's computer and help characterize the problem. We have central monitoring system of course but that doesn't always give a complete picture, as the problem could always be on the end user's computer - slow hard drive, not enough memory, wrong name servers, etc. I personally like ICSI Netalyzr for identifying gross issues. http://netalyzr.icsi.berkeley.edu/ -- /*=[ Jake Khuon kh...@neebu.net ]=+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==*/
Re: Network diagnostics for the end user
I personally like ICSI Netalyzr for identifying gross issues. http://netalyzr.icsi.berkeley.edu/ +42
