Level 3 was down in KC, Chi, and San Jose (at least) for us between about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.
-Steve > -----Original Message----- > From: [email protected] [mailto:[email protected]] > Sent: Monday, November 07, 2011 10:05 AM > To: [email protected] > Subject: NANOG Digest, Vol 46, Issue 15 > > Send NANOG mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.nanog.org/mailman/listinfo/nanog > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NANOG digest..." > > > Today's Topics: > > 1. Re: Time Warner Telecom problems (Jon Lewis) > 2. Re: Performance Issues - PTR Records (Leigh Porter) > 3. Re: Time Warner Telecom problems (Ray Van Dolson) > 4. General Internet Instability (Jared Mauch) > 5. Re: TATA problems? (Pierre-Yves Maunier) > 6. Re: TATA problems? (Leigh Porter) > 7. Re: Time Warner Telecom problems (Joe Greco) > 8. Re: TATA problems? (Kelly Kane) > 9. Re: Time Warner Telecom problems (Blake Hudson) > 10. RE: Time Warner Telecom problems (Thomas York) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST) > From: Jon Lewis <[email protected]> > To: Peter Pauly <[email protected]> > Cc: [email protected] > Subject: Re: Time Warner Telecom problems > Message-ID: <[email protected]> > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > > On Mon, 7 Nov 2011, Peter Pauly wrote: > > > Gizmodo is reporting problems at Time Warner Telecom .... we're > suffering > > from it too and calls to the NOC have not been answered so far... does > > anyone have any further information? > > > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us > > I noticed just a little while ago that we're having a lot of DNS fail. > Initial findings were that several of the root-servers we were trying to > reach via our TWTelecom link were unreachable after 2 hops into TWT. > > 4 64-128-130-233.static.twtelecom.NET (64.128.130.233) 2.399 ms 2.298 > ms 2.338 ms > 5 mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18) 11.571 ms > 11.552 ms 9.467 ms > 6 * * * > 7 * * * > 8 * * * > > For instance, a.root-servers.net is pingable from a rackspace server, but > not from our network (unless I shut off TWT, at which point it is, but > it's apparently not the same a.root-servers.net instance rackspace sees). > I assume this is one of the root-servers being anycast. > > Shutting off our BGP with TWT didn't appear to help (though the > root-servers became reachable)...so I assume there's more going on than > just TWT routing fail. > > ---------------------------------------------------------------------- > Jon Lewis, MCP :) | I route > Senior Network Engineer | therefore you are > Atlantic Net | > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ > > > > ------------------------------ > > Message: 2 > Date: Mon, 7 Nov 2011 15:29:30 +0000 > From: Leigh Porter <[email protected]> > To: Bj?rn Mork <[email protected]> > Cc: "[email protected]" <[email protected]> > Subject: Re: Performance Issues - PTR Records > Message-ID: <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > > > On 7 Nov 2011, at 14:03, "Bj?rn Mork" <[email protected]> wrote: > > > Leigh Porter <[email protected]> writes: > > > >> Indeed, there is no way I would allow that either. But really, > >> providing a reverse zone and forward zone to match is a case of five > >> minutes and a shell script or a DNS that as Steinar said, will > >> synthesise results. > >> > >> It's really not all that difficult.. > > > > No, not at all. It's just totally pointless. Any IPv6 address is just > > as pretty as a synthesized name. Maybe even prettier. Do you prefer > > "2001:db8:1::2" or "20010db8000100000000000000000002.rev.example.com"? > > > > If we're going to provide any reverse DNS for end users, then it is > > because we can create names which actually improves something. > > > > > > Bj?rn > > > > > > Yup it is pointless.. Mine are all ipadrress.domain which is of course, > pointless.. I suppose at least somebody would glean that perhaps its a > home user rather than a business or server on that address but that's all. > > With IPv6 arguably even more pointless as you say. > > ______________________________________________________________________ > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > ______________________________________________________________________ > > > > ------------------------------ > > Message: 3 > Date: Mon, 7 Nov 2011 07:28:18 -0800 > From: Ray Van Dolson <[email protected]> > To: [email protected] > Subject: Re: Time Warner Telecom problems > Message-ID: <[email protected]> > Content-Type: text/plain; charset=us-ascii > > On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote: > > Gizmodo is reporting problems at Time Warner Telecom .... we're > suffering > > from it too and calls to the NOC have not been answered so far... does > > anyone have any further information? > > > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us > > FWIW, my home TWC connection dropped this morning for about 15 minutes > (Southern California around 6:30AM'ish). Still could ping the default > gateway, but packets weren't traversing much beyond that. > > Didn't investigate further, just headed into work. > > Ray > > > > ------------------------------ > > Message: 4 > Date: Mon, 7 Nov 2011 10:31:31 -0500 > From: Jared Mauch <[email protected]> > To: Tom Hill <[email protected]> > Cc: [email protected] > Subject: General Internet Instability > Message-ID: <[email protected]> > Content-Type: text/plain; charset=us-ascii > > On Nov 7, 2011, at 10:08 AM, Tom Hill wrote: > > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: > >> We seem to be having some problems with our tata links - first seen in > EU > >> about 45 minutes ago, now we're seeing problems in NA. I'm focused on > DNS, > >> so I'm seeing a lot of timeouts/servfails, but our networking folks are > >> talking about links dropping. > >> > >> Anyone else seeing oddness on the NA Internet right now? > >> > >> http://downrightnow.com/ confirms - something is up. > > > > There are widespread issues across the Internet; certain versions of > > Juniper firmware have core dumped after seeing a particular BGP 'UPDATE' > > message. > > > > (That's the running theory at least). > > > > It's affected multiple service providers, globally, not just those > > connected to TATA. > > > Pretty much any major BGP event will impact multiple providers. > > A threshold you should use to view the general instability (which I find > valuable, you may as well) is route views data. > > If you look at the BGP UPDATES archive sizes, you can see when something > happens, e.g.: > > http://archive.routeviews.org/bgpdata/2011.11/UPDATES/ > > Take a look at the size of the updates.20111107.1400.bz2 file and the 1415 > file. They are abnormally large compared to a normal period of time. > This shows there were a lot of updates out there being processed and a > reference to levels of instability. > > If you are not feeding route views or similar community projects, please > consider doing so. It helps paint the view for those doing analysis. > > - Jared > > > ------------------------------ > > Message: 5 > Date: Mon, 7 Nov 2011 16:33:15 +0100 > From: Pierre-Yves Maunier <[email protected]> > To: Tom Hill <[email protected]> > Cc: [email protected] > Subject: Re: TATA problems? > Message-ID: > <CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npnqzslctxuxeks9q...@mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > 2011/11/7 Tom Hill <[email protected]> > > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: > > > We seem to be having some problems with our tata links - first seen in > EU > > > about 45 minutes ago, now we're seeing problems in NA. I'm focused on > > DNS, > > > so I'm seeing a lot of timeouts/servfails, but our networking folks > are > > > talking about links dropping. > > > > > > Anyone else seeing oddness on the NA Internet right now? > > > > > > http://downrightnow.com/ confirms - something is up. > > > > There are widespread issues across the Internet; certain versions of > > Juniper firmware have core dumped after seeing a particular BGP 'UPDATE' > > message. > > > > (That's the running theory at least). > > > > It's affected multiple service providers, globally, not just those > > connected to TATA. > > > > Tom > > > > > > > On our side all our 10.3R2.11 core dumped which made all our interfaces > flapped. > I've been told 10.4R1.9 is affected too. > > -- > Pierre-Yves Maunier > > > ------------------------------ > > Message: 6 > Date: Mon, 7 Nov 2011 15:45:18 +0000 > From: Leigh Porter <[email protected]> > To: Pierre-Yves Maunier <[email protected]> > Cc: "[email protected]" <[email protected]> > Subject: Re: TATA problems? > Message-ID: <[email protected]> > Content-Type: text/plain; charset="us-ascii" > > > My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd > seemed stable. > > -- > Leigh > > > On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <[email protected]> wrote: > > > 2011/11/7 Tom Hill <[email protected]> > > > >> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: > >>> We seem to be having some problems with our tata links - first seen in > EU > >>> about 45 minutes ago, now we're seeing problems in NA. I'm focused on > >> DNS, > >>> so I'm seeing a lot of timeouts/servfails, but our networking folks > are > >>> talking about links dropping. > >>> > >>> Anyone else seeing oddness on the NA Internet right now? > >>> > >>> http://downrightnow.com/ confirms - something is up. > >> > >> There are widespread issues across the Internet; certain versions of > >> Juniper firmware have core dumped after seeing a particular BGP > 'UPDATE' > >> message. > >> > >> (That's the running theory at least). > >> > >> It's affected multiple service providers, globally, not just those > >> connected to TATA. > >> > >> Tom > >> > >> > >> > > On our side all our 10.3R2.11 core dumped which made all our interfaces > > flapped. > > I've been told 10.4R1.9 is affected too. > > > > -- > > Pierre-Yves Maunier > > > > ______________________________________________________________________ > > This email has been scanned by the MessageLabs Email Security System. > > For more information please visit http://www.messagelabs.com/email > > ______________________________________________________________________ > > ______________________________________________________________________ > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > ______________________________________________________________________ > > > > ------------------------------ > > Message: 7 > Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST) > From: Joe Greco <[email protected]> > To: [email protected] (Peter Pauly) > Cc: [email protected] > Subject: Re: Time Warner Telecom problems > Message-ID: <[email protected]> > Content-Type: text/plain; charset=us-ascii > > > Gizmodo is reporting problems at Time Warner Telecom .... we're > suffering > > from it too and calls to the NOC have not been answered so far... does > > anyone have any further information? > > > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us > > Actually, it looks to me like they mean "Time Warner", because that's > what they said. > > The company once known as "Time Warner Telecom" has always been a > different entity, and hasn't been known as that in some time, now > being called "twtelecom." Much of that company is what was once > known as inc.net, a Milwaukee area provider of the '90's. > > Time Warner Cable appears to have experienced an implosion this morning, > being out of service for about 11 minutes. During that time, packets > originating here in Milwaukee quickly died in Chicago; > > 1 76.46.192.1 8.320 ms 9.900 ms 7.974 ms > 2 24.160.230.32 7.967 ms 5.975 ms 8.479 ms > 3 24.160.229.132 8.471 ms 7.969 ms 10.991 ms > 4 24.160.229.193 9.972 ms 9.973 ms > 24.160.229.197 9.985 ms > 5 * * * > 6 * * * > > while packets destined for RR all seemed to be headed out to SJC, from > what I can tell. > > ... JG > -- > Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net > "We call it the 'one bite at the apple' rule. Give me one chance [and] > then I > won't contact you again." - Direct Marketing Ass'n position on e-mail > spam(CNN) > With 24 million small businesses in the US alone, that's way too many > apples. > > > > ------------------------------ > > Message: 8 > Date: Mon, 7 Nov 2011 07:55:33 -0800 > From: Kelly Kane <[email protected]> > To: Tim Vollebregt <[email protected]> > Cc: [email protected] > Subject: Re: TATA problems? > Message-ID: > <CAKfXD0zrH9TJkN=7doge1uaq8zxvszyvtjyflltbic6+umy...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <[email protected]> wrote: > > > > On #IX there are rumours about Junos version 10.3R2.11 being core dumped > and > > rebooted, which makes sense. > > Perhaps related to Juniper PSN-2011-08-327? Did the whole router > reboot, or just the service module? > > We saw one TATA session, and one Abovenet session flap. > > Kelly > > > > ------------------------------ > > Message: 9 > Date: Mon, 07 Nov 2011 10:02:13 -0600 > From: Blake Hudson <[email protected]> > To: [email protected] > Subject: Re: Time Warner Telecom problems > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > > Joe Greco wrote the following on 11/7/2011 9:54 AM: > >> Gizmodo is reporting problems at Time Warner Telecom .... we're > suffering > >> from it too and calls to the NOC have not been answered so far... does > >> anyone have any further information? > >> > >> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us > > Actually, it looks to me like they mean "Time Warner", because that's > > what they said. > > > > The company once known as "Time Warner Telecom" has always been a > > different entity, and hasn't been known as that in some time, now > > being called "twtelecom." Much of that company is what was once > > known as inc.net, a Milwaukee area provider of the '90's. > > > > Time Warner Cable appears to have experienced an implosion this morning, > > being out of service for about 11 minutes. During that time, packets > > originating here in Milwaukee quickly died in Chicago; > > Using the looking glass from TWtelecom, we saw 30-60min outage (roughly > 8:30AM to 9:30AM CST) between the Kansas City location and our own > server room in Kansas City. Other TWtelecom locations appeared to be > unaffected. Perhaps TWtelecom is served by Timewarner or shares > equipment in KC. Either way, none of our KC customers who were served > via TWtelecom or Timewarner were able to reach us. Packets would hit > Level 3 Communications and die in either direction at the border between > L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. > http://lglass.twtelecom.net/ > > > > ------------------------------ > > Message: 10 > Date: Mon, 7 Nov 2011 11:04:59 -0500 > From: "Thomas York" <[email protected]> > To: "'Blake Hudson'" <[email protected]>, <[email protected]> > Subject: RE: Time Warner Telecom problems > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8" > > FWIW, We saw issues here in Indianapolis between TWTC and L3 up until a > few minutes ago. > > --Thomas York > > -----Original Message----- > From: Blake Hudson [mailto:[email protected]] > Sent: Monday, November 07, 2011 11:02 AM > To: [email protected] > Subject: Re: Time Warner Telecom problems > > > Joe Greco wrote the following on 11/7/2011 9:54 AM: > >> Gizmodo is reporting problems at Time Warner Telecom .... we're > >> suffering from it too and calls to the NOC have not been answered so > >> far... does anyone have any further information? > >> > >> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us > > Actually, it looks to me like they mean "Time Warner", because that's > > what they said. > > > > The company once known as "Time Warner Telecom" has always been a > > different entity, and hasn't been known as that in some time, now > > being called "twtelecom." Much of that company is what was once known > > as inc.net, a Milwaukee area provider of the '90's. > > > > Time Warner Cable appears to have experienced an implosion this > > morning, being out of service for about 11 minutes. During that time, > > packets originating here in Milwaukee quickly died in Chicago; > > Using the looking glass from TWtelecom, we saw 30-60min outage (roughly > 8:30AM to 9:30AM CST) between the Kansas City location and our own server > room in Kansas City. Other TWtelecom locations appeared to be unaffected. > Perhaps TWtelecom is served by Timewarner or shares equipment in KC. > Either way, none of our KC customers who were served via TWtelecom or > Timewarner were able to reach us. Packets would hit Level 3 Communications > and die in either direction at the border between > L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. > http://lglass.twtelecom.net/ > > > > > > End of NANOG Digest, Vol 46, Issue 15 > *************************************
