Re: max-prefix and platform tcam limits: they are things
On Fri, 5 Oct 2012, jim deleskie wrote: Just ask yourself how many times you have seen a Godaddy IP/NOC person post anything to NANOG or to any other technical forum? -Hank Yes that math would work, but if your device can't handle 1x Internet routing and your running without some serious max-prefix/filters it says even more about your IP eng team then I'd be willing to comment on. -jim On Fri, Oct 5, 2012 at 9:17 PM, wrote: On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said: But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this? If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x
Re: max-prefix and platform tcam limits: they are things
In case you missed it On Oct 5, 2012, at 7:05 PM, jim deleskie wrote: > I know that I should know better then comment on networks others then > my own, ( and I know to never comment on my own publicly :) ) > > > But here goes, 210x the size of normal really? 210% I'd have a hard > time believing. Did anyone else anywhere see a route leak equal to > larger then the entire Internet that day, anywhere else that could of > caused this? > > I won't even get into max-prefix and how we've managed this long with > someone people still not setting them. > > > -jim > On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela wrote: >> Submitted without comment: >> http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/ >> >> -Tk > >
Re: max-prefix and platform tcam limits: they are things
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/5/2012 8:17 PM, valdis.kletni...@vt.edu wrote: > On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said: > >> But here goes, 210x the size of normal really? 210% I'd have a >> hard time believing. Did anyone else anywhere see a route leak >> equal to larger then the entire Internet that day, anywhere else >> that could of caused this? > > If the device was only expecting 2K or so internal routes, getting > hit with the 440K routes in the DFZ would be 210x > On outages GoDaddy provided a tiny bit more information. [quote] Obviously the explanation of the incident had to be consumed by the general public, however we encountered an unknown bug that was found which started the domino effect. Aside from this group, that level of detail wouldn't be understood by a majority of the recipients. With that said, please feel free to take this off list with Jason or Myself. Mike Dob Manager, Network Engineering [/quote] No information has been provided on what sort of "unknown bug" this was. A bug in code that GoDaddy wrote? A bug in their route servers or router OS, which others may also use and might want to be aware of? - -DMM -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQb32GAAoJECp6zT7OFmGa5wYIAIWp9vUwS/5zM73cAXlUrpwR 5U3XuUn3fasq8JyuNFhe99aDhkQY+i5tQEFhhhB60dVfWcyVGYsO1Ny0FMXupYfE Ely29vxutWHMDxX39XTvmmtNkeSsZ2cOtkqF14If+43/CccrDwDDiC06YoSyxb/x JEjWMhcthcw8rbndzF3P+bRCerdyxPpeQLzNy+l0/SbjobsLwzDA28CPW2kL82Bh 67dgqdXiMVFARC8rc91bYAoJ+NtkLs/GwYSbgXdNCk5dGrZvOk1rVWzaKxBrNV8T rldU43GRzeq2bJAKo0fx17/HE4j9qlfeBIW+bihwgkMpzw8p3kRS9S0WU4cGxGM= =1nls -END PGP SIGNATURE-
Re: max-prefix and platform tcam limits: they are things
On 10/5/12 5:05 PM, jim deleskie wrote: I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) ) But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this? it's pretty easy to inadvertently leak a copy of the internet from one vrf to another and effectively install two copies of the internet routes in your fib... There are plently of cases where you might to that or something similar on purpose, which is all good and well if you have 2million route fib capacity but less awesome if you have 512K route capacity linecards at this point. if you get those routes from a private peer on some non-internet-vrf well that might imply that your filter policy needs some tuning. I won't even get into max-prefix and how we've managed this long with someone people still not setting them. -jim On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela wrote: Submitted without comment: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/ -Tk
Re: max-prefix and platform tcam limits: they are things
On Fri, 5 Oct 2012, jim deleskie wrote: I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) ) But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this? Is it plausible that Godaddy's internal network only normally has a few thousand BGP routes? 210 x a few thousand would run most modern gear out of FIB space. The "my DNS is broken, are we really being DDoS'd on udp/53 at the same time?" thing, I've seen, and I can imagine it being very confusing to someone seeing it for the first time. -- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: max-prefix and platform tcam limits: they are things
Yes that math would work, but if your device can't handle 1x Internet routing and your running without some serious max-prefix/filters it says even more about your IP eng team then I'd be willing to comment on. -jim On Fri, Oct 5, 2012 at 9:17 PM, wrote: > On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said: > >> But here goes, 210x the size of normal really? 210% I'd have a hard >> time believing. Did anyone else anywhere see a route leak equal to >> larger then the entire Internet that day, anywhere else that could of >> caused this? > > If the device was only expecting 2K or so internal routes, getting hit with > the 440K routes in the DFZ would be 210x
Re: max-prefix and platform tcam limits: they are things
On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said: > But here goes, 210x the size of normal really? 210% I'd have a hard > time believing. Did anyone else anywhere see a route leak equal to > larger then the entire Internet that day, anywhere else that could of > caused this? If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x pgpov9A0dTXMJ.pgp Description: PGP signature
Re: max-prefix and platform tcam limits: they are things
I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) ) But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this? I won't even get into max-prefix and how we've managed this long with someone people still not setting them. -jim On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela wrote: > Submitted without comment: > http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/ > > -Tk >
