If 30 minutes would have been long enough. I wonder how long it really takes for BGP changes to propagate.
Having contracted for FB before, let me tell you their management style is horrible. Conference calls to plan conference calls. I got my ass chewed for being honest on a call. Can’t cause shame or embarrassment. They had an engineering company that royally screwed things up but I had to act like they were the best company on the planet during calls. Lots of infighting between different FB departments. Lots of silos. The managers are imperious. In any event, I am betting they are in the process of convening a NASA type challenger explosion panel to have a year’s worth of conference calls to make even more bureaucracy to prevent something like this from happening again. Back in olden times, when everyone that owned a router trusted everyone else, someone announced they could take all IPs over BGP. I think they were in Champaign Urbana. I was in Quincy Ill at the time. In any event it brought our part of the internet to a screeching halt. I think there was an RFCs created that very day and solutions created in just a few more days over that one. From: Cassidy B. Larson Sent: Tuesday, October 5, 2021 11:12 AM To: AnimalFarm Microwave Users Group Subject: Re: [AFMUG] facebook outage I wonder if they did a "commit confirmed 30”? If not, would that have avoided the whole thing? On Oct 5, 2021, at 11:08 AM, Carl Peterson <[email protected]> wrote: "an engineer put in a config change on a BGP router that propagated to the other BGP routers and cut off access" Sounds like something I would do which is why I generally insist on having a remote tech physically in the building with a console cable if I don't have OOB console access when changing BGP on my edge. On Tue, Oct 5, 2021 at 11:34 AM Robert <[email protected]> wrote: FB has posted that an engineer put in a config change on a BGP router that propagated to the other BGP routers and cut off access. Sounds like it took a while to track down the config change. I do know someone who is intimate with that group but he's not talking... Drinking, but not talking... On 10/5/21 9:17 AM, Bill Prince wrote: I see it differently. In the beginning cloudflare thought something was wrong with their systems. It doesn't explain what happened inside facebook, probably because only facebook knows what went wrong. That's not cloudflare's problem. We may never know what happened at facebook probably because it's very embarrassing to facebook. bp <part15sbs{at}gmail{dot}com>On 10/5/2021 9:01 AM, Robert wrote: I really don't like that cloudflare posted this, as it doesn't do anything to tell why. It's not a root cause, it's an effect and cloudflare looks like they are just trying to say how smart they are. On 10/5/21 8:40 AM, Bill Prince wrote: https://blog.cloudflare.com/october-2021-facebook-outage/ bp <part15sbs{at}gmail{dot}com>On 10/5/2021 7:39 AM, Chuck McCown via AF wrote: Sign him up for a SNAP card. Sent from my iPhone On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:[email protected] wrote: Heard the Zuch got around $6 or $7 billion trimmed from his net worth yesterday. Now he's only worth around $121 billion. bp <part15sbs{at}gmail{dot}com>On 10/5/2021 6:52 AM, [email protected] wrote: I am very curious what it was that took 6 hours to figure out or undo. …..but I wouldn’t judge too harshly. When any contraption gets big and complex it’s easy to overlook one seemingly small thing. Remember this XKCD? <image001.png> From: AF mailto:[email protected] On Behalf Of Steve Jones Sent: Monday, October 04, 2021 6:51 PM To: AnimalFarm Microwave Users Group mailto:[email protected] Subject: Re: [AFMUG] facebook outage so it turns out FB murdered the announcements for the DNS server prefixes, including the registrar. https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ This really seems like an elementary design flaw. Even a moron like me knows not to poke the DNS subnets all at once, and we probably get way less than 1k daily lookups across all the domains we host DNS for. Ill bet nobody involved in this makes less than 80k a year. facebook lost 4% stock value today attributed to this on top of the almost percent attributed to the "whistleblower". I get nervous when I lose the company a number 10 wrench. On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <[email protected]> wrote: it is alive here again ----- Original Message ----- From: CBB - Jay Fuller To: AnimalFarm Microwave Users Group Sent: Monday, October 4, 2021 4:44 PM Subject: Re: [AFMUG] facebook outage yup. resolving here. but nothing coming up yet. ----- Original Message ----- From: Mike Hammett To: AnimalFarm Microwave Users Group Sent: Monday, October 4, 2021 4:36 PM Subject: Re: [AFMUG] facebook outage They're back... ish. ----- Mike Hammett Intelligent Computing Solutions <~WRD0002.jpg> <~WRD0002.jpg> <~WRD0002.jpg> <~WRD0002.jpg> Midwest Internet Exchange <~WRD0002.jpg> <~WRD0002.jpg> <~WRD0002.jpg> The Brothers WISP <~WRD0002.jpg> <~WRD0002.jpg> ---------------------------------------------------------- From: "CBB - Jay Fuller" <[email protected]> To: [email protected] Sent: Monday, October 4, 2021 1:13:01 PM Subject: [AFMUG] facebook outage anyone else feeling alone and isolated without facebook? I need my safe space (no, not myspace) back. babaahahah. -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com ---------------------------------------------------------- -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com ------------------------------------------------------------ -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- Carl Peterson PORT NETWORKS 401 E Pratt St, Ste 2553 Baltimore, MD 21202 (410) 637-3707 -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com -------------------------------------------------------------------------------- -- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com
