Probably better luck finding a 10mm wrench On Tue, Oct 5, 2021, 4:26 PM Nate Burke <[email protected]> wrote:
> No, they just had to find a laptop that had a serial port on it. They > probably only are issued company tablets. > > On 10/5/2021 4:22 PM, Steve Jones wrote: > > Id like to think some lowly janitor was picked up from his vacation by a > helicopter right from his john boat on the lake, whisked to an airport and > loaded onto a leer jet to california, then police escorted to FB HQ to pull > out his big ol key ring and open the door > > On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF <[email protected]> > wrote: > >> Probably more like some kind of automation with a bug in it. I got a >> kick out of FB actually saying that this caused some of their people to be >> physically locked out of data center equipment they needed to access to fix >> it. I don’t think I would have ever admitted that. >> >> *From:* Nate Burke >> *Sent:* Tuesday, October 5, 2021 2:45 PM >> *To:* AnimalFarm Microwave Users Group >> *Subject:* Re: [AFMUG] facebook outage >> >> I think the more plausible explanation is that Zuckerberg was in a bar >> with some guys he was trying to impress saying something like 'I do SO run >> Facebook' and then whips out his phone and instead of deleting their >> profile, he presses the 'delete all' button by accident. >> >> I've been at some small businesses where they can't change an Ethernet >> port speed without a 3 person peer review of the config update, I have >> trouble believing that one guy mistyped something. >> >> On 10/5/2021 12:17 PM, Steve Jones wrote: >> >> ill bet whatever their process was, its not what it is and delayed >> commits/fallbacks are instituted forthwith. Not by the guy who committed >> this set of changes though, hes at the bottom of a river now >> >> On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson <[email protected]> >> wrote: >> >>> I wonder if they did a "commit confirmed 30”? If not, would that have >>> avoided the whole thing? >>> >>> >>> On Oct 5, 2021, at 11:08 AM, Carl Peterson <[email protected]> >>> wrote: >>> >>> "an engineer put in a config change on a BGP router that propagated to >>> the other BGP routers and cut off access" >>> >>> Sounds like something I would do which is why I generally insist on >>> having a remote tech physically in the building with a console cable if I >>> don't have OOB console access when changing BGP on my edge. >>> >>> On Tue, Oct 5, 2021 at 11:34 AM Robert <[email protected]> wrote: >>> >>>> FB has posted that an engineer put in a config change on a BGP router >>>> that propagated to the other BGP routers and cut off access. Sounds like >>>> it took a while to track down the config change. I do know someone who is >>>> intimate with that group but he's not talking... Drinking, but not >>>> talking... >>>> >>>> On 10/5/21 9:17 AM, Bill Prince wrote: >>>> >>>> I see it differently. In the beginning cloudflare thought something was >>>> wrong with their systems. It doesn't explain what happened inside facebook, >>>> probably because only facebook knows what went wrong. That's not >>>> cloudflare's problem. We may never know what happened at facebook probably >>>> because it's very embarrassing to facebook. >>>> >>>> >>>> >>>> bp >>>> <part15sbs{at}gmail{dot}com> >>>> >>>> On 10/5/2021 9:01 AM, Robert wrote: >>>> >>>> I really don't like that cloudflare posted this, as it doesn't do >>>> anything to tell why. It's not a root cause, it's an effect and cloudflare >>>> looks like they are just trying to say how smart they are. >>>> >>>> On 10/5/21 8:40 AM, Bill Prince wrote: >>>> >>>> >>>> >>>> https://blog.cloudflare.com/october-2021-facebook-outage/ >>>> >>>> >>>> >>>> bp >>>> <part15sbs{at}gmail{dot}com> >>>> >>>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote: >>>> >>>> Sign him up for a SNAP card. >>>> >>>> Sent from my iPhone >>>> >>>> On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:[email protected] >>>> <[email protected]> wrote: >>>> >>>> >>>> >>>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth >>>> yesterday. Now he's only worth around $121 billion. >>>> >>>> >>>> >>>> bp >>>> <part15sbs{at}gmail{dot}com> >>>> >>>> On 10/5/2021 6:52 AM, [email protected] wrote: >>>> >>>> I am very curious what it was that took 6 hours to figure out or undo. >>>> >>>> >>>> >>>> …..but I wouldn’t judge too harshly. When any contraption gets big and >>>> complex it’s easy to overlook one seemingly small thing. >>>> >>>> >>>> >>>> Remember this XKCD? >>>> >>>> >>>> <image001.png> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From:* AF mailto:[email protected] <[email protected]> *On >>>> Behalf Of *Steve Jones >>>> *Sent:* Monday, October 04, 2021 6:51 PM >>>> *To:* AnimalFarm Microwave Users Group mailto:[email protected] >>>> <[email protected]> >>>> *Subject:* Re: [AFMUG] facebook outage >>>> >>>> >>>> >>>> so it turns out FB murdered the announcements for the DNS server >>>> prefixes, including the registrar. >>>> >>>> >>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ >>>> >>>> >>>> >>>> This really seems like an elementary design flaw. Even a moron like me >>>> knows not to poke the DNS subnets all at once, and we probably get way less >>>> than 1k daily lookups across all the domains we host DNS for. Ill bet >>>> nobody involved in this makes less than 80k a year. facebook lost 4% stock >>>> value today attributed to this on top of the almost percent attributed to >>>> the "whistleblower". I get nervous when I lose the company a number 10 >>>> wrench. >>>> >>>> >>>> >>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller < >>>> [email protected]> wrote: >>>> >>>> >>>> >>>> it is alive here again >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> >>>> *From:* CBB - Jay Fuller >>>> >>>> *To:* AnimalFarm Microwave Users Group >>>> >>>> *Sent:* Monday, October 4, 2021 4:44 PM >>>> >>>> *Subject:* Re: [AFMUG] facebook outage >>>> >>>> >>>> >>>> >>>> >>>> yup. resolving here. but nothing coming up yet. >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> >>>> *From:* Mike Hammett >>>> >>>> *To:* AnimalFarm Microwave Users Group >>>> >>>> *Sent:* Monday, October 4, 2021 4:36 PM >>>> >>>> *Subject:* Re: [AFMUG] facebook outage >>>> >>>> >>>> >>>> They're back... ish. >>>> >>>> >>>> >>>> ----- >>>> Mike Hammett >>>> Intelligent Computing Solutions <http://www.ics-il.com/> >>>> <https://www.facebook.com/ICSIL> >>>> <~WRD0002.jpg> <https://www.facebook.com/ICSIL> >>>> <~WRD0002.jpg> >>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> >>>> <~WRD0002.jpg> >>>> <https://www.linkedin.com/company/intelligent-computing-solutions> >>>> <~WRD0002.jpg> >>>> <https://twitter.com/ICSIL> >>>> Midwest Internet Exchange <http://www.midwest-ix.com/> >>>> <~WRD0002.jpg> >>>> <https://www.facebook.com/mdwestix> >>>> <~WRD0002.jpg> >>>> <https://www.linkedin.com/company/midwest-internet-exchange> >>>> <~WRD0002.jpg> >>>> <https://twitter.com/mdwestix> >>>> The Brothers WISP <http://www.thebrotherswisp.com/> >>>> <~WRD0002.jpg> >>>> <https://www.facebook.com/thebrotherswisp> >>>> <~WRD0002.jpg> >>>> >>>> >>>> >>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> >>>> ------------------------------ >>>> >>>> *From: *"CBB - Jay Fuller" <[email protected]> >>>> *To: *[email protected] >>>> *Sent: *Monday, October 4, 2021 1:13:01 PM >>>> *Subject: *[AFMUG] facebook outage >>>> >>>> >>>> >>>> anyone else feeling alone and isolated without facebook? I need my >>>> safe space (no, not myspace) back. >>>> >>>> >>>> >>>> babaahahah. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>>> >>>> ------------------------------ >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>>> ------------------------------ >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> AF mailing list >>>> [email protected] >>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>> >>> >>> >>> -- >>> >>> Carl Peterson >>> >>> *PORT NETWORKS* >>> >>> 401 E Pratt St, Ste 2553 >>> >>> Baltimore, MD 21202 >>> >>> (410) 637-3707 >>> -- >>> AF mailing list >>> [email protected] >>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>> >>> >>> -- >>> AF mailing list >>> [email protected] >>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>> >> >> >> >> ------------------------------ >> -- >> AF mailing list >> [email protected] >> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >> -- >> AF mailing list >> [email protected] >> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >> > > > > -- > AF mailing list > [email protected] > http://af.afmug.com/mailman/listinfo/af_af.afmug.com >
-- AF mailing list [email protected] http://af.afmug.com/mailman/listinfo/af_af.afmug.com
