Another Perspective - Kentik's View on the Facebook Outage

https://www.kentik.com/blog/facebooks-historic-outage-explained/?mkt_tok=ODY5LVBBRC04ODcAAAF_9diyPhC6WFsJNFpS4z2ggF-DWEc6FksyD3aW8B3am5tUvtTOJDYl2MIgMdsmmqOLTL-BpUugbQHAIXCT677LE0OxHM8Dy-mqRorJQXnAjg4 Mark.

RE: massive facebook outage presently

would have help here. Can you elaborate a bit please? Jean -Original Message- From: NANOG On Behalf Of Glenn Kelley Sent: October 4, 2021 8:18 PM To: nanog@nanog.org Subject: Re: massive facebook outage presently This is why you should have Routers that are Firmware Defaulted to y

Re: massive facebook outage presently

This is why you should have Routers that are Firmware Defaulted to your own network.  ALWAYS Be it Calix or even a Mikrotik which you have setup with Netboot - having these default to your own setup is REALLY a game changer. Without it - you are rolling trucks or at minimum taking heavy call

Re: massive facebook outage presently

I find it hilarious and ironic that their CTO had to use a competitor’s platform to confirm their outage. - dave > On Oct 4, 2021, at 16:45, Hank Nussbacher wrote: > > On 04/10/2021 22:05, Jason Kuehl wrote: > > BGP related: > https://twitter.com/SGgrc/status/1445116435731296256 > as also

Re: massive facebook outage presently

How come such a large operation does not have an out of bound access in case of emergencies ??? Somebody's getting fired ! -J On Mon, Oct 4, 2021 at 3:51 PM Aaron C. de Bruyn via NANOG wrote: > It looks like it might take a while according to a news reporter's tweet: > > "Was just on phone

Re: massive facebook outage presently

Seems unlikely that FB internal controls would allow such a backdoor ... "Never to get lost, is not living" - Rebecca Solnit Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl wrote: > On Mon, 4 Oct 2021 at 21:58,

Re: facebook outage

* jllee9...@gmail.com (John Lee) [Tue 05 Oct 2021, 01:06 CEST]: I was seeing NXDOMAIN errors, so I wonder if they had a DNS outage of some sort?? Were you using host(1)? Please don't, and use dig(1) instead. There were as far as I know at no point NXDOMAINs being returned, but due to the

Re: massive facebook outage presently

On Tue, 2021-10-05 at 06:31 +0200, Mark Tinka wrote: > Q: What is automation? > A: Breaking the network at scale. P J Plauger (I think) once defined a computer as a mechanism allowing the deletion of vast quantities of irreplaceable data using simple mnemonic commands. He defined a network as a

Re: massive facebook outage presently

On 10/4/21 20:48, Luke Guillory wrote: I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical

Re: massive facebook outage presently

On 10/4/21 22:33, Eric Kuhnke wrote: I am starting to see reports that in ISPs with very large numbers of residential users, customers are starting to press the factory-reset buttons on their home routers/modems/whatever, in an attempt to make Facebook work. This is resulting in much

Re: massive facebook outage presently

On 10/4/21 22:27, Łukasz Bromirski wrote: I bet FB tested the change on smaller scale and everything was fine, and only then started to roll this over wider network and at that point „something” broke. Or some bug needed a moment to start cascading issues around the infra. This is the

Re: massive facebook outage presently

On 10/4/21 22:23, Baldur Norddahl wrote: Not in such a primitive fashion no. But they could definitely have a secondary network that will continue to work even if something goes wrong with the primary. On IPv6, no less :-). On a serious note, I can't even imagine what it takes to run a

Re: facebook outage

Ok, I lied, I’m still awake. I got my first successful Facebook main page load at 23:13 UTC, for an overall duration of 8:33, or 513 minutes. Multiplied by three billion users, that’s 1.54 trillion person-minutes. That’s a tera-lapse! Have we had one of those before?

Re: massive facebook outage presently

On Mon, Oct 04, 2021 at 05:50:07PM -0400, b...@theworld.com wrote: > One might think in over six hours they could point facebook.com's DNS > somewhere else and put up a page with some info about the outage > there, that this would be a practiced firedrill. Perhaps, if they didn't decide to be

Re: facebook outage

I was seeing NXDOMAIN errors, so I wonder if they had a DNS outage of some sort?? On Mon, Oct 4, 2021 at 5:14 PM Bill Woodcock wrote: > They’re starting to pick themselves back up off the floor in the last two > or three minutes. A few answers getting out. I imagine it’ll take a while >

Re: massive facebook outage presently

* b...@theworld.com (b...@theworld.com) [Tue 05 Oct 2021, 00:01 CEST]: I only mean a single, simple information page like the "sorry we're working on it" I saw just before they came back. Which would subsequently receive the world's FB session cookies. -- Niels.

Re: facebook outage

> On Oct 5, 2021, at 12:16 AM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl >> wrote: >> >> >> >> man. 4. okt. 2021 23.33 skrev Bill Woodcock : >> >> >>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >>> >>> >>> On Oct 4, 2021, at 11:10 PM,

Re: facebook outage

> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl > wrote: > > > > man. 4. okt. 2021 23.33 skrev Bill Woodcock : > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back

Re: facebook outage (resolving)

Getting the odd message through, but DNS looks good via their Toronto, Canada pop % traceroute -q1 -I a.dns.facebook.com traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte packets  1  torix-core1-10G (67.43.129.248)  0.140 ms  2  facebook-a.ip4.torontointernetxchange.net

Re: facebook outage

Hopefully this will show people they can enjoy life and survive without it and that it will be just like myspace some day :) On Mon, Oct 4, 2021 at 5:31 PM Jeff Shultz wrote: > Now they just need to get the site itself back up. > > On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock wrote: > >> >> >>

Re: facebook outage

> On Oct 4, 2021, at 11:50 PM, Ryan Brooks wrote: > DNS was a victim in this outage, not the cause. You are absolutely correct. However, people who don’t have this problem avoid having this problem by not putting all their DNS eggs in one basket. And then forgetting where they put the

Re: facebook outage

On 10/4/21 2:41 PM, Baldur Norddahl wrote: man. 4. okt. 2021 23.33 skrev Bill Woodcock >: > On Oct 4, 2021, at 11:21 PM, Bill Woodcock mailto:wo...@pch.net>> wrote: > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:

Re: facebook outage

On Oct 4, 2021, at 5:30 PM, Bill Woodcock wrote: > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >>> >>> They’re starting to pick themselves back up off the floor in the last two >>> or three minutes. A few answers getting out. I

Re: facebook outage

On Mon, Oct 4, 2021 at 2:49 PM Baldur Norddahl wrote: > > > man. 4. okt. 2021 23.33 skrev Bill Woodcock : > >> >> >> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> > >> > >> > >> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >> >> >> >> They’re starting to pick themselves back

Re: massive facebook outage presently

One might think in over six hours they could point facebook.com's DNS somewhere else and put up a page with some info about the outage there, that this would be a practiced firedrill. Yeah yeah cache blah blah but it'd get around and at least would be coming from them. I'd imagine some mutual

Re: facebook outage

> On Oct 4, 2021, at 4:30 PM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> >> >> >>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >>> >>> They’re starting to pick themselves back up off the floor in the last two >>> or three minutes. A few

RE: massive facebook outage presently

Back and working by the looks.

Re: facebook outage

man. 4. okt. 2021 23.33 skrev Bill Woodcock : > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back up off the floor in the last > two or three minutes. A few answers

Re: massive facebook outage presently

17:35EDT: I'm suddenly getting: Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can. Go Back Facebook © 2020 · Help Center and: % host facebook.com facebook.com has address

Re: facebook outage

Can't wait to see the RFO! -Mike On Mon, Oct 4, 2021 at 2:35 PM Bill Woodcock wrote: > > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back up off the floor in the last

Re: facebook outage

> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >> >> They’re starting to pick themselves back up off the floor in the last two or >> three minutes. A few answers getting out. I imagine it’ll take a while >> before things

Re: facebook outage

Now they just need to get the site itself back up. On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock wrote: > > > > On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > > > > They’re starting to pick themselves back up off the floor in the last > two or three minutes. A few answers getting out. I

Re: facebook outage

> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > > They’re starting to pick themselves back up off the floor in the last two or > three minutes. A few answers getting out. I imagine it’ll take a while > before things stabilize, though. nd we’re back: WoodyNet-2:.ssh woody$ dig

Re: massive facebook outage presently

On 10/4/21 4:47 PM, Jean St-Laurent via NANOG wrote: > > dig @c.gtld-servers.net. facebook.com. NS > ;; > facebook.com. 172800 IN NS a.ns.facebook.com. > facebook.com. 172800 IN NS b.ns.facebook.com. > facebook.com. 172800 IN NS

Re: massive facebook outage presently

Starting to see some prefixes return on our peering session with them in Los Angeles but DNS still not resolving. -- Jay Hennigan - j...@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

Re: facebook outage

They’re starting to pick themselves back up off the floor in the last two or three minutes. A few answers getting out. I imagine it’ll take a while before things stabilize, though. -Bill signature.asc Description: Message signed with OpenPGP

Re: massive facebook outage presently

Not all employees are having this issue But it is curious why the records can be moved to the name servers that are still being advertised over the Internet still . On Mon, Oct 4, 2021 at 4:53 PM Aaron C. de Bruyn via NANOG wrote: > It looks like it might take a while according to a news

Re: massive facebook outage presently

Yes, We've seen that. On 10/4/21 4:33 PM, Eric Kuhnke wrote: I am starting to see reports that in ISPs with very large numbers of residential users, customers are starting to press the factory-reset buttons on their home routers/modems/whatever, in an attempt to make Facebook work. This is

Re: massive facebook outage presently

On 10/4/21 18:22, Eric Kuhnke wrote: Considering the massive impact of this it would be interesting to see some traffic graphs from ISPs that have PNIs with Facebook, or high volume peering sessions across an IX, showing traffic to FB falling off a cliff. Our PNI's and FNA's in Africa

RE: massive facebook outage presently

c.ns.facebook.com. facebook.com. 172800 IN NS d.ns.facebook.com. What happens if the NS aren’t back within 48 hours? Jean From: NANOG On Behalf Of Eric Kuhnke Sent: October 4, 2021 4:33 PM To: Jay Hennigan ; nanog@nanog.org list Subject: Re: massive facebook outage presently

Re: massive facebook outage presently

I’m not the only one who finds this timing suspicious, Starting with the publishers of 60 Minutes themselves :-) CBS: The outage comes the morning after "60 Minutes" aired an interview with a whistleblower who said Facebook is aware of how it amplifies hate, misinformation and unrest but

Re: massive facebook outage presently

It looks like it might take a while according to a news reporter's tweet: "Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors."

Re: massive facebook outage presently

Jay Hennigan wrote: On 10/4/21 09:27, Tom Beecher wrote: Curious if there is a malicious angle after the 60 Minutes story last night. Hanlon's razor. "Never attribute to malice that which is adequately explained by stupidity" Corollary - "But don't rule out malice." "Once is

Re: massive facebook outage presently

On 04/10/2021 22:05, Jason Kuehl wrote: BGP related: https://twitter.com/SGgrc/status/1445116435731296256 as also related by FB CTO: https://twitter.com/atoonk/status/1445121351707070468 -Hank https://twitter.com/disclosetv/status/1445100931947892736?s=20

Re: massive facebook outage presently

Subject: Re: massive facebook outage presently On 10/4/21 12:11, b...@theworld.com wrote: > > Although I believe it's generally true that if a company appears > prominently in the news it's liable to be attacked I assume because > the miscreants sit around thinking "hmm, who shall w

Re: massive facebook outage presently

I am starting to see reports that in ISPs with very large numbers of residential users, customers are starting to press the factory-reset buttons on their home routers/modems/whatever, in an attempt to make Facebook work. This is resulting in much heavier than normal first tier support volumes.

Re: massive facebook outage presently

If there isn't an undernetwork capable of being backdoored with the proper keys (I'd be shocked if there isn't - the big players have very good infra and DR people), I suspect there will be one soonish. It doesnt do much good to have DR plans and keys otherwise if you can't even get to the locks

Re: massive facebook outage presently

Dual homing won’t help you if your automation template will do „no router bgp X” and at this point session will terminate as suddenly advertisement will be withdrawn… It won’t you either if the change triggers some obscure bug in your BGP stack. I bet FB tested the change on smaller scale and

Re: massive facebook outage presently

On Mon, Oct 4, 2021 at 11:59 AM Jason Kuehl wrote: > I mean, you're an idiot if you post that public on the internet about > your own place of work. What do you think would happen? Nothing? He should > never of said anything, but now the Facebook hitman got him. > > Some of us have done that,

Re: massive facebook outage presently

Not in such a primitive fashion no. But they could definitely have a secondary network that will continue to work even if something goes wrong with the primary. On Mon, 4 Oct 2021 at 22:16, PJ Capelli wrote: > Seems unlikely that FB internal controls would allow such a backdoor ... > > "Never

Re: massive facebook outage presently

On Mon, 4 Oct 2021 at 21:58, Michael Thomas wrote: > > On 10/4/21 11:48 AM, Luke Guillory wrote: > > > I believe the original change was 'automatic' (as in configuration done > via a web interface). However, now that connection to the outside world is > down, remote access to those tools don't

Re: massive facebook outage presently

On 10/4/21 11:48 AM, Luke Guillory wrote: I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical

Re: massive facebook outage presently

Peace, On Mon, Oct 4, 2021, 10:17 PM Jean St-Laurent via NANOG wrote: > Maybe the key to solve this issue is in an email sent to > some_very_important_t...@facebook.com Yeah except MX records on facebook dot com aren't working either -- Töma

Re: massive facebook outage presently

On 10/4/21 12:11, b...@theworld.com wrote: Although I believe it's generally true that if a company appears prominently in the news it's liable to be attacked I assume because the miscreants sit around thinking "hmm, who shall we attack today oh look at that shiny headline!" I'd hate to ascribe

RE: massive facebook outage presently

Maybe the key to solve this issue is in an email sent to some_very_important_t...@facebook.com -Original Message- From: NANOG On Behalf Of tomocha Sent: October 4, 2021 2:32 PM To: nanog@nanog.org Subject: Re: massive facebook outage presently Hi Some of the DNS addresses

Re: massive facebook outage presently

Although I believe it's generally true that if a company appears prominently in the news it's liable to be attacked I assume because the miscreants sit around thinking "hmm, who shall we attack today oh look at that shiny headline!" I'd hate to ascribe any altruistic motivation w/o some evidence

Re: massive facebook outage presently

Hi Some of the DNS addresses are no longer prefix from AS32934. On 2021/10/05 1:23, Hugo Slabbert wrote: Looks like their auth DNS dropped out of the DFZ: https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20

Re: massive facebook outage presently

https://twitter.com/disclosetv/status/1445100931947892736?s=20 On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks wrote: > Didn't write that part of the automation script and that coder left... > > > I got a mail that Facebook was leaving NLIX. Maybe someone botched the > > script so they took down all

RE: massive facebook outage presently

Didn't write that part of the automation script and that coder left... > I got a mail that Facebook was leaving NLIX. Maybe someone botched the > script so they took down all BGP sessions instead of just NLIX and now > they can't access the equipment to put it back... :-)

Re: massive facebook outage presently

I mean, you're an idiot if you post that public on the internet about your own place of work. What do you think would happen? Nothing? He should never of said anything, but now the Facebook hitman got him. Facebook will have to send out a Reason For Outage with all the services it's effecting,

RE: massive facebook outage presently

/1445068309288951820 "About five minutes before Facebook's DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook's ASN." From: NANOG On Behalf Of Baldur Norddahl Sent: Monday, October 4, 2021 1:41 PM To: NANOG Subject: Re: massive facebook outage

Re: massive facebook outage presently

- On Oct 4, 2021, at 11:41 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: Hi, > I got a mail that Facebook was leaving NLIX. Maybe someone botched the script > so > they took down all BGP sessions instead of just NLIX and now they can't access > the equipment to put it back... :-)

Re: massive facebook outage presently

You laugh but that kind of sounds like what happened so far as oops we isolated prod and are scrambling on DR. There was someone supposedly live tweeting from their incident response for a bit before their account panic deleted. On Mon, Oct 4, 2021, 13:42 Baldur Norddahl wrote: > I got a mail

Re: massive facebook outage presently

I got a mail that Facebook was leaving NLIX. Maybe someone botched the script so they took down all BGP sessions instead of just NLIX and now they can't access the equipment to put it back... :-) man. 4. okt. 2021 20.31 skrev Billy Croan : > I know what this is. They forgot to update the

Re: massive facebook outage presently

I know what this is. They forgot to update the credit card on their godaddy account and the domain lapsed. I guess it will be facebook.info when they get it back online. The post mortem should be an interesting read. On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl wrote: > Looks like they

Re: massive facebook outage presently

Wishful thinking .. but hey, one is allowed to dream. /M On 04/10/2021 18:57, Jay Hennigan wrote: On 10/4/21 10:35, Mel Beckman wrote: Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose

Re: massive facebook outage presently

On 10/4/21 10:35, Mel Beckman wrote: Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose to profit from misinformation and political unrest through deliberate escalation of conflicts. Occam’s

Re: massive facebook outage presently

With Facebook down, how are people doing their vaccine research? It’s got to be more than just DNS. Jonathan Kalbfeld office: +1 310 317 7933 fax:+1 310 317 7901 home: +1 310 317 7909 mobile: +1 310 227 1662 ThoughtWave Technologies, Inc. Studio City, CA 91604

Re: massive facebook outage presently

Hi, Oops, this was not supposed to go to the list, apologies for the clutter. Thanks, Sabri - On Oct 4, 2021, at 10:46 AM, Sabri Berisha sa...@cluecentral.net wrote: > - On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com

Re: massive facebook outage presently

- On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com wrote: Hi Anne, > On a related note, what do you think the scene is like in FB HQ right now? > (shaking head) Very quiet, as their offices are still closed for all but essentials :) But, from experience I can tell

RE: massive facebook outage presently

Yes, embedded ISP CDN’s show a huge drop -Aaron From: NANOG On Behalf Of Eric Kuhnke Sent: Monday, October 4, 2021 11:22 AM To: George Herbert ; nanog@nanog.org list Subject: Re: massive facebook outage presently Considering the massive impact of this it would be interesting to see

Re: massive facebook outage presently

Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose to profit from misinformation and political unrest through deliberate escalation of conflicts. Occam’s razor says “When multiple causes are

Re: massive facebook outage presently

* m...@beckman.org (Mel Beckman) [Mon 04 Oct 2021, 18:23 CEST]: Here’s a screenshot: [cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001] Please don't do this on the NANOG list. -- Niels.

Re: massive facebook outage presently

uctivity is up 100% today. > > > > -richey > > > > *From: *NANOG on > behalf of Jason Kuehl > *Date: *Monday, October 4, 2021 at 12:45 PM > *To: *Mel Beckman > *Cc: *nanog@nanog.org list > *Subject: *Re: massive facebook outage presently > > Looks li

Re: massive facebook outage presently

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share Sent from my Verizon, Samsung Galaxy smartphoneGet Outlook for Android

Re: massive facebook outage presently

> On Oct 4, 2021, at 10:34 AM, Adrian Minta wrote: > > https://www.youtube.com/watch?v=gmk673M5BoA Because it's helpful to know what a link is about before clicking on it, Adrian's link goes to "The Onion Movie - Internet Down" On a related note, what do you think the scene is like in FB HQ

Re: massive facebook outage presently

t; DFZ. > > > > *From:* NANOG *On Behalf Of > *Dmitry > Sherman > *Sent:* Monday, October 4, 2021 12:10 PM > *To:* Eric Kuhnke ; nanog@nanog.org list < > nanog@nanog.org> > *Subject:* RE: massive facebook outage presently > > > > same problem in Israel > >

Re: massive facebook outage presently

On 10/4/21 09:27, Tom Beecher wrote: Curious if there is a malicious angle after the 60 Minutes story last night. Hanlon's razor. "Never attribute to malice that which is adequately explained by stupidity" Corollary - "But don't rule out malice." -- Jay Hennigan - j...@west.net Network

Re: massive facebook outage presently

In other news worker productivity is up 100% today. -richey From: NANOG on behalf of Jason Kuehl Date: Monday, October 4, 2021 at 12:45 PM To: Mel Beckman Cc: nanog@nanog.org list Subject: Re: massive facebook outage presently Looks like they run there own nameservers and I see the soa

Re: massive facebook outage presently

On Mon, Oct 4, 2021 at 9:47 AM Jason Kuehl wrote: > Looks like they run there own nameservers and I see the soa records are > even missing. > > On Mon, Oct 4, 2021, 12:23 PM Mel Beckman wrote: > >> Here’s a screenshot: >> >> >> >> -mel beckman >> >> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke

Re: massive facebook outage presently

I see the same. Still see those prefixes via direct peering, but DFZ doesn't have them. Their auths are not reachable even though I have routes for them. Some other weirdness I'm poking a bit on that doesn't seem like it could possibly be related to FB DNS though. On Mon, Oct 4, 2021 at 12:39 PM

Re: massive facebook outage presently

https://www.youtube.com/watch?v=gmk673M5BoA On 10/4/21 7:03 PM, Eric Kuhnke wrote: https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volume of

Re: massive facebook outage presently

hnke > *Sent:* Monday, 4 October 2021 19:03 > *To:* nanog@nanog.org list > *Subject:* massive facebook outage presently > > > > https://downdetector.com/status/facebook/ > > > > Normally not worth mentioning random $service having an outage here, but > this will u

RE: massive facebook outage presently

Looks bigger than DNS. Some people are saying they’ve disappeared off the DFZ. From: NANOG On Behalf Of Dmitry Sherman Sent: Monday, October 4, 2021 12:10 PM To: Eric Kuhnke ; nanog@nanog.org list Subject: RE: massive facebook outage presently same problem in Israel From: NANOG

Re: massive facebook outage presently

; *Sent:* Monday, 4 October 2021 19:03 > *To:* nanog@nanog.org list > *Subject:* massive facebook outage presently > > > > https://downdetector.com/status/facebook/ > > > > Normally not worth mentioning random $service having an outage here, but > this will u

Re: massive facebook outage presently

Looks like they run there own nameservers and I see the soa records are even missing. On Mon, Oct 4, 2021, 12:23 PM Mel Beckman wrote: > Here’s a screenshot: > > > > -mel beckman > > On Oct 4, 2021, at 9:06 AM, Eric Kuhnke wrote: > >  > https://downdetector.com/status/facebook/ > > Normally

Re: massive facebook outage presently

Not so much a DNS failure as an "oops, where did the routes to our auth servers go?" moment. or, in this case, several moments... ^_^;; Matt On Mon, Oct 4, 2021 at 9:07 AM Eric Kuhnke wrote: > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having

Re: massive facebook outage presently

Looks like their auth DNS dropped out of the DFZ: https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20 -- Hugo Slabbert On Mon, Oct 4, 2021 at 9:21 AM George Metz wrote: > Also impacting Instagram and, apparently,

Re: massive facebook outage presently

Considering the massive impact of this it would be interesting to see some traffic graphs from ISPs that have PNIs with Facebook, or high volume peering sessions across an IX, showing traffic to FB falling off a cliff. On Mon, Oct 4, 2021 at 12:16 PM George Herbert wrote: > And WhatsApp and

Re: massive facebook outage presently

And WhatsApp and Instagram. Twitter users nationwide agree anecdotally. What I’m getting is DNS failure. -George Sent from my iPhone > On Oct 4, 2021, at 9:07 AM, Eric Kuhnke wrote: > >  > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having

RE: massive facebook outage presently

same problem in Israel From: NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] On Behalf Of Eric Kuhnke Sent: Monday, 4 October 2021 19:03 To: nanog@nanog.org list Subject: massive facebook outage presently https://downdetector.com/status/facebook/ Normally not worth mentioning

Re: massive facebook outage presently

Here’s a screenshot: [cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001] -mel beckman On Oct 4, 2021, at 9:06 AM, Eric Kuhnke wrote:  https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large

Re: massive facebook outage presently

Also impacting Instagram and, apparently, WhatsApp. On Mon, Oct 4, 2021 at 12:05 PM Eric Kuhnke wrote: > > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having an outage here, but this > will undoubtedly generate a large volume of customer service

Re: massive facebook outage presently

> On 4 Oct 2021, at 18:03, Eric Kuhnke wrote: > > https://downdetector.com/status/facebook/ > > > Normally not worth mentioning random $service having an outage here, but this > will undoubtedly generate a large volume of customer service calls.

Re: massive facebook outage presently

yeah we are seeing same, appears that none of their NS are responding to dns queries affecting all their properties, whatsapp, facebook, instagram etc chris On Mon, Oct 4, 2021 at 12:06 PM Eric Kuhnke wrote: > https://downdetector.com/status/facebook/ > > Normally not worth mentioning

massive facebook outage presently

https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volume of customer service calls. Appears to be failure in DNS resolution.

Re: Facebook outage

On Sun, Apr 14, 2019 at 7:33 AM Siyuan Miao wrote: > Dear community, > > It seems that Facebook network is partially down. > I wouldn't say partially. And just a month and a day since their 10+ hour record-breaking outage. Of course FB will downplay it in their eventual RFO as "some users" had

Re: Facebook outage

That should have read FB not DB. On Sun, Apr 14, 2019, 07:41 Dovid Bender wrote: > It's all over Twitter. DB, What's app and Instagram are all having issues. > > On Sun, Apr 14, 2019, 07:32 Siyuan Miao wrote: > >> Dear community, >> >> It seems that Facebook network is partially down. >> >>

Re: Facebook outage

It's all over Twitter. DB, What's app and Instagram are all having issues. On Sun, Apr 14, 2019, 07:32 Siyuan Miao wrote: > Dear community, > > It seems that Facebook network is partially down. > > Here's a traceroute from some locations to fbcdn Hong Kong (157.240.15.37): > > Host > Loss%

Facebook outage

Dear community, It seems that Facebook network is partially down. Here's a traceroute from some locations to fbcdn Hong Kong (157.240.15.37): Host Loss% Snt Last Avg Best Wrst StDev 1. ??? 2. fra-in8-01sw.voxility.net 0.6% 1810.5 0.6 0.4 4.9 0.5 3.

  1   2   >