subject:"massive facebook outage presently"




On 10/4/21 20:48, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.




Q: What is automation?
A: Breaking the network at scale.

I suppose we shall know more in the coming days, but to me, it smells 
like a wide-scale automatic update roll-out that didn't go to plan. 
Isn't the first time a major content provider has suffered this; likely 
won't be the last.


In the end, the best thing for all of us is that it is a teaching 
moment, and we move the needle forward.


Mark.

Re: massive facebook outage presently





On 10/4/21 22:33, Eric Kuhnke wrote:

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first 
tier support volumes. The longer it stays down the worse this is going 
to get.


A relative could not understand how she received an e-mail from me 
yesterday about a family matter (via GMail) when the Internet was down :-).


Mark.

Re: massive facebook outage presently





On 10/4/21 22:27, Łukasz Bromirski wrote:



I bet FB tested the change on smaller scale and everything was fine, 
and only then started to roll this over wider network and at that 
point „something” broke. Or some bug needed a moment to start 
cascading issues around the infra.


This is the annoyance that is our world.

In the lab, it's always good.

Mark.

Re: massive facebook outage presently





On 10/4/21 22:23, Baldur Norddahl wrote:

Not in such a primitive fashion no. But they could definitely have a 
secondary network that will continue to work even if something goes 
wrong with the primary.


On IPv6, no less :-).

On a serious note, I can't even imagine what it takes to run a network 
of that scale, and plan for situations like this, if, indeed, inline 
access was lost.


Mark.

Re: massive facebook outage presently

2021-10-04 Thread Doug McIntyre

On Mon, Oct 04, 2021 at 05:50:07PM -0400, b...@theworld.com wrote:
> One might think in over six hours they could point facebook.com's DNS
> somewhere else and put up a page with some info about the outage
> there, that this would be a practiced firedrill.

Perhaps, if they didn't decide to be their own registrar as well
and run it all on the same network as it seems.

Maybe company divisions running different parts of infrastructure
should be self-hosted 100% on their own, different AS, different networking 
points, etc.
Or don't try to be everything top down all in the same company.

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker


* b...@theworld.com (b...@theworld.com) [Tue 05 Oct 2021, 00:01 CEST]:
I only mean a single, simple information page like the "sorry we're 
working on it" I saw just before they came back.


Which would subsequently receive the world's FB session cookies.


-- Niels.

Re: massive facebook outage presently

2021-10-04 Thread bzs



One might think in over six hours they could point facebook.com's DNS
somewhere else and put up a page with some info about the outage
there, that this would be a practiced firedrill.

Yeah yeah cache blah blah but it'd get around and at least would be
coming from them. I'd imagine some mutual pact with google's or
amazon's or microsoft's cloud server(s), for example, could handle it
for a few hours, and vice-versa.

I only mean a single, simple information page like the "sorry we're
working on it" I saw just before they came back.

Then again what else would one assume?

17:44 They seem to be back, more or less, slowly loading the usual
crazy sauce.

So out about 6 hours and some.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks

Back and working by the looks.

Re: massive facebook outage presently

2021-10-04 Thread bzs



17:35EDT: I'm suddenly getting:

 Sorry, something went wrong.
We're working on it and we'll get it fixed as soon as we can.

   Go Back

Facebook © 2020 · Help Center

and:

% host facebook.com

facebook.com has address 157.240.241.35
facebook.com has IPv6 address 2a03:2880:f112:182:face:b00c:0:25de
facebook.com mail is handled by 10 smtpin.vvv.facebook.com.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: massive facebook outage presently

2021-10-04 Thread Bryan Fields

On 10/4/21 4:47 PM, Jean St-Laurent via NANOG wrote:
>  
> dig @c.gtld-servers.net. facebook.com. NS
> ;;
> facebook.com.   172800  IN  NS  a.ns.facebook.com.
> facebook.com.   172800  IN  NS  b.ns.facebook.com.
> facebook.com.   172800  IN  NS  c.ns.facebook.com.
> facebook.com.   172800  IN  NS  d.ns.facebook.com.
> 
> What happens if the NS aren’t back within 48 hours?

Facebook can just update it at the registrar.

$ whois facebook.com
   Domain Name: FACEBOOK.COM
   Registry Domain ID: 2320948_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.registrarsafe.com
   Registrar URL: http://www.registrarsafe.com
   Updated Date: 2021-09-22T19:33:41Z
   Creation Date: 1997-03-29T05:00:00Z
   Registry Expiry Date: 2030-03-30T04:00:00Z
   Registrar: RegistrarSafe, LLC

$ dig @c.gtld-servers.net. registrarsafe.com. NS
;;
registrarsafe.com.  172800  IN  NS  a.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  b.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  c.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  d.ns.facebook.com.

crap

Vertical integration is a hell of a thing.
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Re: massive facebook outage presently

Starting to see some prefixes return on our peering session with them in 
Los Angeles but DNS still not resolving.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread None None

Not all employees are having this issue
But it is curious why the records can be moved to the name servers that are
still being advertised over the Internet still .


On Mon, Oct 4, 2021 at 4:53 PM Aaron C. de Bruyn via NANOG 
wrote:

> It looks like it might take a while according to a news reporter's tweet:
>
> "Was just on phone with someone who works for FB who described employees
> unable to enter buildings this morning to begin to evaluate extent of
> outage because their badges weren’t working to access doors."
>
> https://twitter.com/sheeraf/status/1445099150316503057?s=20
>
>
> -A
>
> On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:
>
>> I am starting to see reports that in ISPs with very large numbers of
>> residential users, customers are starting to press the factory-reset
>> buttons on their home routers/modems/whatever, in an attempt to make
>> Facebook work. This is resulting in much heavier than normal first tier
>> support volumes. The longer it stays down the worse this is going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>>
>>> On 10/4/21 12:11, b...@theworld.com wrote:
>>> >
>>> > Although I believe it's generally true that if a company appears
>>> > prominently in the news it's liable to be attacked I assume because
>>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>>> > motivation w/o some evidence like even a credible twitter post (maybe
>>> > they posted that on FB? :-)
>>>
>>> I personally believe that the outage was caused by human error and not
>>> something malicious. Time will tell.
>>>
>>> However, if you missed the 60 Minutes piece, it was a former employee
>>> who spoke out with some rather powerful observations. I don't think that
>>> this type of worldwide outage was caused by an outside bad actor. It is
>>> certainly within the realm of possibility that it was an inside job.
>>>
>>> In other news:
>>>
>>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>>
>>> --
>>> Jay Hennigan - j...@west.net
>>> Network Engineering - CCIE #7880
>>> 503 897-8550 - WB6RDV
>>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Matt Hoppes

Yes,
We've seen that.

On 10/4/21 4:33 PM, Eric Kuhnke wrote:
I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first tier 
support volumes. The longer it stays down the worse this is going to get.

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > wrote:

On 10/4/21 12:11, b...@theworld.com  wrote:
 >
 > Although I believe it's generally true that if a company appears
 > prominently in the news it's liable to be attacked I assume because
 > the miscreants sit around thinking "hmm, who shall we attack today oh
 > look at that shiny headline!" I'd hate to ascribe any altruistic
 > motivation w/o some evidence like even a credible twitter post (maybe
 > they posted that on FB? :-)

I personally believe that the outage was caused by human error and not
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee
who spoke out with some rather powerful observations. I don't think
that
this type of worldwide outage was caused by an outside bad actor. It is
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net 

Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently





On 10/4/21 18:22, Eric Kuhnke wrote:

Considering the massive impact of this it would be interesting to see 
some traffic graphs from ISPs that have PNIs with Facebook, or high 
volume peering sessions across an IX, showing traffic to FB falling 
off a cliff.


Our PNI's and FNA's in Africa have dipped.

But PNI's in Europe have nearly doubled, even though we aren't 
delivering any Facebook services of note to our eyeballs.


Strangest thing...

Mark.

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG

The glue records for the NS are set at 48 hours.

dig @c.gtld-servers.net. facebook.com. NS

;;

facebook.com.   172800  IN  NS  a.ns.facebook.com.

facebook.com.   172800  IN  NS  b.ns.facebook.com.

facebook.com.   172800  IN  NS  c.ns.facebook.com.

facebook.com.   172800  IN  NS  d.ns.facebook.com.

What happens if the NS aren’t back within 48 hours? 

Jean

From: NANOG  On Behalf Of Eric Kuhnke
Sent: October 4, 2021 4:33 PM
To: Jay Hennigan ; nanog@nanog.org list 
Subject: Re: massive facebook outage presently

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset buttons on 
their home routers/modems/whatever, in an attempt to make Facebook work. This 
is resulting in much heavier than normal first tier support volumes. The longer 
it stays down the worse this is going to get.

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan mailto:j...@west.net> > wrote:

On 10/4/21 12:11, b...@theworld.com <mailto:b...@theworld.com>  wrote:
> 
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net <mailto:j...@west.net> 
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman

I’m not the only one who finds this timing suspicious, Starting with the
publishers of 60 Minutes themselves :-)

CBS:

The outage comes the morning after "60 Minutes" aired an interview with a
whistleblower who said Facebook is aware of how it amplifies hate,
misinformation and unrest but claimed the company hides what it knows.

https://www.cbsnews.com/news/facebook-instagram-whatsapp-down-2021-10-04/

https://abcnews.go.com/Technology/facebook-instagram-users-us/story?id=80397437

https://www.cnbc.com/2021/10/04/facebook-shares-drop-5percent-after-site-outage-and-whistleblower-interview.html

https://www.insidenova.com/headlines/facebook-instagram-down-after-60-minutes-whistleblower-report/article_29977530-2531-11ec-b5ae-73311c46edfb.html

https://adage.com/article/digital-marketing-ad-tech-news/what-facebook-telling-advertisers-about-60-minutes-whistleblower/2370346

-mel beckman

On Oct 4, 2021, at 1:36 PM, Blake Dunlap wrote:

If there isn't an undernetwork capable of being backdoored with the proper keys
(I'd be shocked if there isn't - the big players have very good infra and DR
people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't even
get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for their
sres right now and am drinking one in their honor. I just want to know what an
October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl
mailto:baldur.nordd...@gmail.com>> wrote:
Not in such a primitive fashion no. But they could definitely have a secondary
network that will continue to work even if something goes wrong with the
primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli
mailto:pjcape...@pm.me>> wrote:
Seems unlikely that FB internal controls would allow such a backdoor ...

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl
mailto:baldur.nordd...@gmail.com>> wrote:

On Mon, 4 Oct 2021 at 21:58, Michael Thomas
mailto:m...@mtcc.com>> wrote:

On 10/4/21 11:48 AM, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration done via a
web interface). However, now that connection to the outside world is down,
remote access to those tools don't exist anymore, so the emergency procedure is
to gain physical access to the peering routers and do all the configuration
locally.

Assuming that this is what actually happened, what should fb have done
different (beyond the obvious of not screwing up the immediate issue)? This
seems like it's a single point of failure. Should all of the BGP speakers have
been dual homed or something like that? Or should they not have been mixing ops
and production networks? Sorry if this sounds dumb.

Facebook is a huge network. It is doubtful that what is going on is this
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using
someone else's network. Nowadays this could be a simple 4/5G router with a VPN,
to a terminal server that allows the operator to configure the equipment
through the monitor port even when the config is completely destroyed.

Regards,

Baldur

Re: massive facebook outage presently

2021-10-04 Thread Aaron C. de Bruyn via NANOG

It looks like it might take a while according to a news reporter's tweet:

"Was just on phone with someone who works for FB who described employees
unable to enter buildings this morning to begin to evaluate extent of
outage because their badges weren’t working to access doors."

https://twitter.com/sheeraf/status/1445099150316503057?s=20

-A

On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:

> I am starting to see reports that in ISPs with very large numbers of
> residential users, customers are starting to press the factory-reset
> buttons on their home routers/modems/whatever, in an attempt to make
> Facebook work. This is resulting in much heavier than normal first tier
> support volumes. The longer it stays down the worse this is going to get.
>
>
>
> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>
>> On 10/4/21 12:11, b...@theworld.com wrote:
>> >
>> > Although I believe it's generally true that if a company appears
>> > prominently in the news it's liable to be attacked I assume because
>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>> > motivation w/o some evidence like even a credible twitter post (maybe
>> > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former employee
>> who spoke out with some rather powerful observations. I don't think that
>> this type of worldwide outage was caused by an outside bad actor. It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> --
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>
>

Re: massive facebook outage presently

2021-10-04 Thread Miles Fidelman


Jay Hennigan wrote:

On 10/4/21 09:27, Tom Beecher wrote:
Curious if there is a malicious angle after the 60 Minutes story last 
night.


Hanlon's razor.

"Never attribute to malice that which is adequately explained by 
stupidity"


Corollary - "But don't rule out malice."


"Once is happenstance. Twice is coincidence. The third*time it's*enemy 
action'.”


--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: massive facebook outage presently

2021-10-04 Thread Hank Nussbacher


On 04/10/2021 22:05, Jason Kuehl wrote:

BGP related:
https://twitter.com/SGgrc/status/1445116435731296256
as also related by FB CTO:
https://twitter.com/atoonk/status/1445121351707070468

-Hank

https://twitter.com/disclosetv/status/1445100931947892736?s=20 



On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks > wrote:


Didn't write that part of the automation script and that coder left...

 > I got a mail that Facebook was leaving NLIX. Maybe someone
botched the
 > script so they took down all BGP sessions instead of just NLIX
and now
 > they can't access the equipment to put it back... :-)




--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg

No evidence that it’s intentional but…..   What’s going to be the big headline 
tonight?A Facebook whistleblower or a global outage that kept everyone from 
arguing all day long?

-richey

From: NANOG  on behalf of 
Jay Hennigan 
Date: Monday, October 4, 2021 at 3:30 PM
To: nanog@nanog.org 
Subject: Re: massive facebook outage presently
On 10/4/21 12:11, b...@theworld.com wrote:
>
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee
who spoke out with some rather powerful observations. I don't think that
this type of worldwide outage was caused by an outside bad actor. It is
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Eric Kuhnke

I am starting to see reports that in ISPs with very large numbers of
residential users, customers are starting to press the factory-reset
buttons on their home routers/modems/whatever, in an attempt to make
Facebook work. This is resulting in much heavier than normal first tier
support volumes. The longer it stays down the worse this is going to get.



On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:

> On 10/4/21 12:11, b...@theworld.com wrote:
> >
> > Although I believe it's generally true that if a company appears
> > prominently in the news it's liable to be attacked I assume because
> > the miscreants sit around thinking "hmm, who shall we attack today oh
> > look at that shiny headline!" I'd hate to ascribe any altruistic
> > motivation w/o some evidence like even a credible twitter post (maybe
> > they posted that on FB? :-)
>
> I personally believe that the outage was caused by human error and not
> something malicious. Time will tell.
>
> However, if you missed the 60 Minutes piece, it was a former employee
> who spoke out with some rather powerful observations. I don't think that
> this type of worldwide outage was caused by an outside bad actor. It is
> certainly within the realm of possibility that it was an inside job.
>
> In other news:
>
> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap

If there isn't an undernetwork capable of being backdoored with the proper
keys (I'd be shocked if there isn't - the big players have very good infra
and DR people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't
even get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for
their sres right now and am drinking one in their honor. I just want to
know what an October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl 
wrote:

> Not in such a primitive fashion no. But they could definitely have a
> secondary network that will continue to work even if something goes wrong
> with the primary.
>
> On Mon, 4 Oct 2021 at 22:16, PJ Capelli  wrote:
>
>> Seems unlikely that FB internal controls would allow such a backdoor ...
>>
>> "Never to get lost, is not living" - Rebecca Solnit
>>
>> Sent with ProtonMail  Secure Email.
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
>> baldur.nordd...@gmail.com> wrote:
>>
>>
>>
>> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
>>
>>>
>>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>>
>>>
>>> I believe the original change was 'automatic' (as in configuration done
>>> via a web interface). However, now that connection to the outside world is
>>> down, remote access to those tools don't exist anymore, so the emergency
>>> procedure is to gain physical access to the peering routers and do all the
>>> configuration locally.
>>>
>>> Assuming that this is what actually happened, what should fb have done
>>> different (beyond the obvious of not screwing up the immediate issue)? This
>>> seems like it's a single point of failure. Should all of the BGP speakers
>>> have been dual homed or something like that? Or should they not have been
>>> mixing ops and production networks? Sorry if this sounds dumb.
>>>
>>
>> Facebook is a huge network. It is doubtful that what is going on is this
>> simple. So I will make no guesses to what Facebook is or should be doing.
>>
>> However the traditional way for us small timers is to have a backdoor
>> using someone else's network. Nowadays this could be a simple 4/5G router
>> with a VPN, to a terminal server that allows the operator to configure the
>> equipment through the monitor port even when the config is completely
>> destroyed.
>>
>> Regards,
>>
>> Baldur
>>
>>
>>
>>
>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Łukasz Bromirski


Dual homing won’t help you if your automation template will do „no router bgp 
X” and at this point session will terminate as suddenly advertisement will be 
withdrawn…

It won’t you either if the change triggers some obscure bug in your BGP stack.

I bet FB tested the change on smaller scale and everything was fine, and only 
then started to roll this over wider network and at that point „something” 
broke. Or some bug needed a moment to start cascading issues around the infra.

-- 
./

> On 4 Oct 2021, at 22:00, Michael Thomas  wrote:
> 
> 
> 
> 
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>> 
>> I believe the original change was 'automatic' (as in configuration done via 
>> a web interface). However, now that connection to the outside world is down, 
>> remote access to those tools don't exist anymore, so the emergency procedure 
>> is to gain physical access to the peering routers and do all the 
>> configuration locally.
> Assuming that this is what actually happened, what should fb have done 
> different (beyond the obvious of not screwing up the immediate issue)? This 
> seems like it's a single point of failure. Should all of the BGP speakers 
> have been dual homed or something like that? Or should they not have been 
> mixing ops and production networks? Sorry if this sounds dumb.
> 
> Mike

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach

On Mon, Oct 4, 2021 at 11:59 AM Jason Kuehl  wrote:

> I mean, you're an idiot if you post that public on the internet about
> your own place of work. What do you think would happen? Nothing? He should
> never of said anything, but now the Facebook hitman got him.
>
>
Some of us have done that, and survived[0].

But I would be the first to admit I've led a
very charmed life in that regard.   ^_^;

Matt

[0]
https://www.computerworld.com/article/2529621/networking-glitch-knocks-yahoo-offline-for-some.html

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl

Not in such a primitive fashion no. But they could definitely have a
secondary network that will continue to work even if something goes wrong
with the primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli  wrote:

> Seems unlikely that FB internal controls would allow such a backdoor ...
>
> "Never to get lost, is not living" - Rebecca Solnit
>
> Sent with ProtonMail  Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>
>
>
> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
>
>>
>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>
>>
>> I believe the original change was 'automatic' (as in configuration done
>> via a web interface). However, now that connection to the outside world is
>> down, remote access to those tools don't exist anymore, so the emergency
>> procedure is to gain physical access to the peering routers and do all the
>> configuration locally.
>>
>> Assuming that this is what actually happened, what should fb have done
>> different (beyond the obvious of not screwing up the immediate issue)? This
>> seems like it's a single point of failure. Should all of the BGP speakers
>> have been dual homed or something like that? Or should they not have been
>> mixing ops and production networks? Sorry if this sounds dumb.
>>
>
> Facebook is a huge network. It is doubtful that what is going on is this
> simple. So I will make no guesses to what Facebook is or should be doing.
>
> However the traditional way for us small timers is to have a backdoor
> using someone else's network. Nowadays this could be a simple 4/5G router
> with a VPN, to a terminal server that allows the operator to configure the
> equipment through the monitor port even when the config is completely
> destroyed.
>
> Regards,
>
> Baldur
>
>
>
>
>
>

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl

On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:

>
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>
>
> I believe the original change was 'automatic' (as in configuration done
> via a web interface). However, now that connection to the outside world is
> down, remote access to those tools don't exist anymore, so the emergency
> procedure is to gain physical access to the peering routers and do all the
> configuration locally.
>
> Assuming that this is what actually happened, what should fb have done
> different (beyond the obvious of not screwing up the immediate issue)? This
> seems like it's a single point of failure. Should all of the BGP speakers
> have been dual homed or something like that? Or should they not have been
> mixing ops and production networks? Sorry if this sounds dumb.
>

Facebook is a huge network. It is doubtful that what is going on is this
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using
someone else's network. Nowadays this could be a simple 4/5G router with a
VPN, to a terminal server that allows the operator to configure the
equipment through the monitor port even when the config is completely
destroyed.

Regards,

Baldur

Re: massive facebook outage presently

2021-10-04 Thread Michael Thomas



On 10/4/21 11:48 AM, Luke Guillory wrote:


I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.


Assuming that this is what actually happened, what should fb have done 
different (beyond the obvious of not screwing up the immediate issue)? 
This seems like it's a single point of failure. Should all of the BGP 
speakers have been dual homed or something like that? Or should they not 
have been mixing ops and production networks? Sorry if this sounds dumb.


Mike

Re: massive facebook outage presently

2021-10-04 Thread Töma Gavrichenkov

Peace,

On Mon, Oct 4, 2021, 10:17 PM Jean St-Laurent via NANOG 
wrote:

> Maybe the key to solve this issue is in an email sent to
> some_very_important_t...@facebook.com


Yeah except MX records on facebook dot com aren't working either

--
Töma

Re: massive facebook outage presently


On 10/4/21 12:11, b...@theworld.com wrote:


Although I believe it's generally true that if a company appears
prominently in the news it's liable to be attacked I assume because
the miscreants sit around thinking "hmm, who shall we attack today oh
look at that shiny headline!" I'd hate to ascribe any altruistic
motivation w/o some evidence like even a credible twitter post (maybe
they posted that on FB? :-)


I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.


However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.


In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG

Maybe the key to solve this issue is in an email sent to 
some_very_important_t...@facebook.com

-Original Message-
From: NANOG  On Behalf Of tomocha
Sent: October 4, 2021 2:32 PM
To: nanog@nanog.org
Subject: Re: massive facebook outage presently

Hi

Some of the DNS addresses are no longer prefix from AS32934.



On 2021/10/05 1:23, Hugo Slabbert wrote:
> Looks like their auth DNS dropped out of the DFZ:
> 
> https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
> https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20
> 
>

Re: massive facebook outage presently

2021-10-04 Thread bzs

Although I believe it's generally true that if a company appears
prominently in the news it's liable to be attacked I assume because
the miscreants sit around thinking "hmm, who shall we attack today oh
look at that shiny headline!" I'd hate to ascribe any altruistic
motivation w/o some evidence like even a credible twitter post (maybe
they posted that on FB? :-)

I wonder how often the US White House, congress, et al are attacked
since they appear in almost any top headline list often centering on
something someone out there really doesn't like?

On October 4, 2021 at 17:35 m...@beckman.org (Mel Beckman) wrote:
 > Suspiciously, this comes the morning after Facebook whistleblower Frances
 > Haugen disclosed on 60 Minutes that Facebook's own research shows that it 
 > chose
 > to profit from misinformation and political unrest through deliberate
 > escalation of conflicts. Occam’s razor says “When multiple causes are
 > plausible, and CBS 60 Minutes is one of them, go with 60 Minutes.” :)
 > 
 >  -mel 
 > 
 > 
 > On Oct 4, 2021, at 10:30 AM, Michael Spears  wrote:
 > 
 > 
 > https://www.reddit.com/r/sysadmin/comments/q181fv/
 > looks_like_facebook_is_down/?utm_medium=android_app_source=share
 > 
 > Sent from my Verizon, Samsung Galaxy smartphone
 > Get Outlook for Android
 > 

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: massive facebook outage presently

2021-10-04 Thread tomocha


Hi

Some of the DNS addresses are no longer prefix from AS32934.



On 2021/10/05 1:23, Hugo Slabbert wrote:

Looks like their auth DNS dropped out of the DFZ:

https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20

Re: massive facebook outage presently

https://twitter.com/disclosetv/status/1445100931947892736?s=20

On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks  wrote:

> Didn't write that part of the automation script and that coder left...
>
> > I got a mail that Facebook was leaving NLIX. Maybe someone botched the
> > script so they took down all BGP sessions instead of just NLIX and now
> > they can't access the equipment to put it back... :-)
>
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks

Didn't write that part of the automation script and that coder left...

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the 
> script so they took down all BGP sessions instead of just NLIX and now 
> they can't access the equipment to put it back... :-)

Re: massive facebook outage presently

I mean, you're an idiot if you post that public on the internet about
your own place of work. What do you think would happen? Nothing? He should
never of said anything, but now the Facebook hitman got him.

Facebook will have to send out a Reason For Outage with all the services
it's effecting, like login

On Mon, Oct 4, 2021 at 2:46 PM Blake Dunlap  wrote:

> You laugh but that kind of sounds like what happened so far as oops we
> isolated prod and are scrambling on DR. There was someone supposedly live
> tweeting from their incident response for a bit before their account panic
> deleted.
>
> On Mon, Oct 4, 2021, 13:42 Baldur Norddahl 
> wrote:
>
>> I got a mail that Facebook was leaving NLIX. Maybe someone botched the
>> script so they took down all BGP sessions instead of just NLIX and now they
>> can't access the equipment to put it back... :-)
>>
>>
>> man. 4. okt. 2021 20.31 skrev Billy Croan :
>>
>>> I know what this is.  They forgot to update the credit card on their
>>> godaddy account and the domain lapsed.  I guess it will be facebook.info
>>> when they get it back online.  The post mortem should be an interesting
>>> read.
>>>
>>> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
>>> wrote:
>>>
 Looks like they run there own nameservers and I see the soa records are
 even missing.

 On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:

> Here’s a screenshot:
>
>
>
>  -mel beckman
>
> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>
> 
> https://downdetector.com/status/facebook/
>
> Normally not worth mentioning random $service having an outage here,
> but this will undoubtedly generate a large volume of customer service
> calls.
>
> Appears to be failure in DNS resolution.
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: massive facebook outage presently

2021-10-04 Thread Luke Guillory

From what I believe was a FB employee on Reddit, account now deleted it seems.


As many of you know, DNS for FB services has been affected and this is likely a 
symptom of the actual issue, and that's that BGP peering with Facebook peering 
routers has gone down, very likely due to a configuration change that went into 
effect shortly before the outages happened (started roughly 1540 UTC).



There are people now trying to gain access to the peering routers to implement 
fixes, but the people with physical access is separate from the people with 
knowledge of how to actually authenticate to the systems and people who know 
what to actually do, so there is now a logistical challenge with getting all 
that knowledge unified.



Part of this is also due to lower staffing in data centers due to pandemic 
measures.



I believe the original change was 'automatic' (as in configuration done via a 
web interface). However, now that connection to the outside world is down, 
remote access to those tools don't exist anymore, so the emergency procedure is 
to gain physical access to the peering routers and do all the configuration 
locally.



https://twitter.com/jgrahamc/status/1445068309288951820 "About five minutes 
before Facebook's DNS stopped working we saw a large number of BGP changes 
(mostly route withdrawals) for Facebook's ASN."




From: NANOG  On Behalf Of 
Baldur Norddahl
Sent: Monday, October 4, 2021 1:41 PM
To: NANOG 
Subject: Re: massive facebook outage presently

*External Email: Use Caution*
I got a mail that Facebook was leaving NLIX. Maybe someone botched the script 
so they took down all BGP sessions instead of just NLIX and now they can't 
access the equipment to put it back... :-)


man. 4. okt. 2021 20.31 skrev Billy Croan 
mailto:bcr...@unrealservers.net>>:
I know what this is.  They forgot to update the credit card on their 
godaddy account and the domain lapsed.  I guess it will be 
facebook.info<https://link.edgepilot.com/s/7bad5051/Di9CwLEB1E6iB_KlhyWtZA?u=http://facebook.info/>
 when they get it back online.  The post mortem should be an interesting read.

On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
mailto:jason.w.ku...@gmail.com>> wrote:
Looks like they run there own nameservers and I see the soa records are even 
missing.

On Mon, Oct 4, 2021, 12:23 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Here’s a screenshot:

 -mel beckman


On Oct 4, 2021, at 9:06 AM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

https://link.edgepilot.com/s/3926b9ff/bTkszib6zUmYbE_rZxhltQ?u=https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls.

Appears to be failure in DNS resolution.



Links contained in this email have been replaced. If you click on a link in the 
email above, the link will be analyzed for known threats. If a known threat is 
found, you will not be able to proceed to the destination. If suspicious 
content is detected, you will see a warning.

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

- On Oct 4, 2021, at 11:41 AM, Baldur Norddahl baldur.nordd...@gmail.com 
wrote:

Hi,

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the script 
> so
> they took down all BGP sessions instead of just NLIX and now they can't access
> the equipment to put it back... :-)

That's an interesting theory. Once upon a time I saw a billion dollar company 
suffer
a significant outage after enabling EVPN on a remote site. Took down the entire
backbone, including access to the site.

Thanks,

Sabri

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap

You laugh but that kind of sounds like what happened so far as oops we
isolated prod and are scrambling on DR. There was someone supposedly live
tweeting from their incident response for a bit before their account panic
deleted.

On Mon, Oct 4, 2021, 13:42 Baldur Norddahl 
wrote:

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the
> script so they took down all BGP sessions instead of just NLIX and now they
> can't access the equipment to put it back... :-)
>
>
> man. 4. okt. 2021 20.31 skrev Billy Croan :
>
>> I know what this is.  They forgot to update the credit card on their
>> godaddy account and the domain lapsed.  I guess it will be facebook.info
>> when they get it back online.  The post mortem should be an interesting
>> read.
>>
>> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
>> wrote:
>>
>>> Looks like they run there own nameservers and I see the soa records are
>>> even missing.
>>>
>>> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>>>
 Here’s a screenshot:

  -mel beckman

 On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:

 https://downdetector.com/status/facebook/

 Normally not worth mentioning random $service having an outage here,
 but this will undoubtedly generate a large volume of customer service
 calls.

 Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl

I got a mail that Facebook was leaving NLIX. Maybe someone botched the
script so they took down all BGP sessions instead of just NLIX and now they
can't access the equipment to put it back... :-)


man. 4. okt. 2021 20.31 skrev Billy Croan :

> I know what this is.  They forgot to update the credit card on their
> godaddy account and the domain lapsed.  I guess it will be facebook.info
> when they get it back online.  The post mortem should be an interesting
> read.
>
> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
> wrote:
>
>> Looks like they run there own nameservers and I see the soa records are
>> even missing.
>>
>> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>>
>>> Here’s a screenshot:
>>>
>>>
>>>
>>>  -mel beckman
>>>
>>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>>
>>> 
>>> https://downdetector.com/status/facebook/
>>>
>>> Normally not worth mentioning random $service having an outage here, but
>>> this will undoubtedly generate a large volume of customer service calls.
>>>
>>> Appears to be failure in DNS resolution.
>>>
>>>

Re: massive facebook outage presently

2021-10-04 Thread Billy Croan

I know what this is.  They forgot to update the credit card on their
godaddy account and the domain lapsed.  I guess it will be facebook.info
when they get it back online.  The post mortem should be an interesting
read.

On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl  wrote:

> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
>> Here’s a screenshot:
>>
>>
>>
>>  -mel beckman
>>
>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>
>> 
>> https://downdetector.com/status/facebook/
>>
>> Normally not worth mentioning random $service having an outage here, but
>> this will undoubtedly generate a large volume of customer service calls.
>>
>> Appears to be failure in DNS resolution.
>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Martin List-Petersen




Wishful thinking .. but hey, one is allowed to dream.

/M

On 04/10/2021 18:57, Jay Hennigan wrote:

On 10/4/21 10:35, Mel Beckman wrote:
Suspiciously, this comes the morning after Facebook whistleblower 
Frances Haugen disclosed on 60 Minutes that Facebook's own research 
shows that it chose to profit from misinformation and political 
unrest through deliberate escalation of conflicts. Occam’s razor says 
“When multiple causes are plausible, and CBS 60 Minutes is one of 
them, go with 60 Minutes.” :)


It could just be that after the 60 Minutes interview they've shut 
things down in order to divert all power to the shredders.



--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair - http://www.airwire.ie - Phone: 
091 395000
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in 
Ireland No. 508961


--
This email has been checked for viruses by AVG.
https://www.avg.com

Re: massive facebook outage presently


On 10/4/21 10:35, Mel Beckman wrote:
Suspiciously, this comes the morning after Facebook whistleblower 
Frances Haugen disclosed on 60 Minutes that Facebook's own research 
shows that it chose to profit from misinformation and political unrest 
through deliberate escalation of conflicts. Occam’s razor says “When 
multiple causes are plausible, and CBS 60 Minutes is one of them, go 
with 60 Minutes.” :)


It could just be that after the 60 Minutes interview they've shut things 
down in order to divert all power to the shredders.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Jonathan Kalbfeld via NANOG

With Facebook down, how are people doing their vaccine research?

It’s got to be more than just DNS.  

Jonathan Kalbfeld

office: +1 310 317 7933 
fax:+1 310 317 7901 
home:   +1 310 317 7909 
mobile: +1 310 227 1662 

ThoughtWave Technologies, Inc.
Studio City, CA 91604
https://thoughtwave.com 

View our network at 
https://bgp.he.net/AS54380 

+1 213 984 1000

> On Oct 4, 2021, at 10:12 AM, Michael Spears  wrote:
> 
> https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share
> 
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

Hi,

Oops, this was not supposed to go to the list, apologies for the clutter.

Thanks, 

Sabri

- On Oct 4, 2021, at 10:46 AM, Sabri Berisha sa...@cluecentral.net wrote:

> - On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

- On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com 
wrote:

Hi Anne,

> On a related note, what do you think the scene is like in FB HQ right now?
> (shaking head)

Very quiet, as their offices are still closed for all but essentials :)

But, from experience I can tell you how that works. I assume Facebook works in a
similar manner as some of my previous employers. This assumption comes from the
fact that quite a number of my previous colleagues now work at Facebook in 
similar
roles.

First there is the question of detecting the outage. Obviously, Facebook will 
have
a monitoring/SRE team that continuously monitors 1000s of metrics. They observe
a number of metrics go down, and start to investigate. Most likely they will 
have
some sort of overall technical lead (let's call this the Technical Duty 
Officer),
that is responsible for the whole thing. Once the SRE team figured out where the
problem lies, they will alert the TDO. TDO will then hit that big red button and
send out alerts to the appropriate teams to jump on a bridge (let's call that 
the
Technical Crisis Bridge), to fix the issue. 

If done right, whomever was on call for that team will take the lead and 
interface
with adjoining teams, and other team members who are available to help out. 
Looking
at how long this outage lasts, there must be either something very broken, or 
they're
having trouble rolling back a change which was expected to not have impact.

Once the issue is fixed, the TDO will write a report and submit it to the 
Problem
Management group. This group will now contact the teams deemed responsible for 
the
outage. This team will no have an opportunity to explain themselves during a 
post-
mortem. Depending on the scale of the outage, the post-mortem can be a 10 minute
call on a bridge with a Problem Management manager, or in the hot seat during a
60 minute meeting with a bunch of execs.

I've been in that hot seat a few times. Not the most pleasurable experience. 
Perhaps
it's time for a new career :)

Thanks,

Sabri

RE: massive facebook outage presently

2021-10-04 Thread aaron1

Yes, embedded ISP CDN’s show a huge drop

 

-Aaron

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: Monday, October 4, 2021 11:22 AM
To: George Herbert ; nanog@nanog.org list 

Subject: Re: massive facebook outage presently

 

Considering the massive impact of this it would be interesting to see some 
traffic graphs from ISPs that have PNIs with Facebook, or high volume peering 
sessions across an IX, showing traffic to FB falling off a cliff. 

 

On Mon, Oct 4, 2021 at 12:16 PM George Herbert mailto:george.herb...@gmail.com> > wrote:

And WhatsApp and Instagram.  Twitter users nationwide agree anecdotally.

 

What I’m getting is DNS failure. 

 

-George 

Sent from my iPhone





On Oct 4, 2021, at 9:07 AM, Eric Kuhnke mailto:eric.kuh...@gmail.com> > wrote:



https://downdetector.com/status/facebook/

 

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls. 

 

Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman

Suspiciously, this comes the morning after Facebook whistleblower Frances 
Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose 
to profit from misinformation and political unrest through deliberate 
escalation of conflicts. Occam’s razor says “When multiple causes are 
plausible, and CBS 60 Minutes is one of them, go with 60 Minutes.” :)

 -mel

On Oct 4, 2021, at 10:30 AM, Michael Spears  wrote:


https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker


* m...@beckman.org (Mel Beckman) [Mon 04 Oct 2021, 18:23 CEST]:

Here’s a screenshot:

[cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001]


Please don't do this on the NANOG list.


-- Niels.

Re: massive facebook outage presently

2021-10-04 Thread Casey Russell via NANOG

>>  In other news worker productivity is up 100% today.

For everyone except IT workers.

Although, I suppose if you're just counting the number of tickets they can
quickly clear by sending out a "It's the internet, not us".  You could
count that as increased productivity.

Sincerely,
Casey Russell
Network Engineer
<http://www.kanren.net>
785-856-9809
2029 Becker Drive, Suite 282
Lawrence, Kansas 66047
XSEDE Campus Champion
Certified Software Carpentry Instructor
need support? 

On Mon, Oct 4, 2021 at 12:14 PM richey goldberg 
wrote:

> In other news worker productivity is up 100% today.
>
>
>
> -richey
>
>
>
> *From: *NANOG  on
> behalf of Jason Kuehl 
> *Date: *Monday, October 4, 2021 at 12:45 PM
> *To: *Mel Beckman 
> *Cc: *nanog@nanog.org list 
> *Subject: *Re: massive facebook outage presently
>
> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
>
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
> Here’s a screenshot:
>
>
>
> *Error! Filename not specified.*
>
>  -mel beckman
>
>
>
> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>
> 
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>
>

Re: massive facebook outage presently

2021-10-04 Thread Michael Spears

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share
Sent from my Verizon, Samsung Galaxy smartphoneGet Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Anne P. Mitchell, Esq.

> On Oct 4, 2021, at 10:34 AM, Adrian Minta  wrote:
> 
> https://www.youtube.com/watch?v=gmk673M5BoA

Because it's helpful to know what a link is about before clicking on it, 
Adrian's link goes to "The Onion Movie - Internet Down"

On a related note, what do you think the scene is like in FB HQ right now?  
(shaking head)

Anne

--
Outsource your email deliverability headaches to us, and get to the inbox!
www.GetToTheInbox.com

Anne P. Mitchell,  Esq.
CEO Get to the Inbox - We get you into the inbox!
Author: The Email Deliverability Handbook
Author: Section 6 of the Federal Email Marketing Law (CAN-SPAM)
Email Marketing Deliverability and Best Practices Expert
Board of Directors, Denver Internet Exchange
Chair Emeritus, Asilomar Microcomputer Workshop
Former Counsel: MAPS Anti-Spam Blacklist
Location: Boulder, Colorado

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher

They haven't completely dropped off, but the big subnets certainly have.
I'm only seeing 20-odd /24s from them via the DFZ, but everything larger
still directly.

On Mon, Oct 4, 2021 at 12:55 PM  wrote:

> Looks bigger than DNS. Some people are saying they’ve disappeared off the
> DFZ.
>
>
>
> *From:* NANOG  *On Behalf Of 
> *Dmitry
> Sherman
> *Sent:* Monday, October 4, 2021 12:10 PM
> *To:* Eric Kuhnke ; nanog@nanog.org list <
> nanog@nanog.org>
> *Subject:* RE: massive facebook outage presently
>
>
>
> same problem in Israel
>
>
>
>
>
> *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org
> ] *On Behalf Of *Eric Kuhnke
> *Sent:* Monday, 4 October 2021 19:03
> *To:* nanog@nanog.org list 
> *Subject:* massive facebook outage presently
>
>
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>

Re: massive facebook outage presently


On 10/4/21 09:27, Tom Beecher wrote:
Curious if there is a malicious angle after the 60 Minutes story last 
night.


Hanlon's razor.

"Never attribute to malice that which is adequately explained by stupidity"

Corollary - "But don't rule out malice."


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg

In other news worker productivity is up 100% today.

-richey

From: NANOG  on behalf of 
Jason Kuehl 
Date: Monday, October 4, 2021 at 12:45 PM
To: Mel Beckman 
Cc: nanog@nanog.org list 
Subject: Re: massive facebook outage presently
Looks like they run there own nameservers and I see the soa records are even 
missing.

On Mon, Oct 4, 2021, 12:23 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Here’s a screenshot:

Error! Filename not specified.
 -mel beckman

On Oct 4, 2021, at 9:06 AM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls.

Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach

On Mon, Oct 4, 2021 at 9:47 AM Jason Kuehl  wrote:

> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
>> Here’s a screenshot:
>>
>>
>>
>>  -mel beckman
>>
>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>
>> 
>> https://downdetector.com/status/facebook/
>>
>> Normally not worth mentioning random $service having an outage here, but
>> this will undoubtedly generate a large volume of customer service calls.
>>
>> Appears to be failure in DNS resolution.
>>
>>
If you check your BGP routing tables, you'll probably
find that it's not so much the SOA records that are
missing, as it is the prefixes to reach the DNS servers
entirely.

I suspect the DNS entries on the servers themselves
may look fine from inside facebook, leading to a slower
diagnostic and repair, as it's only from the outside world
the missing routing entries in the global table make the
problem so painfully visible.

Having the DNS team frantically checking their servers
may slow the resolution down, if it is indeed a BGP failure
rather than a DNS server failure situation, as it seems to
appear at the moment.  ^_^;

Matt

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher

I see the same. Still see those prefixes via direct peering, but DFZ
doesn't have them. Their auths are not reachable even though I have routes
for them.

Some other weirdness I'm poking a bit on that doesn't seem like it could
possibly be related to FB DNS though.

On Mon, Oct 4, 2021 at 12:39 PM Hugo Slabbert  wrote:

> Looks like their auth DNS dropped out of the DFZ:
>
> https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
> https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20
>
>
> --
> Hugo Slabbert
>
>
> On Mon, Oct 4, 2021 at 9:21 AM George Metz  wrote:
>
>> Also impacting Instagram and, apparently, WhatsApp.
>>
>> On Mon, Oct 4, 2021 at 12:05 PM Eric Kuhnke 
>> wrote:
>> >
>> > https://downdetector.com/status/facebook/
>> >
>> > Normally not worth mentioning random $service having an outage here,
>> but this will undoubtedly generate a large volume of customer service calls.
>> >
>> > Appears to be failure in DNS resolution.
>> >
>>
>

Re: massive facebook outage presently

2021-10-04 Thread Adrian Minta


https://www.youtube.com/watch?v=gmk673M5BoA

On 10/4/21 7:03 PM, Eric Kuhnke wrote:
https://downdetector.com/status/facebook/ 



Normally not worth mentioning random $service having an outage here, 
but this will undoubtedly generate a large volume of customer service 
calls.


Appears to be failure in DNS resolution.


--
Best regards,
Adrian Minta

Re: massive facebook outage presently

Yeah it looks like there dns servers are just dead. I can't get an response
from them.

On Mon, Oct 4, 2021, 12:26 PM Dmitry Sherman  wrote:

> same problem in Israel
>
>
>
>
>
> *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] *On
> Behalf Of *Eric Kuhnke
> *Sent:* Monday, 4 October 2021 19:03
> *To:* nanog@nanog.org list 
> *Subject:* massive facebook outage presently
>
>
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>

RE: massive facebook outage presently

2021-10-04 Thread michael

Looks bigger than DNS. Some people are saying they’ve disappeared off the DFZ.

From: NANOG  On Behalf Of Dmitry 
Sherman
Sent: Monday, October 4, 2021 12:10 PM
To: Eric Kuhnke ; nanog@nanog.org list 
Subject: RE: massive facebook outage presently

same problem in Israel

From: NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] On Behalf Of 
Eric Kuhnke
Sent: Monday, 4 October 2021 19:03
To: nanog@nanog.org <mailto:nanog@nanog.org>  list mailto:nanog@nanog.org> >
Subject: massive facebook outage presently

https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls. 

Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher

Curious if there is a malicious angle after the 60 Minutes story last
night.

On Mon, Oct 4, 2021 at 12:26 PM Dmitry Sherman  wrote:

> same problem in Israel
>
>
>
>
>
> *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] *On
> Behalf Of *Eric Kuhnke
> *Sent:* Monday, 4 October 2021 19:03
> *To:* nanog@nanog.org list 
> *Subject:* massive facebook outage presently
>
>
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>

Re: massive facebook outage presently