date:20211004




On 10/4/21 20:48, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.




Q: What is automation?
A: Breaking the network at scale.

I suppose we shall know more in the coming days, but to me, it smells 
like a wide-scale automatic update roll-out that didn't go to plan. 
Isn't the first time a major content provider has suffered this; likely 
won't be the last.


In the end, the best thing for all of us is that it is a teaching 
moment, and we move the needle forward.


Mark.

Re: massive facebook outage presently





On 10/4/21 22:33, Eric Kuhnke wrote:

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first 
tier support volumes. The longer it stays down the worse this is going 
to get.


A relative could not understand how she received an e-mail from me 
yesterday about a family matter (via GMail) when the Internet was down :-).


Mark.

Re: massive facebook outage presently





On 10/4/21 22:27, Łukasz Bromirski wrote:



I bet FB tested the change on smaller scale and everything was fine, 
and only then started to roll this over wider network and at that 
point „something” broke. Or some bug needed a moment to start 
cascading issues around the infra.


This is the annoyance that is our world.

In the lab, it's always good.

Mark.

Re: massive facebook outage presently





On 10/4/21 22:23, Baldur Norddahl wrote:

Not in such a primitive fashion no. But they could definitely have a 
secondary network that will continue to work even if something goes 
wrong with the primary.


On IPv6, no less :-).

On a serious note, I can't even imagine what it takes to run a network 
of that scale, and plan for situations like this, if, indeed, inline 
access was lost.


Mark.

Re: IRR for IX peers





On 10/4/21 21:55, Nick Hilliard wrote:


 Nearly 30 years on, this is still the state of the art.


Not an unlike an NMS... still can't walk into a shop and just buy one 
that works out of the box :-).


Mark.

Re: Facebook post-mortems...

2021-10-04 Thread Patrick W. Gilmore

Update about the October 4th outage

https://engineering.fb.com/2021/10/04/networking-traffic/outage/

-- 
TTFN,
patrick

> On Oct 4, 2021, at 9:25 PM, Mel Beckman  wrote:
> 
> The CF post mortem looks sensible, and a good summary of what we all saw from 
> the outside with BGP routes being withdrawn. 
> 
> Given the fragility of BGP, this could still end up being a malicious attack. 
> 
> -mel via cell
> 
>> On Oct 4, 2021, at 6:19 PM, Jay Hennigan  wrote:
>> 
>> On 10/4/21 17:58, jcur...@istaff.org wrote:
>>> Fairly abstract - Facebook Engineering - 
>>> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
>>>  
>>> 
>> 
>> I believe that the above link refers to a previous outage. The duration of 
>> the outage doesn't match today's, the technical explanation doesn't align 
>> very well, and many of the comments reference earlier dates.
>> 
>>> Also, Cloudflare’s take on the outage - 
>>> https://blog.cloudflare.com/october-2021-facebook-outage/ 
>>> 
>> 
>> This appears to indeed reference today's event.
>> 
>> -- 
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV

Re: Facebook post-mortems...

2021-10-04 Thread Mel Beckman

The CF post mortem looks sensible, and a good summary of what we all saw from 
the outside with BGP routes being withdrawn. 

Given the fragility of BGP, this could still end up being a malicious attack. 

-mel via cell

> On Oct 4, 2021, at 6:19 PM, Jay Hennigan  wrote:
> 
> On 10/4/21 17:58, jcur...@istaff.org wrote:
>> Fairly abstract - Facebook Engineering - 
>> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
>>  
>> 
> 
> I believe that the above link refers to a previous outage. The duration of 
> the outage doesn't match today's, the technical explanation doesn't align 
> very well, and many of the comments reference earlier dates.
> 
>> Also, Cloudflare’s take on the outage - 
>> https://blog.cloudflare.com/october-2021-facebook-outage/ 
>> 
> 
> This appears to indeed reference today's event.
> 
> -- 
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV

Re: update - Re: Facebook post-mortems...

2021-10-04 Thread Rabbi Rob Thomas


>> Fairly abstract - Facebook Engineering -
>> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
>> 
> 
> My bad - might be best to ignore the above post as it is a
> unconfirmed/undated post-mortem that may reference a different event. 

If I'm reading the source correctly, the timestamp inside is for 09 SEP
2021 14:22:49 GMT (Unix time 1631197369).  Then again, I may not be
reading it correctly.  :)


-- 
Rabbi Rob Thomas   Team Cymru
   "It is easy to believe in freedom of speech for those with whom we
agree." - Leo McKern



OpenPGP_signature
Description: OpenPGP digital signature

Re: update - Re: Facebook post-mortems...



On 10/4/21 6:07 PM, jcur...@istaff.org wrote:
On 4 Oct 2021, at 8:58 PM, jcur...@istaff.org 
 wrote:


Fairly abstract - Facebook Engineering - 
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr 



My bad - might be best to ignore the above post as it is a 
unconfirmed/undated post-mortem that may reference a different event.


One of the replies say it's from February, so year.

Mike

Re: Facebook post-mortems...


On 10/4/21 17:58, jcur...@istaff.org wrote:
Fairly abstract - Facebook Engineering - 
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr 



I believe that the above link refers to a previous outage. The duration 
of the outage doesn't match today's, the technical explanation doesn't 
align very well, and many of the comments reference earlier dates.


Also, Cloudflare’s take on the outage - 
https://blog.cloudflare.com/october-2021-facebook-outage/ 



This appears to indeed reference today's event.

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Facebook post-mortems...



On 10/4/21 5:58 PM, jcur...@istaff.org wrote:
Fairly abstract - Facebook Engineering - 
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr 



Also, Cloudflare’s take on the outage - 
https://blog.cloudflare.com/october-2021-facebook-outage/ 





They have a monkey patch subsystem. Lol.

Mike

update - Re: Facebook post-mortems...

2021-10-04 Thread jcurran

On 4 Oct 2021, at 8:58 PM, jcur...@istaff.org wrote:
> 
> Fairly abstract - Facebook Engineering - 
> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
>  
> 
My bad - might be best to ignore the above post as it is a unconfirmed/undated 
post-mortem that may reference a different event. 

> Also, Cloudflare’s take on the outage - 
> https://blog.cloudflare.com/october-2021-facebook-outage/ 
> 
The Cloudflare writeup looks quite solid (loss of network -> no DNS servers -> 
major issue) 
/John

Re: Facebook post-mortems...

2021-10-04 Thread Rubens Kuhl

The FB one seems to be from a previous event. Downtime doesn't match,
visible flaw effects don't either.


Rubens


On Mon, Oct 4, 2021 at 9:59 PM  wrote:
>
> Fairly abstract - Facebook Engineering - 
> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
>
> Also, Cloudflare’s take on the outage - 
> https://blog.cloudflare.com/october-2021-facebook-outage/
>
> FYI,
> /John
>

Facebook post-mortems...

2021-10-04 Thread jcurran

Fairly abstract - Facebook Engineering - 
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
 


Also, Cloudflare’s take on the outage - 
https://blog.cloudflare.com/october-2021-facebook-outage/ 


FYI,
/John

Re: facebook outage

Ok, I lied, I’m still awake.

I got my first successful Facebook main page load at 23:13 UTC, for an overall 
duration of 8:33, or 513 minutes.  Multiplied by three billion users, that’s 
1.54 trillion person-minutes.

That’s a tera-lapse!

Have we had one of those before?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: massive facebook outage presently

2021-10-04 Thread Doug McIntyre

On Mon, Oct 04, 2021 at 05:50:07PM -0400, b...@theworld.com wrote:
> One might think in over six hours they could point facebook.com's DNS
> somewhere else and put up a page with some info about the outage
> there, that this would be a practiced firedrill.

Perhaps, if they didn't decide to be their own registrar as well
and run it all on the same network as it seems.

Maybe company divisions running different parts of infrastructure
should be self-hosted 100% on their own, different AS, different networking 
points, etc.
Or don't try to be everything top down all in the same company.

Re: facebook outage

2021-10-04 Thread John Lee

I was seeing NXDOMAIN errors, so I wonder if they had a DNS outage of some
sort??

On Mon, Oct 4, 2021 at 5:14 PM Bill Woodcock  wrote:

> They’re starting to pick themselves back up off the floor in the last two
> or three minutes.  A few answers getting out.  I imagine it’ll take a while
> before things stabilize, though.
>
> -Bill
>
>

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker


* b...@theworld.com (b...@theworld.com) [Tue 05 Oct 2021, 00:01 CEST]:
I only mean a single, simple information page like the "sorry we're 
working on it" I saw just before they came back.


Which would subsequently receive the world's FB session cookies.


-- Niels.

Re: facebook outage



> On Oct 5, 2021, at 12:16 AM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl  
>> wrote:
>> 
>> 
>> 
>> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
>> 
>> 
>>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>>> 
>>> 
>>> 
 On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
 
 They’re starting to pick themselves back up off the floor in the last two 
 or three minutes.  A few answers getting out.  I imagine it’ll take a 
 while before things stabilize, though.
>>> 
>>> nd we’re back:
>>> 
>>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>> 
>> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
>> fifteen minutes.
>> 
>> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
>> burns beyond that.
>> 
>> Maybe they’ll get one or two independent secondary authoritatives, so this 
>> doesn’t happen again.  :-)
>> 
>> 
>> We have had dns back for a while here but the site is still down. Not 
>> counting this as over yet.
> 
> Yeah, fair enough.  I went back and looked, and it looks like the BGP 
> withdrawals were around 16:40 UTC?  And as of 22:15 UTC, application-layer 
> services still aren’t up.  Which puts us at 6:35 thus far?

A.  It’s past midnight here, and my brain is failing to convert between 
three timezones accurately.  My apologies.  I’ll stop typing until I’ve had 
some sleep.  Good night.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: IRR for IX peers

>> a SIX peer's customer could be the feed to RIS
> Sure, but how do you describe the policy between your peer and their
> customer in your aut-num?! That's not a thing.

yup

these rat holes are a pita

randy

Re: IRR for IX peers

2021-10-04 Thread Ben Maddison via NANOG

Hi Randy,

On 10/04, Randy Bush wrote:
> hi ben,
> 
> a SIX peer's customer could be the feed to RIS
> 
Sure, but how do you describe the policy between your peer and their
customer in your aut-num?! That's not a thing.

Cheers,

Ben


signature.asc
Description: PGP signature

Re: facebook outage



> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl  
> wrote:
> 
> 
> 
> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
> 
> 
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last two 
> >> or three minutes.  A few answers getting out.  I imagine it’ll take a 
> >> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
> 
> 
> We have had dns back for a while here but the site is still down. Not 
> counting this as over yet.

Yeah, fair enough.  I went back and looked, and it looks like the BGP 
withdrawals were around 16:40 UTC?  And as of 22:15 UTC, application-layer 
services still aren’t up.  Which puts us at 6:35 thus far?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage (resolving)

2021-10-04 Thread mike tancsa

Getting the odd message through, but DNS looks good via their Toronto,
Canada pop


% traceroute -q1 -I a.dns.facebook.com
traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte
packets
 1  torix-core1-10G (67.43.129.248)  0.140 ms
 2  facebook-a.ip4.torontointernetxchange.net (206.108.35.2)  0.880 ms
 3  po103.psw02.yyz1.tfbnw.net (74.119.78.131)  0.365 ms
 4  173.252.67.57 (173.252.67.57)  0.341 ms
 5  edge-star-shv-01-yyz1.facebook.com (31.13.80.8)  0.263 ms
% traceroute6 -q1 -I a.dns.facebook.com
traceroute6 to star.c10r.facebook.com (2a03:2880:f00e:a:face:b00c:0:2)
from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets
 1  toronto-torix-6  0.133 ms
 2  facebook-a.ip6.torontointernetxchange.net  0.532 ms
 3  po103.psw01.yyz1.tfbnw.net  0.322 ms
 4  po1.msw1ah.01.yyz1.tfbnw.net  0.335 ms
 5  edge-star6-shv-01-yyz1.facebook.com  0.267 ms
% traceroute6 -q1 -I d.dns.facebook.com
traceroute6 to star.c10r.facebook.com (2a03:2880:f00e:a:face:b00c:0:2)
from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets
 1  toronto-torix-6  0.126 ms
 2  facebook-a.ip6.torontointernetxchange.net  0.673 ms
 3  po103.psw01.yyz1.tfbnw.net  0.349 ms
 4  po1.msw1ah.01.yyz1.tfbnw.net  0.332 ms
 5  edge-star6-shv-01-yyz1.facebook.com  0.264 ms
% traceroute -q1 -I d.dns.facebook.com
traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte
packets
 1  torix-core1-10G (67.43.129.248)  0.139 ms
 2  facebook-a.ip4.torontointernetxchange.net (206.108.35.2)  41.394 ms
 3  po103.psw02.yyz1.tfbnw.net (74.119.78.131)  0.285 ms
 4  173.252.67.57 (173.252.67.57)  0.309 ms
 5  edge-star-shv-01-yyz1.facebook.com (31.13.80.8)  0.211 ms
%

% host www.facebook.com a.ns.facebook.com
Using domain server:
Name: a.ns.facebook.com
Address: 2a03:2880:f0fc:c:face:b00c:0:35#53
Aliases:

www.facebook.com is an alias for star-mini.c10r.facebook.com.
% host -4 www.facebook.com a.ns.facebook.com
Using domain server:
Name: a.ns.facebook.com
Address: 129.134.30.12#53
Aliases:

www.facebook.com is an alias for star-mini.c10r.facebook.com.
%

On 10/4/2021 5:41 PM, Baldur Norddahl wrote:
>
>
> man. 4. okt. 2021 23.33 skrev Bill Woodcock  >:
>
>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  > wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  > wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in
> the last two or three minutes.  A few answers getting out.  I
> imagine it’ll take a while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com
>  @9.9.9.9 
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five
> hours and fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some
> third-degree burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives,
> so this doesn’t happen again.  :-)
>
>
>
> We have had dns back for a while here but the site is still down. Not
> counting this as over yet.
>
>
>

Re: facebook outage

2021-10-04 Thread chris

Hopefully this will show people they can enjoy life and survive without it
and that it will be just like myspace some day :)

On Mon, Oct 4, 2021 at 5:31 PM Jeff Shultz  wrote:

> Now they just need to get the site itself back up.
>
> On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock  wrote:
>
>>
>>
>> > On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> >
>> > They’re starting to pick themselves back up off the floor in the last
>> two or three minutes.  A few answers getting out.  I imagine it’ll take a
>> while before things stabilize, though.
>>
>> nd we’re back:
>>
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>>
>> ; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
>> ;; global options: +cmd
>> ;; Got answer:
>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>>
>> ;; OPT PSEUDOSECTION:
>> ; EDNS: version: 0, flags:; udp: 512
>> ;; QUESTION SECTION:
>> ;www.facebook.com.  IN  A
>>
>> ;; ANSWER SECTION:
>> www.facebook.com.   3420IN  CNAME
>> star-mini.c10r.facebook.com.
>> star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35
>>
>> ;; Query time: 13 msec
>> ;; SERVER: 9.9.9.9#53(9.9.9.9)
>> ;; WHEN: Mon Oct 04 23:20:41 CEST 2021
>> ;; MSG SIZE  rcvd: 90
>>
>>
>> -Bill
>>
>>
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
> [image:
> https://www.instagram.com/sctc_sctc/]
> 
> 
> 
>
>
>
>
>
>
>
>  This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. 
>

Re: facebook outage

> On Oct 4, 2021, at 11:50 PM, Ryan Brooks  wrote:
> DNS was a victim in this outage, not the cause.

You are absolutely correct.  However, people who don’t have this problem avoid 
having this problem by not putting all their DNS eggs in one basket.

And then forgetting where they put the basket.

-Bill

signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

On 10/4/21 2:41 PM, Baldur Norddahl wrote:

man. 4. okt. 2021 23.33 skrev Bill Woodcock >:

> On Oct 4, 2021, at 11:21 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:
>
>
>
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:
>>
>> They’re starting to pick themselves back up off the floor in
the last two or three minutes.  A few answers getting out.  I
imagine it’ll take a while before things stabilize, though.
>
> nd we’re back:
>
> WoodyNet-2:.ssh woody$ dig www.facebook.com
 @9.9.9.9 

So that was, what…  15:50 UTC to 21:05 UTC, more or less… five
hours and fifteen minutes.

That’s a lot of hair burnt all the way to the scalp, and some
third-degree burns beyond that.

Maybe they’ll get one or two independent secondary authoritatives,
so this doesn’t happen again.  :-)

We have had dns back for a while here but the site is still down. Not 
counting this as over yet.

I got a page to load. Probably trickling out.

Mike

Re: facebook outage

2021-10-04 Thread Patrick W. Gilmore

On Oct 4, 2021, at 5:30 PM, Bill Woodcock  wrote:
> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>>> 
>>> They’re starting to pick themselves back up off the floor in the last two 
>>> or three minutes.  A few answers getting out.  I imagine it’ll take a while 
>>> before things stabilize, though.
>> 
>> nd we’re back:
>> 
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)

If by “independent” you mean “3rd party” (e.g. DynDNS), not sure what an 
external secondary would have done here. While their BGP was misbehaving, the 
app would not work even if you had a static DNS entry.

And while using external / 3rd party secondaries is likely a good idea for many 
companies, almost none of the largest do this. These companies view it as a 
control issue. Giving someone outside your own employees the ability to change 
a DNS name is, frankly, giving another company the ability to take you down.

Taking a sample of FB, cisco, Amazon, NF, Dell, Akamai, Google, MS, CF, only 2 
use 3rd party resolvers.
* NF uses only awsdns, so same problem, just moved to another company they do 
not control.
* Amazon uses Ultra & Dyn. (Anyone else amused amazon.com has no authorities on 
Route 53? At least not from my vantage point.)

That said, plenty of what people may call “big” companies do use 3rd parties, 
e.g. IBM, PayPal, Juniper.

You want to use a 3rd party DNS, go for it. There are lots of reasons to do it. 
But it is not a panacea, and there are reasons not to.

-- 
TTFN,
patrick

Re: facebook outage

2021-10-04 Thread Jeff Shultz

On Mon, Oct 4, 2021 at 2:49 PM Baldur Norddahl 
wrote:

>
>
> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
>
>>
>>
>> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> >
>> >
>> >
>> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> >>
>> >> They’re starting to pick themselves back up off the floor in the last
>> two or three minutes.  A few answers getting out.  I imagine it’ll take a
>> while before things stabilize, though.
>> >
>> > nd we’re back:
>> >
>> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>>
>> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and
>> fifteen minutes.
>>
>> That’s a lot of hair burnt all the way to the scalp, and some
>> third-degree burns beyond that.
>>
>> Maybe they’ll get one or two independent secondary authoritatives, so
>> this doesn’t happen again.  :-)
>>
>
>
> We have had dns back for a while here but the site is still down. Not
> counting this as over yet.
>
>
>
>
I'm getting part of my news feed and notifications. Can't post yet, and
clicking on something usually sends you back to the "Something is broken"
message.

So, no - it's not back up yet. But it's starting to twitch

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: massive facebook outage presently

2021-10-04 Thread bzs



One might think in over six hours they could point facebook.com's DNS
somewhere else and put up a page with some info about the outage
there, that this would be a practiced firedrill.

Yeah yeah cache blah blah but it'd get around and at least would be
coming from them. I'd imagine some mutual pact with google's or
amazon's or microsoft's cloud server(s), for example, could handle it
for a few hours, and vice-versa.

I only mean a single, simple information page like the "sorry we're
working on it" I saw just before they came back.

Then again what else would one assume?

17:44 They seem to be back, more or less, slowly loading the usual
crazy sauce.

So out about 6 hours and some.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: facebook outage

2021-10-04 Thread Ryan Brooks



> On Oct 4, 2021, at 4:30 PM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> 
>> 
>> 
>>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>>> 
>>> They’re starting to pick themselves back up off the floor in the last two 
>>> or three minutes.  A few answers getting out.  I imagine it’ll take a while 
>>> before things stabilize, though.
>> 
>> nd we’re back:
>> 
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
> 

DNS was a victim in this outage, not the cause.

>-Bill

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks

Back and working by the looks.

Re: IRR for IX peers

hi ben,

a SIX peer's customer could be the feed to RIS

randy

Re: facebook outage

man. 4. okt. 2021 23.33 skrev Bill Woodcock :

>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last
> two or three minutes.  A few answers getting out.  I imagine it’ll take a
> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and
> fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree
> burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this
> doesn’t happen again.  :-)
>


We have had dns back for a while here but the site is still down. Not
counting this as over yet.

Re: massive facebook outage presently

2021-10-04 Thread bzs



17:35EDT: I'm suddenly getting:

 Sorry, something went wrong.
We're working on it and we'll get it fixed as soon as we can.

   Go Back

Facebook © 2020 · Help Center

and:

% host facebook.com

facebook.com has address 157.240.241.35
facebook.com has IPv6 address 2a03:2880:f112:182:face:b00c:0:25de
facebook.com mail is handled by 10 smtpin.vvv.facebook.com.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: facebook outage

2021-10-04 Thread Mike Lyon

Can't wait to see the RFO!

-Mike

On Mon, Oct 4, 2021 at 2:35 PM Bill Woodcock  wrote:
>
>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last two 
> >> or three minutes.  A few answers getting out.  I imagine it’ll take a 
> >> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
>
> -Bill
>


-- 
Mike Lyon
mike.l...@gmail.com
http://www.linkedin.com/in/mlyon

Re: IRR for IX peers

2021-10-04 Thread Ben Maddison via NANOG

Hi Randy,

On 10/04, Randy Bush wrote:
> so i have an AS (3130) which peers at the SIX (RSs and some direct).
> 
> in the hope that leak detectors such as artemis would stop false
> positives when they see my prefixes announced customer cones of SIX
> peers, i want to add the SIX peers to my aut-num: policy.
> 
I would be astonished if artemis was parsing your import/export
expressions, but as an academic exercise...

> export:  toAS-SEATTLEIX-RS-CLIENTS  announce AS-RG-SEA
> 
> seems clear and obvious.  but
> 
> import:  from  AS-SEATTLEIX-RS-CLIENTS  accept AS-SEATTLEIX-RS-CLIENTS
> 
> would seem to allow bill's bait and sushi to announce microsoft to me.
> and i am not sure that expansive `from` clause is actually allowed.
> 
The having an as-set in the peering-expr part is fine, but that
particular set appears to contain all of the peers' customer cones,
which is not what you want there.

Additionally it evaluates down to "any route originated by any customer
of any peer, from any customer of any (other) peer". That's not a good
filter, as you pointed out ;-)

In order to express what you want, the SIX needs to create:
- an as-set containing the ASNs of the RS peers (not their customers),
  e.g. AS33108:AS-PEERS;
  and
- for each peer, a set containing that peer's customer cone, each with
  name that contains the peer's ASN, e.g. AS33108:AS-PEERS:AS65000.

Then you can say something like:

import: from AS33108:AS-PEERS accept AS33108:AS-PEERS:PeerAS

The fact that the IX operator needs to maintain these sets is a symptom
of the same issue that makes it near impossible to construct sane
filters for route-server sessions: you have no idea when someone joins,
or what they *should* be announcing.

One of the many reasons you don't see us on route-servers.

> what are others in this space doing?
> 
Mostly, asking people fill-in peeringdb records, and ignoring
import/export attributes entirely.

However we use roughly the above scheme, just in case someone is
reading.

Cheers,

Ben


signature.asc
Description: PGP signature

Re: facebook outage

> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> 
>> They’re starting to pick themselves back up off the floor in the last two or 
>> three minutes.  A few answers getting out.  I imagine it’ll take a while 
>> before things stabilize, though.
> 
> nd we’re back:
> 
> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9

So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
fifteen minutes.

That’s a lot of hair burnt all the way to the scalp, and some third-degree 
burns beyond that.

Maybe they’ll get one or two independent secondary authoritatives, so this 
doesn’t happen again.  :-)

-Bill

signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

2021-10-04 Thread Jeff Shultz

Now they just need to get the site itself back up.

On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock  wrote:

>
>
> > On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >
> > They’re starting to pick themselves back up off the floor in the last
> two or three minutes.  A few answers getting out.  I imagine it’ll take a
> while before things stabilize, though.
>
> nd we’re back:
>
> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> ; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;www.facebook.com.  IN  A
>
> ;; ANSWER SECTION:
> www.facebook.com.   3420IN  CNAME
> star-mini.c10r.facebook.com.
> star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35
>
> ;; Query time: 13 msec
> ;; SERVER: 9.9.9.9#53(9.9.9.9)
> ;; WHEN: Mon Oct 04 23:20:41 CEST 2021
> ;; MSG SIZE  rcvd: 90
>
>
> -Bill
>
>

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: facebook outage



> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> 
> They’re starting to pick themselves back up off the floor in the last two or 
> three minutes.  A few answers getting out.  I imagine it’ll take a while 
> before things stabilize, though.

nd we’re back:

WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9

; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.facebook.com.  IN  A

;; ANSWER SECTION:
www.facebook.com.   3420IN  CNAME   star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35

;; Query time: 13 msec
;; SERVER: 9.9.9.9#53(9.9.9.9)
;; WHEN: Mon Oct 04 23:20:41 CEST 2021
;; MSG SIZE  rcvd: 90


-Bill



signature.asc
Description: Message signed with OpenPGP

Re: massive facebook outage presently

2021-10-04 Thread Bryan Fields

On 10/4/21 4:47 PM, Jean St-Laurent via NANOG wrote:
>  
> dig @c.gtld-servers.net. facebook.com. NS
> ;;
> facebook.com.   172800  IN  NS  a.ns.facebook.com.
> facebook.com.   172800  IN  NS  b.ns.facebook.com.
> facebook.com.   172800  IN  NS  c.ns.facebook.com.
> facebook.com.   172800  IN  NS  d.ns.facebook.com.
> 
> What happens if the NS aren’t back within 48 hours?

Facebook can just update it at the registrar.

$ whois facebook.com
   Domain Name: FACEBOOK.COM
   Registry Domain ID: 2320948_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.registrarsafe.com
   Registrar URL: http://www.registrarsafe.com
   Updated Date: 2021-09-22T19:33:41Z
   Creation Date: 1997-03-29T05:00:00Z
   Registry Expiry Date: 2030-03-30T04:00:00Z
   Registrar: RegistrarSafe, LLC

$ dig @c.gtld-servers.net. registrarsafe.com. NS
;;
registrarsafe.com.  172800  IN  NS  a.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  b.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  c.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  d.ns.facebook.com.

crap

Vertical integration is a hell of a thing.
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Re: massive facebook outage presently

Starting to see some prefixes return on our peering session with them in 
Los Angeles but DNS still not resolving.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: facebook outage

They’re starting to pick themselves back up off the floor in the last two or 
three minutes.  A few answers getting out.  I imagine it’ll take a while before 
things stabilize, though.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: massive facebook outage presently

2021-10-04 Thread None None

Not all employees are having this issue
But it is curious why the records can be moved to the name servers that are
still being advertised over the Internet still .


On Mon, Oct 4, 2021 at 4:53 PM Aaron C. de Bruyn via NANOG 
wrote:

> It looks like it might take a while according to a news reporter's tweet:
>
> "Was just on phone with someone who works for FB who described employees
> unable to enter buildings this morning to begin to evaluate extent of
> outage because their badges weren’t working to access doors."
>
> https://twitter.com/sheeraf/status/1445099150316503057?s=20
>
>
> -A
>
> On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:
>
>> I am starting to see reports that in ISPs with very large numbers of
>> residential users, customers are starting to press the factory-reset
>> buttons on their home routers/modems/whatever, in an attempt to make
>> Facebook work. This is resulting in much heavier than normal first tier
>> support volumes. The longer it stays down the worse this is going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>>
>>> On 10/4/21 12:11, b...@theworld.com wrote:
>>> >
>>> > Although I believe it's generally true that if a company appears
>>> > prominently in the news it's liable to be attacked I assume because
>>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>>> > motivation w/o some evidence like even a credible twitter post (maybe
>>> > they posted that on FB? :-)
>>>
>>> I personally believe that the outage was caused by human error and not
>>> something malicious. Time will tell.
>>>
>>> However, if you missed the 60 Minutes piece, it was a former employee
>>> who spoke out with some rather powerful observations. I don't think that
>>> this type of worldwide outage was caused by an outside bad actor. It is
>>> certainly within the realm of possibility that it was an inside job.
>>>
>>> In other news:
>>>
>>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>>
>>> --
>>> Jay Hennigan - j...@west.net
>>> Network Engineering - CCIE #7880
>>> 503 897-8550 - WB6RDV
>>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Matt Hoppes

Yes,
We've seen that.

On 10/4/21 4:33 PM, Eric Kuhnke wrote:
I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first tier 
support volumes. The longer it stays down the worse this is going to get.

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > wrote:

On 10/4/21 12:11, b...@theworld.com  wrote:
 >
 > Although I believe it's generally true that if a company appears
 > prominently in the news it's liable to be attacked I assume because
 > the miscreants sit around thinking "hmm, who shall we attack today oh
 > look at that shiny headline!" I'd hate to ascribe any altruistic
 > motivation w/o some evidence like even a credible twitter post (maybe
 > they posted that on FB? :-)

I personally believe that the outage was caused by human error and not
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee
who spoke out with some rather powerful observations. I don't think
that
this type of worldwide outage was caused by an outside bad actor. It is
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net 

Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently





On 10/4/21 18:22, Eric Kuhnke wrote:

Considering the massive impact of this it would be interesting to see 
some traffic graphs from ISPs that have PNIs with Facebook, or high 
volume peering sessions across an IX, showing traffic to FB falling 
off a cliff.


Our PNI's and FNA's in Africa have dipped.

But PNI's in Europe have nearly doubled, even though we aren't 
delivering any Facebook services of note to our eyeballs.


Strangest thing...

Mark.

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG

The glue records for the NS are set at 48 hours.

dig @c.gtld-servers.net. facebook.com. NS

;;

facebook.com.   172800  IN  NS  a.ns.facebook.com.

facebook.com.   172800  IN  NS  b.ns.facebook.com.

facebook.com.   172800  IN  NS  c.ns.facebook.com.

facebook.com.   172800  IN  NS  d.ns.facebook.com.

What happens if the NS aren’t back within 48 hours? 

Jean

From: NANOG  On Behalf Of Eric Kuhnke
Sent: October 4, 2021 4:33 PM
To: Jay Hennigan ; nanog@nanog.org list 
Subject: Re: massive facebook outage presently

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset buttons on 
their home routers/modems/whatever, in an attempt to make Facebook work. This 
is resulting in much heavier than normal first tier support volumes. The longer 
it stays down the worse this is going to get.

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan mailto:j...@west.net> > wrote:

On 10/4/21 12:11, b...@theworld.com   wrote:
> 
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net  
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman

I’m not the only one who finds this timing suspicious, Starting with the
publishers of 60 Minutes themselves :-)

CBS:

The outage comes the morning after "60 Minutes" aired an interview with a
whistleblower who said Facebook is aware of how it amplifies hate,
misinformation and unrest but claimed the company hides what it knows.

https://www.cbsnews.com/news/facebook-instagram-whatsapp-down-2021-10-04/

https://abcnews.go.com/Technology/facebook-instagram-users-us/story?id=80397437

https://www.cnbc.com/2021/10/04/facebook-shares-drop-5percent-after-site-outage-and-whistleblower-interview.html

https://www.insidenova.com/headlines/facebook-instagram-down-after-60-minutes-whistleblower-report/article_29977530-2531-11ec-b5ae-73311c46edfb.html

https://adage.com/article/digital-marketing-ad-tech-news/what-facebook-telling-advertisers-about-60-minutes-whistleblower/2370346

-mel beckman

On Oct 4, 2021, at 1:36 PM, Blake Dunlap wrote:

If there isn't an undernetwork capable of being backdoored with the proper keys
(I'd be shocked if there isn't - the big players have very good infra and DR
people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't even
get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for their
sres right now and am drinking one in their honor. I just want to know what an
October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl
mailto:baldur.nordd...@gmail.com>> wrote:
Not in such a primitive fashion no. But they could definitely have a secondary
network that will continue to work even if something goes wrong with the
primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli
mailto:pjcape...@pm.me>> wrote:
Seems unlikely that FB internal controls would allow such a backdoor ...

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl
mailto:baldur.nordd...@gmail.com>> wrote:

On Mon, 4 Oct 2021 at 21:58, Michael Thomas
mailto:m...@mtcc.com>> wrote:

On 10/4/21 11:48 AM, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration done via a
web interface). However, now that connection to the outside world is down,
remote access to those tools don't exist anymore, so the emergency procedure is
to gain physical access to the peering routers and do all the configuration
locally.

Assuming that this is what actually happened, what should fb have done
different (beyond the obvious of not screwing up the immediate issue)? This
seems like it's a single point of failure. Should all of the BGP speakers have
been dual homed or something like that? Or should they not have been mixing ops
and production networks? Sorry if this sounds dumb.

Facebook is a huge network. It is doubtful that what is going on is this
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using
someone else's network. Nowadays this could be a simple 4/5G router with a VPN,
to a terminal server that allows the operator to configure the equipment
through the monitor port even when the config is completely destroyed.

Regards,

Baldur

Re: massive facebook outage presently

2021-10-04 Thread Aaron C. de Bruyn via NANOG

It looks like it might take a while according to a news reporter's tweet:

"Was just on phone with someone who works for FB who described employees
unable to enter buildings this morning to begin to evaluate extent of
outage because their badges weren’t working to access doors."

https://twitter.com/sheeraf/status/1445099150316503057?s=20

-A

On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:

> I am starting to see reports that in ISPs with very large numbers of
> residential users, customers are starting to press the factory-reset
> buttons on their home routers/modems/whatever, in an attempt to make
> Facebook work. This is resulting in much heavier than normal first tier
> support volumes. The longer it stays down the worse this is going to get.
>
>
>
> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>
>> On 10/4/21 12:11, b...@theworld.com wrote:
>> >
>> > Although I believe it's generally true that if a company appears
>> > prominently in the news it's liable to be attacked I assume because
>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>> > motivation w/o some evidence like even a credible twitter post (maybe
>> > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former employee
>> who spoke out with some rather powerful observations. I don't think that
>> this type of worldwide outage was caused by an outside bad actor. It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> --
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>
>

Re: massive facebook outage presently

2021-10-04 Thread Miles Fidelman


Jay Hennigan wrote:

On 10/4/21 09:27, Tom Beecher wrote:
Curious if there is a malicious angle after the 60 Minutes story last 
night.


Hanlon's razor.

"Never attribute to malice that which is adequately explained by 
stupidity"


Corollary - "But don't rule out malice."


"Once is happenstance. Twice is coincidence. The third*time it's*enemy 
action'.”


--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: massive facebook outage presently

2021-10-04 Thread Hank Nussbacher


On 04/10/2021 22:05, Jason Kuehl wrote:

BGP related:
https://twitter.com/SGgrc/status/1445116435731296256
as also related by FB CTO:
https://twitter.com/atoonk/status/1445121351707070468

-Hank

https://twitter.com/disclosetv/status/1445100931947892736?s=20 



On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks > wrote:


Didn't write that part of the automation script and that coder left...

 > I got a mail that Facebook was leaving NLIX. Maybe someone
botched the
 > script so they took down all BGP sessions instead of just NLIX
and now
 > they can't access the equipment to put it back... :-)




--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg

No evidence that it’s intentional but…..   What’s going to be the big headline 
tonight?A Facebook whistleblower or a global outage that kept everyone from 
arguing all day long?

-richey

From: NANOG  on behalf of 
Jay Hennigan 
Date: Monday, October 4, 2021 at 3:30 PM
To: nanog@nanog.org 
Subject: Re: massive facebook outage presently
On 10/4/21 12:11, b...@theworld.com wrote:
>
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee
who spoke out with some rather powerful observations. I don't think that
this type of worldwide outage was caused by an outside bad actor. It is
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Eric Kuhnke

I am starting to see reports that in ISPs with very large numbers of
residential users, customers are starting to press the factory-reset
buttons on their home routers/modems/whatever, in an attempt to make
Facebook work. This is resulting in much heavier than normal first tier
support volumes. The longer it stays down the worse this is going to get.



On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:

> On 10/4/21 12:11, b...@theworld.com wrote:
> >
> > Although I believe it's generally true that if a company appears
> > prominently in the news it's liable to be attacked I assume because
> > the miscreants sit around thinking "hmm, who shall we attack today oh
> > look at that shiny headline!" I'd hate to ascribe any altruistic
> > motivation w/o some evidence like even a credible twitter post (maybe
> > they posted that on FB? :-)
>
> I personally believe that the outage was caused by human error and not
> something malicious. Time will tell.
>
> However, if you missed the 60 Minutes piece, it was a former employee
> who spoke out with some rather powerful observations. I don't think that
> this type of worldwide outage was caused by an outside bad actor. It is
> certainly within the realm of possibility that it was an inside job.
>
> In other news:
>
> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap

If there isn't an undernetwork capable of being backdoored with the proper
keys (I'd be shocked if there isn't - the big players have very good infra
and DR people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't
even get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for
their sres right now and am drinking one in their honor. I just want to
know what an October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl 
wrote:

> Not in such a primitive fashion no. But they could definitely have a
> secondary network that will continue to work even if something goes wrong
> with the primary.
>
> On Mon, 4 Oct 2021 at 22:16, PJ Capelli  wrote:
>
>> Seems unlikely that FB internal controls would allow such a backdoor ...
>>
>> "Never to get lost, is not living" - Rebecca Solnit
>>
>> Sent with ProtonMail  Secure Email.
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
>> baldur.nordd...@gmail.com> wrote:
>>
>>
>>
>> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
>>
>>>
>>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>>
>>>
>>> I believe the original change was 'automatic' (as in configuration done
>>> via a web interface). However, now that connection to the outside world is
>>> down, remote access to those tools don't exist anymore, so the emergency
>>> procedure is to gain physical access to the peering routers and do all the
>>> configuration locally.
>>>
>>> Assuming that this is what actually happened, what should fb have done
>>> different (beyond the obvious of not screwing up the immediate issue)? This
>>> seems like it's a single point of failure. Should all of the BGP speakers
>>> have been dual homed or something like that? Or should they not have been
>>> mixing ops and production networks? Sorry if this sounds dumb.
>>>
>>
>> Facebook is a huge network. It is doubtful that what is going on is this
>> simple. So I will make no guesses to what Facebook is or should be doing.
>>
>> However the traditional way for us small timers is to have a backdoor
>> using someone else's network. Nowadays this could be a simple 4/5G router
>> with a VPN, to a terminal server that allows the operator to configure the
>> equipment through the monitor port even when the config is completely
>> destroyed.
>>
>> Regards,
>>
>> Baldur
>>
>>
>>
>>
>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Łukasz Bromirski


Dual homing won’t help you if your automation template will do „no router bgp 
X” and at this point session will terminate as suddenly advertisement will be 
withdrawn…

It won’t you either if the change triggers some obscure bug in your BGP stack.

I bet FB tested the change on smaller scale and everything was fine, and only 
then started to roll this over wider network and at that point „something” 
broke. Or some bug needed a moment to start cascading issues around the infra.

-- 
./

> On 4 Oct 2021, at 22:00, Michael Thomas  wrote:
> 
> 
> 
> 
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>> 
>> I believe the original change was 'automatic' (as in configuration done via 
>> a web interface). However, now that connection to the outside world is down, 
>> remote access to those tools don't exist anymore, so the emergency procedure 
>> is to gain physical access to the peering routers and do all the 
>> configuration locally.
> Assuming that this is what actually happened, what should fb have done 
> different (beyond the obvious of not screwing up the immediate issue)? This 
> seems like it's a single point of failure. Should all of the BGP speakers 
> have been dual homed or something like that? Or should they not have been 
> mixing ops and production networks? Sorry if this sounds dumb.
> 
> Mike

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach

On Mon, Oct 4, 2021 at 11:59 AM Jason Kuehl  wrote:

> I mean, you're an idiot if you post that public on the internet about
> your own place of work. What do you think would happen? Nothing? He should
> never of said anything, but now the Facebook hitman got him.
>
>
Some of us have done that, and survived[0].

But I would be the first to admit I've led a
very charmed life in that regard.   ^_^;

Matt

[0]
https://www.computerworld.com/article/2529621/networking-glitch-knocks-yahoo-offline-for-some.html

Re: massive facebook outage presently

Not in such a primitive fashion no. But they could definitely have a
secondary network that will continue to work even if something goes wrong
with the primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli  wrote:

> Seems unlikely that FB internal controls would allow such a backdoor ...
>
> "Never to get lost, is not living" - Rebecca Solnit
>
> Sent with ProtonMail  Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>
>
>
> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
>
>>
>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>
>>
>> I believe the original change was 'automatic' (as in configuration done
>> via a web interface). However, now that connection to the outside world is
>> down, remote access to those tools don't exist anymore, so the emergency
>> procedure is to gain physical access to the peering routers and do all the
>> configuration locally.
>>
>> Assuming that this is what actually happened, what should fb have done
>> different (beyond the obvious of not screwing up the immediate issue)? This
>> seems like it's a single point of failure. Should all of the BGP speakers
>> have been dual homed or something like that? Or should they not have been
>> mixing ops and production networks? Sorry if this sounds dumb.
>>
>
> Facebook is a huge network. It is doubtful that what is going on is this
> simple. So I will make no guesses to what Facebook is or should be doing.
>
> However the traditional way for us small timers is to have a backdoor
> using someone else's network. Nowadays this could be a simple 4/5G router
> with a VPN, to a terminal server that allows the operator to configure the
> equipment through the monitor port even when the config is completely
> destroyed.
>
> Regards,
>
> Baldur
>
>
>
>
>
>

Re: IRR for IX peers

>> what are others in this space doing?
> 
> not using import/export lines in their RS or router configs, for
> starters.  Probably you could count the number of IXPs that inspect
> import/export lines on the fingers of one hand, and possibly of one
> finger.
> 
> Generally speaking, IXPs try to aim for filters based on a single
> {as-set,IRRDB set} tuple per RS client configured.  If you're aiming
> for bilat bgp sessions, then this functionality would need to be
> replicated. Nearly 30 years on, this is still the state of the art.

i am not looking for the SIX to filter, though they do filtering.

my issue is

   3130 --- SIX --- martha --- RIS

artemis runs off a RIS feed

martha is telling RIS MARTHA_3130 and artemis is saying that martha
is trying to hijack 3130's prefix.

i was hoping that, if 3130 said it is peering with martha, artemis would
get a clue and stfu

randy

Re: massive facebook outage presently

On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:

>
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>
>
> I believe the original change was 'automatic' (as in configuration done
> via a web interface). However, now that connection to the outside world is
> down, remote access to those tools don't exist anymore, so the emergency
> procedure is to gain physical access to the peering routers and do all the
> configuration locally.
>
> Assuming that this is what actually happened, what should fb have done
> different (beyond the obvious of not screwing up the immediate issue)? This
> seems like it's a single point of failure. Should all of the BGP speakers
> have been dual homed or something like that? Or should they not have been
> mixing ops and production networks? Sorry if this sounds dumb.
>

Facebook is a huge network. It is doubtful that what is going on is this
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using
someone else's network. Nowadays this could be a simple 4/5G router with a
VPN, to a terminal server that allows the operator to configure the
equipment through the monitor port even when the config is completely
destroyed.

Regards,

Baldur

Re: IRR for IX peers

2021-10-04 Thread Rubens Kuhl

Some IX'es set communities telling which member announced that prefix;
if SIX is one of those, that can be used to automate origin
verification.


Rubens

On Mon, Oct 4, 2021 at 2:08 PM Randy Bush  wrote:
>
> so i have an AS (3130) which peers at the SIX (RSs and some direct).
>
> in the hope that leak detectors such as artemis would stop false
> positives when they see my prefixes announced customer cones of SIX
> peers, i want to add the SIX peers to my aut-num: policy.
>
> export:  toAS-SEATTLEIX-RS-CLIENTS  announce AS-RG-SEA
>
> seems clear and obvious.  but
>
> import:  from  AS-SEATTLEIX-RS-CLIENTS  accept AS-SEATTLEIX-RS-CLIENTS
>
> would seem to allow bill's bait and sushi to announce microsoft to me.
> and i am not sure that expansive `from` clause is actually allowed.
>
> what are others in this space doing?
>
> [ and let's not descend into the rat-hole of dissing the IRR.  i have
>   heard of this RPKI thing and might try it some day. ]
>
> randy
>
> ---
> ra...@psg.com
> `gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
> signatures are back, thanks to dmarc header butchery

Re: massive facebook outage presently



On 10/4/21 11:48 AM, Luke Guillory wrote:


I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.


Assuming that this is what actually happened, what should fb have done 
different (beyond the obvious of not screwing up the immediate issue)? 
This seems like it's a single point of failure. Should all of the BGP 
speakers have been dual homed or something like that? Or should they not 
have been mixing ops and production networks? Sorry if this sounds dumb.


Mike

Re: IRR for IX peers

2021-10-04 Thread Nick Hilliard


Randy Bush wrote on 04/10/2021 17:44:

what are others in this space doing?


not using import/export lines in their RS or router configs, for 
starters.  Probably you could count the number of IXPs that inspect 
import/export lines on the fingers of one hand, and possibly of one finger.


Generally speaking, IXPs try to aim for filters based on a single 
{as-set,IRRDB set} tuple per RS client configured.  If you're aiming for 
bilat bgp sessions, then this functionality would need to be replicated. 
Nearly 30 years on, this is still the state of the art.


Nick

Re: massive facebook outage presently

2021-10-04 Thread Töma Gavrichenkov

Peace,

On Mon, Oct 4, 2021, 10:17 PM Jean St-Laurent via NANOG 
wrote:

> Maybe the key to solve this issue is in an email sent to
> some_very_important_t...@facebook.com


Yeah except MX records on facebook dot com aren't working either

--
Töma

Re: massive facebook outage presently


On 10/4/21 12:11, b...@theworld.com wrote:


Although I believe it's generally true that if a company appears
prominently in the news it's liable to be attacked I assume because
the miscreants sit around thinking "hmm, who shall we attack today oh
look at that shiny headline!" I'd hate to ascribe any altruistic
motivation w/o some evidence like even a credible twitter post (maybe
they posted that on FB? :-)


I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.


However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.


In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG

Maybe the key to solve this issue is in an email sent to 
some_very_important_t...@facebook.com

-Original Message-
From: NANOG  On Behalf Of tomocha
Sent: October 4, 2021 2:32 PM
To: nanog@nanog.org
Subject: Re: massive facebook outage presently

Hi

Some of the DNS addresses are no longer prefix from AS32934.



On 2021/10/05 1:23, Hugo Slabbert wrote:
> Looks like their auth DNS dropped out of the DFZ:
> 
> https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
> https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20
> 
>

Re: massive facebook outage presently

2021-10-04 Thread bzs

Although I believe it's generally true that if a company appears
prominently in the news it's liable to be attacked I assume because
the miscreants sit around thinking "hmm, who shall we attack today oh
look at that shiny headline!" I'd hate to ascribe any altruistic
motivation w/o some evidence like even a credible twitter post (maybe
they posted that on FB? :-)

I wonder how often the US White House, congress, et al are attacked
since they appear in almost any top headline list often centering on
something someone out there really doesn't like?

On October 4, 2021 at 17:35 m...@beckman.org (Mel Beckman) wrote:
 > Suspiciously, this comes the morning after Facebook whistleblower Frances
 > Haugen disclosed on 60 Minutes that Facebook's own research shows that it 
 > chose
 > to profit from misinformation and political unrest through deliberate
 > escalation of conflicts. Occam’s razor says “When multiple causes are
 > plausible, and CBS 60 Minutes is one of them, go with 60 Minutes.” :)
 > 
 >  -mel 
 > 
 > 
 > On Oct 4, 2021, at 10:30 AM, Michael Spears  wrote:
 > 
 > 
 > https://www.reddit.com/r/sysadmin/comments/q181fv/
 > looks_like_facebook_is_down/?utm_medium=android_app_source=share
 > 
 > Sent from my Verizon, Samsung Galaxy smartphone
 > Get Outlook for Android
 > 

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: massive facebook outage presently

2021-10-04 Thread tomocha


Hi

Some of the DNS addresses are no longer prefix from AS32934.



On 2021/10/05 1:23, Hugo Slabbert wrote:

Looks like their auth DNS dropped out of the DFZ:

https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20
https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl

https://twitter.com/disclosetv/status/1445100931947892736?s=20

On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks  wrote:

> Didn't write that part of the automation script and that coder left...
>
> > I got a mail that Facebook was leaving NLIX. Maybe someone botched the
> > script so they took down all BGP sessions instead of just NLIX and now
> > they can't access the equipment to put it back... :-)
>
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks

Didn't write that part of the automation script and that coder left...

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the 
> script so they took down all BGP sessions instead of just NLIX and now 
> they can't access the equipment to put it back... :-)

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl

I mean, you're an idiot if you post that public on the internet about
your own place of work. What do you think would happen? Nothing? He should
never of said anything, but now the Facebook hitman got him.

Facebook will have to send out a Reason For Outage with all the services
it's effecting, like login

On Mon, Oct 4, 2021 at 2:46 PM Blake Dunlap  wrote:

> You laugh but that kind of sounds like what happened so far as oops we
> isolated prod and are scrambling on DR. There was someone supposedly live
> tweeting from their incident response for a bit before their account panic
> deleted.
>
> On Mon, Oct 4, 2021, 13:42 Baldur Norddahl 
> wrote:
>
>> I got a mail that Facebook was leaving NLIX. Maybe someone botched the
>> script so they took down all BGP sessions instead of just NLIX and now they
>> can't access the equipment to put it back... :-)
>>
>>
>> man. 4. okt. 2021 20.31 skrev Billy Croan :
>>
>>> I know what this is.  They forgot to update the credit card on their
>>> godaddy account and the domain lapsed.  I guess it will be facebook.info
>>> when they get it back online.  The post mortem should be an interesting
>>> read.
>>>
>>> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
>>> wrote:
>>>
 Looks like they run there own nameservers and I see the soa records are
 even missing.

 On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:

> Here’s a screenshot:
>
>
>
>  -mel beckman
>
> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>
> 
> https://downdetector.com/status/facebook/
>
> Normally not worth mentioning random $service having an outage here,
> but this will undoubtedly generate a large volume of customer service
> calls.
>
> Appears to be failure in DNS resolution.
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: massive facebook outage presently

2021-10-04 Thread Luke Guillory

From what I believe was a FB employee on Reddit, account now deleted it seems.


As many of you know, DNS for FB services has been affected and this is likely a 
symptom of the actual issue, and that's that BGP peering with Facebook peering 
routers has gone down, very likely due to a configuration change that went into 
effect shortly before the outages happened (started roughly 1540 UTC).



There are people now trying to gain access to the peering routers to implement 
fixes, but the people with physical access is separate from the people with 
knowledge of how to actually authenticate to the systems and people who know 
what to actually do, so there is now a logistical challenge with getting all 
that knowledge unified.



Part of this is also due to lower staffing in data centers due to pandemic 
measures.



I believe the original change was 'automatic' (as in configuration done via a 
web interface). However, now that connection to the outside world is down, 
remote access to those tools don't exist anymore, so the emergency procedure is 
to gain physical access to the peering routers and do all the configuration 
locally.



https://twitter.com/jgrahamc/status/1445068309288951820 "About five minutes 
before Facebook's DNS stopped working we saw a large number of BGP changes 
(mostly route withdrawals) for Facebook's ASN."




From: NANOG  On Behalf Of 
Baldur Norddahl
Sent: Monday, October 4, 2021 1:41 PM
To: NANOG 
Subject: Re: massive facebook outage presently

*External Email: Use Caution*
I got a mail that Facebook was leaving NLIX. Maybe someone botched the script 
so they took down all BGP sessions instead of just NLIX and now they can't 
access the equipment to put it back... :-)


man. 4. okt. 2021 20.31 skrev Billy Croan 
mailto:bcr...@unrealservers.net>>:
I know what this is.  They forgot to update the credit card on their 
godaddy account and the domain lapsed.  I guess it will be 
facebook.info
 when they get it back online.  The post mortem should be an interesting read.

On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
mailto:jason.w.ku...@gmail.com>> wrote:
Looks like they run there own nameservers and I see the soa records are even 
missing.

On Mon, Oct 4, 2021, 12:23 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Here’s a screenshot:

 -mel beckman


On Oct 4, 2021, at 9:06 AM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

https://link.edgepilot.com/s/3926b9ff/bTkszib6zUmYbE_rZxhltQ?u=https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls.

Appears to be failure in DNS resolution.



Links contained in this email have been replaced. If you click on a link in the 
email above, the link will be analyzed for known threats. If a known threat is 
found, you will not be able to proceed to the destination. If suspicious 
content is detected, you will see a warning.

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

- On Oct 4, 2021, at 11:41 AM, Baldur Norddahl baldur.nordd...@gmail.com 
wrote:

Hi,

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the script 
> so
> they took down all BGP sessions instead of just NLIX and now they can't access
> the equipment to put it back... :-)

That's an interesting theory. Once upon a time I saw a billion dollar company 
suffer
a significant outage after enabling EVPN on a remote site. Took down the entire
backbone, including access to the site.

Thanks,

Sabri

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap

You laugh but that kind of sounds like what happened so far as oops we
isolated prod and are scrambling on DR. There was someone supposedly live
tweeting from their incident response for a bit before their account panic
deleted.

On Mon, Oct 4, 2021, 13:42 Baldur Norddahl 
wrote:

> I got a mail that Facebook was leaving NLIX. Maybe someone botched the
> script so they took down all BGP sessions instead of just NLIX and now they
> can't access the equipment to put it back... :-)
>
>
> man. 4. okt. 2021 20.31 skrev Billy Croan :
>
>> I know what this is.  They forgot to update the credit card on their
>> godaddy account and the domain lapsed.  I guess it will be facebook.info
>> when they get it back online.  The post mortem should be an interesting
>> read.
>>
>> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
>> wrote:
>>
>>> Looks like they run there own nameservers and I see the soa records are
>>> even missing.
>>>
>>> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>>>
 Here’s a screenshot:

  -mel beckman

 On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:

 https://downdetector.com/status/facebook/

 Normally not worth mentioning random $service having an outage here,
 but this will undoubtedly generate a large volume of customer service
 calls.

 Appears to be failure in DNS resolution.

Re: massive facebook outage presently

I got a mail that Facebook was leaving NLIX. Maybe someone botched the
script so they took down all BGP sessions instead of just NLIX and now they
can't access the equipment to put it back... :-)


man. 4. okt. 2021 20.31 skrev Billy Croan :

> I know what this is.  They forgot to update the credit card on their
> godaddy account and the domain lapsed.  I guess it will be facebook.info
> when they get it back online.  The post mortem should be an interesting
> read.
>
> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
> wrote:
>
>> Looks like they run there own nameservers and I see the soa records are
>> even missing.
>>
>> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>>
>>> Here’s a screenshot:
>>>
>>>
>>>
>>>  -mel beckman
>>>
>>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>>
>>> 
>>> https://downdetector.com/status/facebook/
>>>
>>> Normally not worth mentioning random $service having an outage here, but
>>> this will undoubtedly generate a large volume of customer service calls.
>>>
>>> Appears to be failure in DNS resolution.
>>>
>>>

Re: The Outage

2021-10-04 Thread Andy Ringsmuth



> On Oct 4, 2021, at 1:31 PM, Matthew Petach  wrote:
> 
> 
> 
> On Mon, Oct 4, 2021 at 10:58 AM Andy Ringsmuth  wrote:
> I suppose it could just be me.
> 
> Or it could be more than just me.
> 
> Anyone else noticing that at the same time as the giant FB outage, both the 
> outages and outages-discussion lists are suddenly slow as molasses in January 
> too?
> 
> Checking some headers on messages that do make it through to outages, I’m 
> seeing delays up to an hour and a half.
> 
> I sent one messages to outages-discussion at 11:41 CDT (75 minutes ago) and 
> haven’t seen it show up yet.
> 
> Isn't it "outages-discuss", not "outages-discussion"?
> 
> Does it show up faster if you send to outages-discuss instead?
> 
> Matt

I was typing the list names from memory, not from my address book.

Another list member confirmed messages are showing up in the web archive for 
the outages lists but the delays are in delivery.


-Andy

Re: The Outage

2021-10-04 Thread Matthew Petach

On Mon, Oct 4, 2021 at 10:58 AM Andy Ringsmuth  wrote:

> I suppose it could just be me.
>
> Or it could be more than just me.
>
> Anyone else noticing that at the same time as the giant FB outage, both
> the outages and outages-discussion lists are suddenly slow as molasses in
> January too?
>
> Checking some headers on messages that do make it through to outages, I’m
> seeing delays up to an hour and a half.
>
> I sent one messages to outages-discussion at 11:41 CDT (75 minutes ago)
> and haven’t seen it show up yet.
>

Isn't it "outages-discuss", not "outages-discussion"?

Does it show up faster if you send to outages-discuss instead?

Matt

Re: massive facebook outage presently

2021-10-04 Thread Billy Croan

I know what this is.  They forgot to update the credit card on their
godaddy account and the domain lapsed.  I guess it will be facebook.info
when they get it back online.  The post mortem should be an interesting
read.

On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl  wrote:

> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
>> Here’s a screenshot:
>>
>>
>>
>>  -mel beckman
>>
>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>
>> 
>> https://downdetector.com/status/facebook/
>>
>> Normally not worth mentioning random $service having an outage here, but
>> this will undoubtedly generate a large volume of customer service calls.
>>
>> Appears to be failure in DNS resolution.
>>
>>

Re: massive facebook outage presently

2021-10-04 Thread Martin List-Petersen




Wishful thinking .. but hey, one is allowed to dream.

/M

On 04/10/2021 18:57, Jay Hennigan wrote:

On 10/4/21 10:35, Mel Beckman wrote:
Suspiciously, this comes the morning after Facebook whistleblower 
Frances Haugen disclosed on 60 Minutes that Facebook's own research 
shows that it chose to profit from misinformation and political 
unrest through deliberate escalation of conflicts. Occam’s razor says 
“When multiple causes are plausible, and CBS 60 Minutes is one of 
them, go with 60 Minutes.” :)


It could just be that after the 60 Minutes interview they've shut 
things down in order to divert all power to the shredders.



--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair - http://www.airwire.ie - Phone: 
091 395000
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in 
Ireland No. 508961


--
This email has been checked for viruses by AVG.
https://www.avg.com

Re: The Outage

2021-10-04 Thread Mike Hammett

Jared, would you be able to post any system load graphs showing what the mail 
server is like during these kinda of massive outages? 


With that many users on the mailing list, I can only imagine that it takes 
forever to process sending those messages out to tens of thousands of people at 
a time, especially when half of the messages send to the list are outside of 
the intent of the list. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Andy Ringsmuth"  
To: "NANOG"  
Sent: Monday, October 4, 2021 12:55:26 PM 
Subject: The Outage 

I suppose it could just be me. 

Or it could be more than just me. 

Anyone else noticing that at the same time as the giant FB outage, both the 
outages and outages-discussion lists are suddenly slow as molasses in January 
too? 

Checking some headers on messages that do make it through to outages, I’m 
seeing delays up to an hour and a half. 

I sent one messages to outages-discussion at 11:41 CDT (75 minutes ago) and 
haven’t seen it show up yet. 

 
Andy Ringsmuth 
5609 Harding Drive 
Lincoln, NE 68521-5831 
(402) 304-0083 
a...@andyring.com 

“Better even die free, than to live slaves.” - Frederick Douglas, 1863

Re: massive facebook outage presently


On 10/4/21 10:35, Mel Beckman wrote:
Suspiciously, this comes the morning after Facebook whistleblower 
Frances Haugen disclosed on 60 Minutes that Facebook's own research 
shows that it chose to profit from misinformation and political unrest 
through deliberate escalation of conflicts. Occam’s razor says “When 
multiple causes are plausible, and CBS 60 Minutes is one of them, go 
with 60 Minutes.” :)


It could just be that after the 60 Minutes interview they've shut things 
down in order to divert all power to the shredders.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

The Outage

2021-10-04 Thread Andy Ringsmuth

I suppose it could just be me.

Or it could be more than just me.

Anyone else noticing that at the same time as the giant FB outage, both the 
outages and outages-discussion lists are suddenly slow as molasses in January 
too?

Checking some headers on messages that do make it through to outages, I’m 
seeing delays up to an hour and a half.

I sent one messages to outages-discussion at 11:41 CDT (75 minutes ago) and 
haven’t seen it show up yet.


Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas, 1863

Re: massive facebook outage presently

2021-10-04 Thread Jonathan Kalbfeld via NANOG

With Facebook down, how are people doing their vaccine research?

It’s got to be more than just DNS.  

Jonathan Kalbfeld

office: +1 310 317 7933 
fax:+1 310 317 7901 
home:   +1 310 317 7909 
mobile: +1 310 227 1662 

ThoughtWave Technologies, Inc.
Studio City, CA 91604
https://thoughtwave.com 

View our network at 
https://bgp.he.net/AS54380 

+1 213 984 1000

> On Oct 4, 2021, at 10:12 AM, Michael Spears  wrote:
> 
> https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share
> 
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

Hi,

Oops, this was not supposed to go to the list, apologies for the clutter.

Thanks, 

Sabri

- On Oct 4, 2021, at 10:46 AM, Sabri Berisha sa...@cluecentral.net wrote:

> - On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha

- On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com 
wrote:

Hi Anne,

> On a related note, what do you think the scene is like in FB HQ right now?
> (shaking head)

Very quiet, as their offices are still closed for all but essentials :)

But, from experience I can tell you how that works. I assume Facebook works in a
similar manner as some of my previous employers. This assumption comes from the
fact that quite a number of my previous colleagues now work at Facebook in 
similar
roles.

First there is the question of detecting the outage. Obviously, Facebook will 
have
a monitoring/SRE team that continuously monitors 1000s of metrics. They observe
a number of metrics go down, and start to investigate. Most likely they will 
have
some sort of overall technical lead (let's call this the Technical Duty 
Officer),
that is responsible for the whole thing. Once the SRE team figured out where the
problem lies, they will alert the TDO. TDO will then hit that big red button and
send out alerts to the appropriate teams to jump on a bridge (let's call that 
the
Technical Crisis Bridge), to fix the issue. 

If done right, whomever was on call for that team will take the lead and 
interface
with adjoining teams, and other team members who are available to help out. 
Looking
at how long this outage lasts, there must be either something very broken, or 
they're
having trouble rolling back a change which was expected to not have impact.

Once the issue is fixed, the TDO will write a report and submit it to the 
Problem
Management group. This group will now contact the teams deemed responsible for 
the
outage. This team will no have an opportunity to explain themselves during a 
post-
mortem. Depending on the scale of the outage, the post-mortem can be a 10 minute
call on a bridge with a Problem Management manager, or in the hot seat during a
60 minute meeting with a bunch of execs.

I've been in that hot seat a few times. Not the most pleasurable experience. 
Perhaps
it's time for a new career :)

Thanks,

Sabri

RE: massive facebook outage presently

2021-10-04 Thread aaron1

Yes, embedded ISP CDN’s show a huge drop

 

-Aaron

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: Monday, October 4, 2021 11:22 AM
To: George Herbert ; nanog@nanog.org list 

Subject: Re: massive facebook outage presently

 

Considering the massive impact of this it would be interesting to see some 
traffic graphs from ISPs that have PNIs with Facebook, or high volume peering 
sessions across an IX, showing traffic to FB falling off a cliff. 

 

On Mon, Oct 4, 2021 at 12:16 PM George Herbert mailto:george.herb...@gmail.com> > wrote:

And WhatsApp and Instagram.  Twitter users nationwide agree anecdotally.

 

What I’m getting is DNS failure. 

 

-George 

Sent from my iPhone





On Oct 4, 2021, at 9:07 AM, Eric Kuhnke mailto:eric.kuh...@gmail.com> > wrote:



https://downdetector.com/status/facebook/

 

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls. 

 

Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman

Suspiciously, this comes the morning after Facebook whistleblower Frances 
Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose 
to profit from misinformation and political unrest through deliberate 
escalation of conflicts. Occam’s razor says “When multiple causes are 
plausible, and CBS 60 Minutes is one of them, go with 60 Minutes.” :)

 -mel

On Oct 4, 2021, at 10:30 AM, Michael Spears  wrote:


https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker


* m...@beckman.org (Mel Beckman) [Mon 04 Oct 2021, 18:23 CEST]:

Here’s a screenshot:

[cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001]


Please don't do this on the NANOG list.


-- Niels.

Re: massive facebook outage presently

2021-10-04 Thread Casey Russell via NANOG

>>  In other news worker productivity is up 100% today.

For everyone except IT workers.

Although, I suppose if you're just counting the number of tickets they can
quickly clear by sending out a "It's the internet, not us".  You could
count that as increased productivity.

Sincerely,
Casey Russell
Network Engineer

785-856-9809
2029 Becker Drive, Suite 282
Lawrence, Kansas 66047
XSEDE Campus Champion
Certified Software Carpentry Instructor
need support? 

On Mon, Oct 4, 2021 at 12:14 PM richey goldberg 
wrote:

> In other news worker productivity is up 100% today.
>
>
>
> -richey
>
>
>
> *From: *NANOG  on
> behalf of Jason Kuehl 
> *Date: *Monday, October 4, 2021 at 12:45 PM
> *To: *Mel Beckman 
> *Cc: *nanog@nanog.org list 
> *Subject: *Re: massive facebook outage presently
>
> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
>
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
> Here’s a screenshot:
>
>
>
> *Error! Filename not specified.*
>
>  -mel beckman
>
>
>
> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>
> 
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>
>

Re: massive facebook outage presently

2021-10-04 Thread Michael Spears

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app_source=share
Sent from my Verizon, Samsung Galaxy smartphoneGet Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Anne P. Mitchell, Esq.

> On Oct 4, 2021, at 10:34 AM, Adrian Minta  wrote:
> 
> https://www.youtube.com/watch?v=gmk673M5BoA

Because it's helpful to know what a link is about before clicking on it, 
Adrian's link goes to "The Onion Movie - Internet Down"

On a related note, what do you think the scene is like in FB HQ right now?  
(shaking head)

Anne

--
Outsource your email deliverability headaches to us, and get to the inbox!
www.GetToTheInbox.com

Anne P. Mitchell,  Esq.
CEO Get to the Inbox - We get you into the inbox!
Author: The Email Deliverability Handbook
Author: Section 6 of the Federal Email Marketing Law (CAN-SPAM)
Email Marketing Deliverability and Best Practices Expert
Board of Directors, Denver Internet Exchange
Chair Emeritus, Asilomar Microcomputer Workshop
Former Counsel: MAPS Anti-Spam Blacklist
Location: Boulder, Colorado

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher

They haven't completely dropped off, but the big subnets certainly have.
I'm only seeing 20-odd /24s from them via the DFZ, but everything larger
still directly.

On Mon, Oct 4, 2021 at 12:55 PM  wrote:

> Looks bigger than DNS. Some people are saying they’ve disappeared off the
> DFZ.
>
>
>
> *From:* NANOG  *On Behalf Of 
> *Dmitry
> Sherman
> *Sent:* Monday, October 4, 2021 12:10 PM
> *To:* Eric Kuhnke ; nanog@nanog.org list <
> nanog@nanog.org>
> *Subject:* RE: massive facebook outage presently
>
>
>
> same problem in Israel
>
>
>
>
>
> *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org
> ] *On Behalf Of *Eric Kuhnke
> *Sent:* Monday, 4 October 2021 19:03
> *To:* nanog@nanog.org list 
> *Subject:* massive facebook outage presently
>
>
>
> https://downdetector.com/status/facebook/
>
>
>
> Normally not worth mentioning random $service having an outage here, but
> this will undoubtedly generate a large volume of customer service calls.
>
>
>
> Appears to be failure in DNS resolution.
>
>
>

Re: massive facebook outage presently


On 10/4/21 09:27, Tom Beecher wrote:
Curious if there is a malicious angle after the 60 Minutes story last 
night.


Hanlon's razor.

"Never attribute to malice that which is adequately explained by stupidity"

Corollary - "But don't rule out malice."


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg

In other news worker productivity is up 100% today.

-richey

From: NANOG  on behalf of 
Jason Kuehl 
Date: Monday, October 4, 2021 at 12:45 PM
To: Mel Beckman 
Cc: nanog@nanog.org list 
Subject: Re: massive facebook outage presently
Looks like they run there own nameservers and I see the soa records are even 
missing.

On Mon, Oct 4, 2021, 12:23 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Here’s a screenshot:

Error! Filename not specified.
 -mel beckman

On Oct 4, 2021, at 9:06 AM, Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:

https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but this 
will undoubtedly generate a large volume of customer service calls.

Appears to be failure in DNS resolution.

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach

On Mon, Oct 4, 2021 at 9:47 AM Jason Kuehl  wrote:

> Looks like they run there own nameservers and I see the soa records are
> even missing.
>
> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>
>> Here’s a screenshot:
>>
>>
>>
>>  -mel beckman
>>
>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>
>> 
>> https://downdetector.com/status/facebook/
>>
>> Normally not worth mentioning random $service having an outage here, but
>> this will undoubtedly generate a large volume of customer service calls.
>>
>> Appears to be failure in DNS resolution.
>>
>>
If you check your BGP routing tables, you'll probably
find that it's not so much the SOA records that are
missing, as it is the prefixes to reach the DNS servers
entirely.

I suspect the DNS entries on the servers themselves
may look fine from inside facebook, leading to a slower
diagnostic and repair, as it's only from the outside world
the missing routing entries in the global table make the
problem so painfully visible.

Having the DNS team frantically checking their servers
may slow the resolution down, if it is indeed a BGP failure
rather than a DNS server failure situation, as it seems to
appear at the moment.  ^_^;

Matt

IRR for IX peers