Re: facebook outage

[Niels Bakker]

* jllee9...@gmail.com (John Lee) [Tue 05 Oct 2021, 01:06 CEST]:
I was seeing NXDOMAIN errors, so I wonder if they had a DNS outage 
of some sort??


Were you using host(1)? Please don't, and use dig(1) instead. There 
were as far as I know at no point NXDOMAINs being returned, but due to 
the SERVFAILs host(1) was silently appending your local search domain 
to your query which would lead to incorrect NXDOMAIN output.



-- Niels.

Re: facebook outage

[Bill Woodcock]

Ok, I lied, I’m still awake.

I got my first successful Facebook main page load at 23:13 UTC, for an overall 
duration of 8:33, or 513 minutes.  Multiplied by three billion users, that’s 
1.54 trillion person-minutes.

That’s a tera-lapse!

Have we had one of those before?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

[John Lee]

I was seeing NXDOMAIN errors, so I wonder if they had a DNS outage of some
sort??

On Mon, Oct 4, 2021 at 5:14 PM Bill Woodcock  wrote:

> They’re starting to pick themselves back up off the floor in the last two
> or three minutes.  A few answers getting out.  I imagine it’ll take a while
> before things stabilize, though.
>
> -Bill
>
>

Re: facebook outage

[Bill Woodcock]

> On Oct 5, 2021, at 12:16 AM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl  
>> wrote:
>> 
>> 
>> 
>> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
>> 
>> 
>>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>>> 
>>> 
>>> 
 On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
 
 They’re starting to pick themselves back up off the floor in the last two 
 or three minutes.  A few answers getting out.  I imagine it’ll take a 
 while before things stabilize, though.
>>> 
>>> nd we’re back:
>>> 
>>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>> 
>> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
>> fifteen minutes.
>> 
>> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
>> burns beyond that.
>> 
>> Maybe they’ll get one or two independent secondary authoritatives, so this 
>> doesn’t happen again.  :-)
>> 
>> 
>> We have had dns back for a while here but the site is still down. Not 
>> counting this as over yet.
> 
> Yeah, fair enough.  I went back and looked, and it looks like the BGP 
> withdrawals were around 16:40 UTC?  And as of 22:15 UTC, application-layer 
> services still aren’t up.  Which puts us at 6:35 thus far?

A.  It’s past midnight here, and my brain is failing to convert between 
three timezones accurately.  My apologies.  I’ll stop typing until I’ve had 
some sleep.  Good night.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

[Bill Woodcock]

> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl  
> wrote:
> 
> 
> 
> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
> 
> 
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last two 
> >> or three minutes.  A few answers getting out.  I imagine it’ll take a 
> >> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
> 
> 
> We have had dns back for a while here but the site is still down. Not 
> counting this as over yet.

Yeah, fair enough.  I went back and looked, and it looks like the BGP 
withdrawals were around 16:40 UTC?  And as of 22:15 UTC, application-layer 
services still aren’t up.  Which puts us at 6:35 thus far?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage (resolving)

[mike tancsa]

Getting the odd message through, but DNS looks good via their Toronto,
Canada pop


% traceroute -q1 -I a.dns.facebook.com
traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte
packets
 1  torix-core1-10G (67.43.129.248)  0.140 ms
 2  facebook-a.ip4.torontointernetxchange.net (206.108.35.2)  0.880 ms
 3  po103.psw02.yyz1.tfbnw.net (74.119.78.131)  0.365 ms
 4  173.252.67.57 (173.252.67.57)  0.341 ms
 5  edge-star-shv-01-yyz1.facebook.com (31.13.80.8)  0.263 ms
% traceroute6 -q1 -I a.dns.facebook.com
traceroute6 to star.c10r.facebook.com (2a03:2880:f00e:a:face:b00c:0:2)
from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets
 1  toronto-torix-6  0.133 ms
 2  facebook-a.ip6.torontointernetxchange.net  0.532 ms
 3  po103.psw01.yyz1.tfbnw.net  0.322 ms
 4  po1.msw1ah.01.yyz1.tfbnw.net  0.335 ms
 5  edge-star6-shv-01-yyz1.facebook.com  0.267 ms
% traceroute6 -q1 -I d.dns.facebook.com
traceroute6 to star.c10r.facebook.com (2a03:2880:f00e:a:face:b00c:0:2)
from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets
 1  toronto-torix-6  0.126 ms
 2  facebook-a.ip6.torontointernetxchange.net  0.673 ms
 3  po103.psw01.yyz1.tfbnw.net  0.349 ms
 4  po1.msw1ah.01.yyz1.tfbnw.net  0.332 ms
 5  edge-star6-shv-01-yyz1.facebook.com  0.264 ms
% traceroute -q1 -I d.dns.facebook.com
traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte
packets
 1  torix-core1-10G (67.43.129.248)  0.139 ms
 2  facebook-a.ip4.torontointernetxchange.net (206.108.35.2)  41.394 ms
 3  po103.psw02.yyz1.tfbnw.net (74.119.78.131)  0.285 ms
 4  173.252.67.57 (173.252.67.57)  0.309 ms
 5  edge-star-shv-01-yyz1.facebook.com (31.13.80.8)  0.211 ms
%

% host www.facebook.com a.ns.facebook.com
Using domain server:
Name: a.ns.facebook.com
Address: 2a03:2880:f0fc:c:face:b00c:0:35#53
Aliases:

www.facebook.com is an alias for star-mini.c10r.facebook.com.
% host -4 www.facebook.com a.ns.facebook.com
Using domain server:
Name: a.ns.facebook.com
Address: 129.134.30.12#53
Aliases:

www.facebook.com is an alias for star-mini.c10r.facebook.com.
%

On 10/4/2021 5:41 PM, Baldur Norddahl wrote:
>
>
> man. 4. okt. 2021 23.33 skrev Bill Woodcock  >:
>
>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  > wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  > wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in
> the last two or three minutes.  A few answers getting out.  I
> imagine it’ll take a while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com
>  @9.9.9.9 
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five
> hours and fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some
> third-degree burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives,
> so this doesn’t happen again.  :-)
>
>
>
> We have had dns back for a while here but the site is still down. Not
> counting this as over yet.
>
>
>

Re: facebook outage

[chris]

Hopefully this will show people they can enjoy life and survive without it
and that it will be just like myspace some day :)

On Mon, Oct 4, 2021 at 5:31 PM Jeff Shultz  wrote:

> Now they just need to get the site itself back up.
>
> On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock  wrote:
>
>>
>>
>> > On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> >
>> > They’re starting to pick themselves back up off the floor in the last
>> two or three minutes.  A few answers getting out.  I imagine it’ll take a
>> while before things stabilize, though.
>>
>> nd we’re back:
>>
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>>
>> ; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
>> ;; global options: +cmd
>> ;; Got answer:
>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>>
>> ;; OPT PSEUDOSECTION:
>> ; EDNS: version: 0, flags:; udp: 512
>> ;; QUESTION SECTION:
>> ;www.facebook.com.  IN  A
>>
>> ;; ANSWER SECTION:
>> www.facebook.com.   3420IN  CNAME
>> star-mini.c10r.facebook.com.
>> star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35
>>
>> ;; Query time: 13 msec
>> ;; SERVER: 9.9.9.9#53(9.9.9.9)
>> ;; WHEN: Mon Oct 04 23:20:41 CEST 2021
>> ;; MSG SIZE  rcvd: 90
>>
>>
>> -Bill
>>
>>
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
> [image:
> https://www.instagram.com/sctc_sctc/]
> 
> 
> 
>
>
>
>
>
>
>
>  This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. 
>

Re: facebook outage

[Bill Woodcock]

> On Oct 4, 2021, at 11:50 PM, Ryan Brooks  wrote:
> DNS was a victim in this outage, not the cause.

You are absolutely correct.  However, people who don’t have this problem avoid 
having this problem by not putting all their DNS eggs in one basket.

And then forgetting where they put the basket.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

[Michael Thomas]

On 10/4/21 2:41 PM, Baldur Norddahl wrote:



man. 4. okt. 2021 23.33 skrev Bill Woodcock >:




> On Oct 4, 2021, at 11:21 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:
>
>
>
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:
>>
>> They’re starting to pick themselves back up off the floor in
the last two or three minutes.  A few answers getting out.  I
imagine it’ll take a while before things stabilize, though.
>
> nd we’re back:
>
> WoodyNet-2:.ssh woody$ dig www.facebook.com
 @9.9.9.9 

So that was, what…  15:50 UTC to 21:05 UTC, more or less… five
hours and fifteen minutes.

That’s a lot of hair burnt all the way to the scalp, and some
third-degree burns beyond that.

Maybe they’ll get one or two independent secondary authoritatives,
so this doesn’t happen again.  :-)



We have had dns back for a while here but the site is still down. Not 
counting this as over yet.





I got a page to load. Probably trickling out.

Mike

Re: facebook outage

[Patrick W. Gilmore]

On Oct 4, 2021, at 5:30 PM, Bill Woodcock  wrote:
> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>>> 
>>> They’re starting to pick themselves back up off the floor in the last two 
>>> or three minutes.  A few answers getting out.  I imagine it’ll take a while 
>>> before things stabilize, though.
>> 
>> nd we’re back:
>> 
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)

If by “independent” you mean “3rd party” (e.g. DynDNS), not sure what an 
external secondary would have done here. While their BGP was misbehaving, the 
app would not work even if you had a static DNS entry.

And while using external / 3rd party secondaries is likely a good idea for many 
companies, almost none of the largest do this. These companies view it as a 
control issue. Giving someone outside your own employees the ability to change 
a DNS name is, frankly, giving another company the ability to take you down.

Taking a sample of FB, cisco, Amazon, NF, Dell, Akamai, Google, MS, CF, only 2 
use 3rd party resolvers.
* NF uses only awsdns, so same problem, just moved to another company they do 
not control.
* Amazon uses Ultra & Dyn. (Anyone else amused amazon.com has no authorities on 
Route 53? At least not from my vantage point.)

That said, plenty of what people may call “big” companies do use 3rd parties, 
e.g. IBM, PayPal, Juniper.

You want to use a 3rd party DNS, go for it. There are lots of reasons to do it. 
But it is not a panacea, and there are reasons not to.

-- 
TTFN,
patrick

Re: facebook outage

[Jeff Shultz]

On Mon, Oct 4, 2021 at 2:49 PM Baldur Norddahl 
wrote:

>
>
> man. 4. okt. 2021 23.33 skrev Bill Woodcock :
>
>>
>>
>> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> >
>> >
>> >
>> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> >>
>> >> They’re starting to pick themselves back up off the floor in the last
>> two or three minutes.  A few answers getting out.  I imagine it’ll take a
>> while before things stabilize, though.
>> >
>> > nd we’re back:
>> >
>> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>>
>> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and
>> fifteen minutes.
>>
>> That’s a lot of hair burnt all the way to the scalp, and some
>> third-degree burns beyond that.
>>
>> Maybe they’ll get one or two independent secondary authoritatives, so
>> this doesn’t happen again.  :-)
>>
>
>
> We have had dns back for a while here but the site is still down. Not
> counting this as over yet.
>
>
>
>
I'm getting part of my news feed and notifications. Can't post yet, and
clicking on something usually sends you back to the "Something is broken"
message.

So, no - it's not back up yet. But it's starting to twitch

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: facebook outage

[Ryan Brooks]

> On Oct 4, 2021, at 4:30 PM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
>> 
>> 
>> 
>>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>>> 
>>> They’re starting to pick themselves back up off the floor in the last two 
>>> or three minutes.  A few answers getting out.  I imagine it’ll take a while 
>>> before things stabilize, though.
>> 
>> nd we’re back:
>> 
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
> 

DNS was a victim in this outage, not the cause.

>-Bill

Re: facebook outage

[Baldur Norddahl]

man. 4. okt. 2021 23.33 skrev Bill Woodcock :

>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last
> two or three minutes.  A few answers getting out.  I imagine it’ll take a
> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and
> fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree
> burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this
> doesn’t happen again.  :-)
>


We have had dns back for a while here but the site is still down. Not
counting this as over yet.

Re: facebook outage

[Mike Lyon]

Can't wait to see the RFO!

-Mike

On Mon, Oct 4, 2021 at 2:35 PM Bill Woodcock  wrote:
>
>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last two 
> >> or three minutes.  A few answers getting out.  I imagine it’ll take a 
> >> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
> fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree 
> burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this 
> doesn’t happen again.  :-)
>
> -Bill
>


-- 
Mike Lyon
mike.l...@gmail.com
http://www.linkedin.com/in/mlyon

Re: facebook outage

[Bill Woodcock]

> On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> 
> 
> 
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
>> 
>> They’re starting to pick themselves back up off the floor in the last two or 
>> three minutes.  A few answers getting out.  I imagine it’ll take a while 
>> before things stabilize, though.
> 
> nd we’re back:
> 
> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9

So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and 
fifteen minutes.

That’s a lot of hair burnt all the way to the scalp, and some third-degree 
burns beyond that.

Maybe they’ll get one or two independent secondary authoritatives, so this 
doesn’t happen again.  :-)

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

[Jeff Shultz]

Now they just need to get the site itself back up.

On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock  wrote:

>
>
> > On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >
> > They’re starting to pick themselves back up off the floor in the last
> two or three minutes.  A few answers getting out.  I imagine it’ll take a
> while before things stabilize, though.
>
> nd we’re back:
>
> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> ; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;www.facebook.com.  IN  A
>
> ;; ANSWER SECTION:
> www.facebook.com.   3420IN  CNAME
> star-mini.c10r.facebook.com.
> star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35
>
> ;; Query time: 13 msec
> ;; SERVER: 9.9.9.9#53(9.9.9.9)
> ;; WHEN: Mon Oct 04 23:20:41 CEST 2021
> ;; MSG SIZE  rcvd: 90
>
>
> -Bill
>
>

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: facebook outage

[Bill Woodcock]

> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> 
> They’re starting to pick themselves back up off the floor in the last two or 
> three minutes.  A few answers getting out.  I imagine it’ll take a while 
> before things stabilize, though.

nd we’re back:

WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9

; <<>> DiG 9.10.6 <<>> www.facebook.com @9.9.9.9
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32839
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.facebook.com.  IN  A

;; ANSWER SECTION:
www.facebook.com.   3420IN  CNAME   star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 6  IN  A   157.240.19.35

;; Query time: 13 msec
;; SERVER: 9.9.9.9#53(9.9.9.9)
;; WHEN: Mon Oct 04 23:20:41 CEST 2021
;; MSG SIZE  rcvd: 90


-Bill



signature.asc
Description: Message signed with OpenPGP

Re: facebook outage

[Bill Woodcock]

They’re starting to pick themselves back up off the floor in the last two or 
three minutes.  A few answers getting out.  I imagine it’ll take a while before 
things stabilize, though.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Facebook outage

[Jay Farrell via NANOG]

On Sun, Apr 14, 2019 at 7:33 AM Siyuan Miao  wrote:

> Dear community,
>
> It seems that Facebook network is partially down.
>

I wouldn't say partially. And just a month and a day since their 10+ hour
record-breaking outage. Of course FB will downplay it in their eventual RFO
as "some users" had difficulty connecting. ;-)

Re: Facebook outage

[Dovid Bender]

That should have read FB not DB.

On Sun, Apr 14, 2019, 07:41 Dovid Bender  wrote:

> It's all over Twitter. DB, What's app and Instagram are all having issues.
>
> On Sun, Apr 14, 2019, 07:32 Siyuan Miao  wrote:
>
>> Dear community,
>>
>> It seems that Facebook network is partially down.
>>
>> Here's a traceroute from some locations to fbcdn Hong Kong
>> (157.240.15.37):
>>
>>  Host
>>Loss%   Snt   Last   Avg  Best  Wrst StDev
>>  1. ???
>>  2. fra-in8-01sw.voxility.net
>> 0.6%   1810.5   0.6   0.4   4.9   0.5
>>  3. fra-in8-01c.voxility.net
>>  0.0%   181   14.7   3.7   0.3  44.1   8.5
>>  4. fra-in3-02gw.voxility.net
>> 0.0%   1810.9   3.5   0.7 195.2  14.5
>>  5. fra-eq5-02gw.voxility.net
>> 0.0%   1810.6   0.5   0.3   0.7   0.0
>>  6. fra-eq5-03gw.voxility.net
>> 0.0%   1810.6   0.7   0.4   6.6   0.7
>>  7. ae10.pr02.fra5.tfbnw.net
>>  0.0%   1811.2   3.5   0.8  35.0   5.2
>>  8. ???
>>  9. ae121.ar03.fra2.tfbnw.net
>> 0.0%   1814.4   2.1   1.0  21.1   2.2
>> 10. ae32.bb02.fra2.tfbnw.net
>>  0.0%   1815.1   4.6   1.4  14.0   2.7
>> 11. ae15.bb03.odn2.tfbnw.net
>>  0.0%   181   19.2  19.8  18.5  26.3   1.2
>> 12. ae12.dr03.odn2.tfbnw.net
>> 92.8%   181   27.8  24.4  18.3  43.4   8.4
>> 13. ???
>>
>>  Host
>>Loss%   Snt   Last   Avg  Best  Wrst StDev
>>  1. lax-cs1-02sw.voxility.net
>> 0.0%   1940.3   2.5   0.2  36.9   5.1
>>  2. lax-cs1-01c.voxility.net
>>  0.0%   1946.1   2.8   0.2  47.3   6.4
>>  3. lax-cs1-02gw.voxility.net
>> 0.0%   1940.2   0.1   0.1   0.3   0.0
>>  4. ce-0-13-0-1.r00.lsanca07.us.bb.gin.ntt.net
>>  0.0%   1940.9   0.5   0.4   1.0   0.0
>>  5. ae-3.r01.lsanca07.us.bb.gin.ntt.net
>> 0.0%   1940.5   0.5   0.4   0.9   0.0
>>  6. ae-0.tata-communications.lsanca07.us.bb.gin.ntt.net
>> 0.0%   1940.7   1.0   0.4  28.8   2.8
>>  7. if-et-20-2.hcore2.kv8-chiba.as6453.net
>>  0.0%   194  158.6 158.8 158.4 175.5   1.8
>>  8. if-et-1-2.hcore1.kv8-chiba.as6453.net
>> 0.0%   194  152.7 152.8 152.5 155.8   0.2
>>  9. if-ae-16-2.tcore1.hk2-hong-kong.as6453.net
>>  0.0%   194  162.1 163.0 162.0 213.5   4.9
>> 10. ???
>>
>> What do you think ?
>>
>>

Re: Facebook outage

[Dovid Bender]

It's all over Twitter. DB, What's app and Instagram are all having issues.

On Sun, Apr 14, 2019, 07:32 Siyuan Miao  wrote:

> Dear community,
>
> It seems that Facebook network is partially down.
>
> Here's a traceroute from some locations to fbcdn Hong Kong (157.240.15.37):
>
>  Host
>  Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. ???
>  2. fra-in8-01sw.voxility.net
> 0.6%   1810.5   0.6   0.4   4.9   0.5
>  3. fra-in8-01c.voxility.net
>0.0%   181   14.7   3.7   0.3  44.1   8.5
>  4. fra-in3-02gw.voxility.net
> 0.0%   1810.9   3.5   0.7 195.2  14.5
>  5. fra-eq5-02gw.voxility.net
> 0.0%   1810.6   0.5   0.3   0.7   0.0
>  6. fra-eq5-03gw.voxility.net
> 0.0%   1810.6   0.7   0.4   6.6   0.7
>  7. ae10.pr02.fra5.tfbnw.net
>0.0%   1811.2   3.5   0.8  35.0   5.2
>  8. ???
>  9. ae121.ar03.fra2.tfbnw.net
> 0.0%   1814.4   2.1   1.0  21.1   2.2
> 10. ae32.bb02.fra2.tfbnw.net
>0.0%   1815.1   4.6   1.4  14.0   2.7
> 11. ae15.bb03.odn2.tfbnw.net
>0.0%   181   19.2  19.8  18.5  26.3   1.2
> 12. ae12.dr03.odn2.tfbnw.net
>   92.8%   181   27.8  24.4  18.3  43.4   8.4
> 13. ???
>
>  Host
>  Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. lax-cs1-02sw.voxility.net
> 0.0%   1940.3   2.5   0.2  36.9   5.1
>  2. lax-cs1-01c.voxility.net
>0.0%   1946.1   2.8   0.2  47.3   6.4
>  3. lax-cs1-02gw.voxility.net
> 0.0%   1940.2   0.1   0.1   0.3   0.0
>  4. ce-0-13-0-1.r00.lsanca07.us.bb.gin.ntt.net
>0.0%   1940.9   0.5   0.4   1.0   0.0
>  5. ae-3.r01.lsanca07.us.bb.gin.ntt.net
> 0.0%   1940.5   0.5   0.4   0.9   0.0
>  6. ae-0.tata-communications.lsanca07.us.bb.gin.ntt.net
> 0.0%   1940.7   1.0   0.4  28.8   2.8
>  7. if-et-20-2.hcore2.kv8-chiba.as6453.net
>0.0%   194  158.6 158.8 158.4 175.5   1.8
>  8. if-et-1-2.hcore1.kv8-chiba.as6453.net
> 0.0%   194  152.7 152.8 152.5 155.8   0.2
>  9. if-ae-16-2.tcore1.hk2-hong-kong.as6453.net
>0.0%   194  162.1 163.0 162.0 213.5   4.9
> 10. ???
>
> What do you think ?
>
>

Re: Facebook outage?

[Jay Ashworth]

- Original Message -
 From: Larry Sheldon larryshel...@cox.net

 On 1/27/2015 00:47, Damien Burke wrote:
  Facebook outage? Everyone panic!
 
  https://twitter.com/search?q=facebooksrc=typd
 
 Let the record show that I noticed it quite a while ago, but did NOT
 go for first NANOG mention.

Proud of you, Larry.

Let the record show that *I* haven't seen any outages all day, from Sprint
LTE.

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth  Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Facebook outage?

[Owen DeLong]

 On Jan 27, 2015, at 10:15 , Larry Sheldon larryshel...@cox.net wrote:
 
 On 1/27/2015 09:02, Roy wrote:
 
 According to one joker, the crash was caused by too many pictures of the
 Northeast blizzard :-)
 
 Cat-picture server went down.

Putting those two things together, I think it was because really, nobody wants 
to see a bunch of frozen pussy.


Get your minds out of the gutter.

Owen

Re: Facebook outage?

[Colin Johnston]

implement service routers for pop machines using cbac checking and acl for 
private address range spoofing.
block china ranges since never respond to abuse reports.
move on

Colin

 On 27 Jan 2015, at 07:23, Ken Chase m...@sizone.org wrote:
 
 cable was replugged, insta/fb back up here.
 
 /kc
 
 On Tue, Jan 27, 2015 at 02:04:58AM -0500, Zachary said:
 Seems unlikely, probably taking credit for someone tripping over a cable.
 
 -- 
 Ken Chase - m...@sizone.org Toronto

Re: Facebook outage?

[Anthony Jeffries]

It is working here in rural Oregon as well. Kudos to the Facebook team
for such a quick recovery.

On Tue, 2015-01-27 at 01:13 -0600, Larry Sheldon wrote:
 On 1/27/2015 00:58, Larry Sheldon wrote:
  On 1/27/2015 00:47, Damien Burke wrote:
  Facebook outage? Everyone panic!
 
  https://twitter.com/search?q=facebooksrc=typd
 
  Let the record show that I noticed it quite a while ago, but did NOT go
  for first NANOG mention.
 
 It is back up in Omaha.
 

Re: Facebook outage?

[Roy]

According to one joker, the crash was caused by too many pictures of the 
Northeast blizzard :-)

Re: Facebook outage?

[Larry Sheldon]

On 1/27/2015 09:02, Roy wrote:


According to one joker, the crash was caused by too many pictures of the
Northeast blizzard :-)


Cat-picture server went down.


--
The unique Characteristics of System Administrators:

The fact that they are infallible; and,

The fact that they learn from their mistakes.


Quis custodiet ipsos custodes

Re: Facebook outage?

[Larry Sheldon]

On 1/27/2015 00:58, Larry Sheldon wrote:

On 1/27/2015 00:47, Damien Burke wrote:

Facebook outage? Everyone panic!

https://twitter.com/search?q=facebooksrc=typd


Let the record show that I noticed it quite a while ago, but did NOT go
for first NANOG mention.


It is back up in Omaha.

--
The unique Characteristics of System Administrators:

The fact that they are infallible; and,

The fact that they learn from their mistakes.


Quis custodiet ipsos custodes

RE: Facebook outage?

[John van Oppen]

Dead here at AS11404 from all locations where we PNI or public peer...   

must be bad over there, v4 dies at their edge, v6 makes it in but no page loads.

John

Re: Facebook outage?

[Larry Sheldon]

On 1/27/2015 00:47, Damien Burke wrote:

Facebook outage? Everyone panic!

https://twitter.com/search?q=facebooksrc=typd


Let the record show that I noticed it quite a while ago, but did NOT go 
for first NANOG mention.


--
The unique Characteristics of System Administrators:

The fact that they are infallible; and,

The fact that they learn from their mistakes.


Quis custodiet ipsos custodes

Re: Facebook outage?

[Tim Raphael]

And it appears to be back for me.

- Tim


 On 27 Jan 2015, at 3:08 pm, Tim Raphael raphael.timo...@gmail.com wrote:
 
 Instagram used to use Amazon AWS before being purchased by Facebook.
 There has been a slow migration onto FB infrastructure, so yes, a mixture of 
 addresses like that makes sense.
 
 - Tim
 
 
 On 27 Jan 2015, at 2:58 pm, Christopher Morrow morrowc.li...@gmail.com 
 wrote:
 
 On Tue, Jan 27, 2015 at 1:56 AM, Jason Canady ja...@unlimitednet.us wrote:
 Instagram appears to be down as well, but that would make sense since they 
 are part of Facebook.
 
 
 $ dig +short facebook.com
 173.252.120.6
 
 NetRange:   173.252.64.0 - 173.252.127.255
 CIDR:   173.252.64.0/18
 NetName:FACEBOOK-INC
 
 
 but
 $ dig +short instagram.com
 54.209.14.128
 107.23.173.176
 54.175.77.206
 54.208.246.103
 107.23.166.70
 54.236.148.28
 54.209.197.196
 54.236.177.12
 
 
 those are amazon addresses... err, not sure the connection makes sense 
 though?
 
 -chris
 

RE: Facebook outage?

[Damien Burke]

I hear that AIM and hipchat is also having issues. 

Any other major company down too?

-Original Message-
From: John van Oppen [mailto:jvanop...@spectrumnet.us] 
Sent: Monday, January 26, 2015 10:49 PM
To: Damien Burke; nanog@nanog.org
Subject: RE: Facebook outage?

Dead here at AS11404 from all locations where we PNI or public peer...   

must be bad over there, v4 dies at their edge, v6 makes it in but no page loads.

John

Re: Facebook outage?

[Zachary]

Seems unlikely, probably taking credit for someone tripping over a cable.
On Jan 27, 2015 2:01 AM, Trent Farrell tfarr...@riotgames.com wrote:

 https://twitter.com/LizardMafia/status/559963134006292481

 On Mon, Jan 26, 2015 at 10:50 PM, Damien Burke dam...@supremebytes.com
 wrote:

  I hear that AIM and hipchat is also having issues.
 
  Any other major company down too?
 
  -Original Message-
  From: John van Oppen [mailto:jvanop...@spectrumnet.us]
  Sent: Monday, January 26, 2015 10:49 PM
  To: Damien Burke; nanog@nanog.org
  Subject: RE: Facebook outage?
 
  Dead here at AS11404 from all locations where we PNI or public peer...
 
  must be bad over there, v4 dies at their edge, v6 makes it in but no page
  loads.
 
  John
 



 --

 *Trent Farrell*

 *Riot Games*

 *IP Network Engineer*

 E: tfarr...@riotgames.com | IE:  +353 83 446 6809 | US: +1 424 285 9825

 Summoner name: Foro

Re: Facebook outage?

[Jason Canady]

Instagram appears to be down as well, but that would make sense since they are 
part of Facebook. 

On Jan 27, 2015, at 1:50, Damien Burke dam...@supremebytes.com wrote:

 I hear that AIM and hipchat is also having issues. 
 
 Any other major company down too?
 
 -Original Message-
 From: John van Oppen [mailto:jvanop...@spectrumnet.us] 
 Sent: Monday, January 26, 2015 10:49 PM
 To: Damien Burke; nanog@nanog.org
 Subject: RE: Facebook outage?
 
 Dead here at AS11404 from all locations where we PNI or public peer...   
 
 must be bad over there, v4 dies at their edge, v6 makes it in but no page 
 loads.
 
 John

Re: Facebook outage?

[Chaim Rieger]

Hacking group Lizard Sqaud claims to have taken down @facebook
https://twitter.com/facebook/@instagram
https://twitter.com/instagram/@Tinder
https://twitter.com/Tinder/@aim https://twitter.com/aim/@Myspace
https://twitter.com/Myspace/

Re: Facebook outage?

[Trent Farrell]

https://twitter.com/LizardMafia/status/559963134006292481

On Mon, Jan 26, 2015 at 10:50 PM, Damien Burke dam...@supremebytes.com
wrote:

 I hear that AIM and hipchat is also having issues.

 Any other major company down too?

 -Original Message-
 From: John van Oppen [mailto:jvanop...@spectrumnet.us]
 Sent: Monday, January 26, 2015 10:49 PM
 To: Damien Burke; nanog@nanog.org
 Subject: RE: Facebook outage?

 Dead here at AS11404 from all locations where we PNI or public peer...

 must be bad over there, v4 dies at their edge, v6 makes it in but no page
 loads.

 John




-- 

*Trent Farrell*

*Riot Games*

*IP Network Engineer*

E: tfarr...@riotgames.com | IE:  +353 83 446 6809 | US: +1 424 285 9825

Summoner name: Foro

Re: Facebook outage?

[Bryan Tong]

Dead here from a close peering link.

  1 10ge11-3.core1.lax1.he.net (65.49.27.149) [AS 6939] 4 msec 12 msec 0
msec
  2 10ge1-3.core1.lax2.he.net (72.52.92.122) [AS 6939] 0 msec 8 msec 4 msec
  3 any2ix.coresite.com (206.72.210.161) 0 msec 4 msec 0 msec
  4 be2.bb01.lax1.tfbnw.net (31.13.30.24) [AS 32934] [MPLS: Label 20180 Exp
2] 64 msec 68 msec
be2.bb02.lax1.tfbnw.net (31.13.30.26) [AS 32934] [MPLS: Label 18881 Exp
2] 56 msec
  5 ae31.bb01.atl1.tfbnw.net (204.15.20.76) [AS 32934] [MPLS: Label 688077
Exp 2] 100 msec
ae12.bb01.atl1.tfbnw.net (31.13.28.109) [AS 32934] [MPLS: Label 706669
Exp 2] 56 msec
ae31.bb01.atl1.tfbnw.net (204.15.20.76) [AS 32934] [MPLS: Label 687613
Exp 2] 76 msec
  6 be25.bb01.frc3.tfbnw.net (204.15.23.53) [AS 32934] [MPLS: Label 16526
Exp 2] 60 msec 64 msec 72 msec
  7 ae60.dr03.frc3.tfbnw.net (74.119.79.11) [AS 32934] 96 msec
ae60.dr01.frc3.tfbnw.net (204.15.23.247) [AS 32934] 56 msec
ae60.dr03.frc3.tfbnw.net (74.119.79.11) [AS 32934] 64 msec
  8  *  *  *
  9  *  *  *


On Mon, Jan 26, 2015 at 11:57 PM, Christopher Morrow 
morrowc.li...@gmail.com wrote:

 snow, it's a terrible thing.

 On Tue, Jan 27, 2015 at 1:50 AM, Damien Burke dam...@supremebytes.com
 wrote:
  I hear that AIM and hipchat is also having issues.
 
  Any other major company down too?
 
  -Original Message-
  From: John van Oppen [mailto:jvanop...@spectrumnet.us]
  Sent: Monday, January 26, 2015 10:49 PM
  To: Damien Burke; nanog@nanog.org
  Subject: RE: Facebook outage?
 
  Dead here at AS11404 from all locations where we PNI or public peer...
 
  must be bad over there, v4 dies at their edge, v6 makes it in but no
 page loads.
 
  John




-- 
eSited LLC
(701) 390-9638

Re: Facebook outage?

[Tim Raphael]

Instagram used to use Amazon AWS before being purchased by Facebook.
There has been a slow migration onto FB infrastructure, so yes, a mixture of 
addresses like that makes sense.

- Tim


 On 27 Jan 2015, at 2:58 pm, Christopher Morrow morrowc.li...@gmail.com 
 wrote:
 
 On Tue, Jan 27, 2015 at 1:56 AM, Jason Canady ja...@unlimitednet.us wrote:
 Instagram appears to be down as well, but that would make sense since they 
 are part of Facebook.
 
 
 $ dig +short facebook.com
 173.252.120.6
 
 NetRange:   173.252.64.0 - 173.252.127.255
 CIDR:   173.252.64.0/18
 NetName:FACEBOOK-INC
 
 
 but
 $ dig +short instagram.com
 54.209.14.128
 107.23.173.176
 54.175.77.206
 54.208.246.103
 107.23.166.70
 54.236.148.28
 54.209.197.196
 54.236.177.12
 
 
 those are amazon addresses... err, not sure the connection makes sense though?
 
 -chris

Re: Facebook outage?

[Christopher Morrow]

snow, it's a terrible thing.

On Tue, Jan 27, 2015 at 1:50 AM, Damien Burke dam...@supremebytes.com wrote:
 I hear that AIM and hipchat is also having issues.

 Any other major company down too?

 -Original Message-
 From: John van Oppen [mailto:jvanop...@spectrumnet.us]
 Sent: Monday, January 26, 2015 10:49 PM
 To: Damien Burke; nanog@nanog.org
 Subject: RE: Facebook outage?

 Dead here at AS11404 from all locations where we PNI or public peer...

 must be bad over there, v4 dies at their edge, v6 makes it in but no page 
 loads.

 John

Re: Facebook outage?

[Christopher Morrow]

On Tue, Jan 27, 2015 at 1:56 AM, Jason Canady ja...@unlimitednet.us wrote:
 Instagram appears to be down as well, but that would make sense since they 
 are part of Facebook.


$ dig +short facebook.com
173.252.120.6

NetRange:   173.252.64.0 - 173.252.127.255
CIDR:   173.252.64.0/18
NetName:FACEBOOK-INC


but
$ dig +short instagram.com
54.209.14.128
107.23.173.176
54.175.77.206
54.208.246.103
107.23.166.70
54.236.148.28
54.209.197.196
54.236.177.12


those are amazon addresses... err, not sure the connection makes sense though?

-chris

Re: Facebook outage?

[Ken Chase]

Some busted links there, but

http://www.bbc.co.uk/newsbeat/30306319

/kc

On Mon, Jan 26, 2015 at 10:58:21PM -0800, Chaim Rieger said:
  Hacking group Lizard Sqaud claims to have taken down @facebook
  https://twitter.com/facebook/@instagram
  https://twitter.com/instagram/@Tinder
  https://twitter.com/Tinder/@aim https://twitter.com/aim/@Myspace
  https://twitter.com/Myspace/

--
Ken Chase - m...@sizone.org Toronto

Re: Facebook outage?

[Ken Chase]

cable was replugged, insta/fb back up here.

/kc

On Tue, Jan 27, 2015 at 02:04:58AM -0500, Zachary said:
  Seems unlikely, probably taking credit for someone tripping over a cable.

-- 
Ken Chase - m...@sizone.org Toronto

Re: Facebook outage?

[Ken Chase]

down from toronto. instagram too, of course.

/kc
-- 
Ken Chase - m...@sizone.org Toronto

Re: Facebook outage?

[Suresh Ramasubramanian]

It is back now fwiw
On Jan 27, 2015 12:18 PM, Damien Burke dam...@supremebytes.com wrote:

 Facebook outage? Everyone panic!

 https://twitter.com/search?q=facebooksrc=typd

 -Damien

Re: Facebook outage?

[Gary Josack]

The js console for hipchat shows tons of connection errors and 503s to
facebook.com. It's like just a facebook outage and breaking sites that have
facebook login options.

On Mon, Jan 26, 2015 at 10:58 PM, Chaim Rieger chaim.rie...@gmail.com
wrote:

 Hacking group Lizard Sqaud claims to have taken down @facebook
 https://twitter.com/facebook/@instagram
 https://twitter.com/instagram/@Tinder
 https://twitter.com/Tinder/@aim https://twitter.com/aim/@Myspace
 https://twitter.com/Myspace/