RE: Meta outage
On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun wrote: >> What I found intriguing was that I was logged out by Google Docs at >> the same moment FB logged me out. Downdetector showed a number of >> other supposedly unrelated services with large outage report spikes >> at roughly the same time. > I was logged out of Honeywell's Total Connect Comfort site (remote > thermostat control) at the same time FB logged me out. I'm not using > OAUTH logins anywhere. > > I had recently changed SSID and the thermostat wasn't accepting remote > commands. I thought maybe the SSID change had broken it and had just > deleted the device from the TCC site to try adding it back when I was > logged out. > > It's funny timing because earlier today I was telling an employee how > we don't like to have our maintenance overlap with vendor's > maintenance/repair window because when something breaks while we are > making changes we can't always tell who is at fault. Interesting. I have a script that makes a call to TCC every so often to read temperature data, and it failed from 8:50AM PST to 10:30AM PST. Ray
RE: Meta outage
On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun wrote: > What I found intriguing was that I was logged out by Google Docs at the same > moment FB logged me out. Downdetector showed a number of other supposedly > unrelated services with large outage report spikes at roughly the same time. I was logged out of Honeywell's Total Connect Comfort site (remote thermostat control) at the same time FB logged me out. I'm not using OAUTH logins anywhere. I had recently changed SSID and the thermostat wasn't accepting remote commands. I thought maybe the SSID change had broken it and had just deleted the device from the TCC site to try adding it back when I was logged out. It's funny timing because earlier today I was telling an employee how we don't like to have our maintenance overlap with vendor's maintenance/repair window because when something breaks while we are making changes we can't always tell who is at fault.
Re: Meta outage
On Tue, 05 Mar 2024 11:06:11 -0500, Jay Ashworth wrote: >It's making the general press this hour so of course you already know about it >but my question is this: who peers with meta and have you seen BGP sessions >drop or the like? Do you operate meta CDN nodes in your network? Are they >screaming for help? > >This doesn't sound like it's a network layer problem but I'm curious. What I found intriguing was that I was logged out by Google Docs at the same moment FB logged me out. Downdetector showed a number of other supposedly unrelated services with large outage report spikes at roughly the same time. mdr -- "There are no laws here, only agreements." -- Masahiko
Re: Meta outage
That's from 2010. You can see if you look at the non-mobile version of the page (requires login though): https://www.facebook.com/notes/10158791436142200/ [image: image.png] On Tue, Mar 5, 2024 at 1:32 PM Jorge Amodio wrote: > > https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F > > - Jorge (mobile) > > > On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG > wrote: > > > > Does meta keep a board somewhere to tell the world it’s down? > > > > *From:* NANOG *On Behalf Of *Jay > Ashworth > *Sent:* Tuesday, March 05, 2024 11:06 AM > *To:* nanog@nanog.org > *Subject:* Meta outage > > > > WARNING: This message originated outside of Ford Motor Company. Use > caution when opening attachments, clicking links, or responding. > > > > It's making the general press this hour so of course you already know > about it but my question is this: who peers with meta and have you seen BGP > sessions drop or the like? Do you operate meta CDN nodes in your network? > Are they screaming for help? > > This doesn't sound like it's a network layer problem but I'm curious. > > Cheers, > -- jra > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > >
Re: Meta outage
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F - Jorge (mobile) > On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG wrote: > > > Does meta keep a board somewhere to tell the world it’s down? > > From: NANOG On Behalf Of Jay > Ashworth > Sent: Tuesday, March 05, 2024 11:06 AM > To: nanog@nanog.org > Subject: Meta outage > > WARNING: This message originated outside of Ford Motor Company. Use caution > when opening attachments, clicking links, or responding. > > It's making the general press this hour so of course you already know about > it but my question is this: who peers with meta and have you seen BGP > sessions drop or the like? Do you operate meta CDN nodes in your network? Are > they screaming for help? > > This doesn't sound like it's a network layer problem but I'm curious. > > Cheers, > -- jra > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [External] Re: Meta outage
On Tue, Mar 5, 2024 at 10:38 AM Chris K wrote: > see: Status and outages of Meta business products (metastatus.com) Things seem to be returning now, aside from the status page, which still appears blank. Interesting. -- Hunter Fuller (they) Router Jockey VBH M-1C +1 256 824 5331 Office of Information Technology The University of Alabama in Huntsville Network Engineering
Re: Meta outage
On Tue, Mar 05, 2024 at 04:23:42PM +, Kain, Becki (.) via NANOG wrote https://metastatus.com/ a message of 210 lines which said: > Does meta keep a board somewhere to tell the world it’s down? https://metastatus.com/
Re: Meta outage
see: Status and outages of Meta business products (metastatus.com)<https://metastatus.com/> From: NANOG on behalf of Kain, Becki (.) via NANOG Sent: Tuesday, March 5, 2024 11:23 AM To: Jay Ashworth ; nanog@nanog.org Subject: RE: Meta outage Does meta keep a board somewhere to tell the world it’s down? From: NANOG On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
RE: Meta outage
Thank you From: Jay Ashworth Sent: Tuesday, March 05, 2024 11:25 AM To: Kain, Becki (.) ; nanog@nanog.org Subject: RE: Meta outage Yes: metastatus.com It isn't happy. On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)" mailto:bka...@ford.com>> wrote: Does meta keep a board somewhere to tell the world it’s down? From: NANOG mailto:nanog-bounces+bkain1=ford@nanog.org>> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Meta outage
This time it looks like AAA issue on Facebook side. All peerings are up and cdns are reachable Dominik Dobrowolski W dniu wt., 5.03.2024 o 17:14 Stephane Bortzmeyer napisał(a): > On Tue, Mar 05, 2024 at 11:06:11AM -0500, > Jay Ashworth wrote > a message of 124 lines which said: > > > This doesn't sound like it's a network layer problem but I'm curious. > > We can see the start page, authentification fails with an error > message. It does not look like a network issue. > >
RE: Meta outage
Yes: metastatus.com It isn't happy. On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)" wrote: >Does meta keep a board somewhere to tell the world it’s down? > >From: NANOG On Behalf Of Jay Ashworth >Sent: Tuesday, March 05, 2024 11:06 AM >To: nanog@nanog.org >Subject: Meta outage > >WARNING: This message originated outside of Ford Motor Company. Use caution >when opening attachments, clicking links, or responding. > >It's making the general press this hour so of course you already know about it >but my question is this: who peers with meta and have you seen BGP sessions >drop or the like? Do you operate meta CDN nodes in your network? Are they >screaming for help? > >This doesn't sound like it's a network layer problem but I'm curious. > >Cheers, >-- jra >-- >Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
RE: Meta outage
Does meta keep a board somewhere to tell the world it’s down? From: NANOG On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Meta outage
My first thought, I'll admit, is that they turned a LLM loose on their configuration controls, and it "naturally" put itself in route 666. I sure hope from whatever subbasement clueful admins are still in over there, they can put it back together. Otherwise, productivity might improve as users flee to Office 365 to get some work done. I keep worrying about KEYTRAP -- https://www.youtube.com/watch?v=N0Tmvv5jJKs Epik Mellon: Making the Internet Better Dave Täht CSO, LibreQos
Re: Meta outage
They reported a major failure (fat finger source) on one of their DB clusters. -J On Tue, Mar 5, 2024 at 10:07 AM Jay Ashworth wrote: > It's making the general press this hour so of course you already know > about it but my question is this: who peers with meta and have you seen BGP > sessions drop or the like? Do you operate meta CDN nodes in your network? > Are they screaming for help? > > This doesn't sound like it's a network layer problem but I'm curious. > > Cheers, > -- jra > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. >
Re: Meta outage
On Tue, Mar 05, 2024 at 11:06:11AM -0500, Jay Ashworth wrote a message of 124 lines which said: > This doesn't sound like it's a network layer problem but I'm curious. We can see the start page, authentification fails with an error message. It does not look like a network issue.
Meta outage
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
