Re: [outages] Multi-state 911 outages reported Weds night

I did look around a little before posting it and I saw corroborating information from a couple different sources, but nobody had delved deeply enough yet to be looking for commonalities. It looks as if the outages were only a couple of hours - some of them at least. On April 18, 2024 11:26:33

Re: [outages] After-Action Report: Facebook/Meta outage

Yeah, but I shouldn't have gotten bitten. I'm supposed to be holding myself to a higher standard, as a moderator of this particular group. On March 5, 2024 2:58:00 PM EST, Ross Tajvar wrote: >Indeed - this happens every time facebook has a major outage, because they >stopped putting out posts

[outages] FACEBOOK/META recovering

No further messages are expected to be necessary on the main list. I know no one's actually listening, but please govern yourselves accordingly. :-) -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list

Re: [outages] Facebook/ Insta down?

I am hearing some uninformed speculations somebody fat fingered their major DB cluster, but really guys, take this over to the discussion list please. Press hard, you are making 30,000 copies. On March 5, 2024 11:19:32 AM EST, Tim Reichhart via Outages wrote: >It would allow me to login on

[outages] REMINDER: MAIN LIST NOT FOR CHAT

The only other messages we should see on the main list are "it's back now". Please take the discussions over to -discuss. Cheers, -- jr '' a -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list

Re: [outages] Facebook/ Insta down?

No, but hopefully you'll be the last. :-) Third of a million reports on down detector. They apparently screwed the pooch on authentication, somewhere early in the 10:00 EST hour. No further comments are necessary on the main list except for "it's back now", and make sure you're the first

Re: [outages] Equinix CH1 Overheating

d all available doors and have engaged local rental company to source >> additional fans and portable spot coolers. >> > >On Sun, Jan 14, 2024 at 10:19 PM Jay Ashworth via Outages < >outages@outages.org> wrote: > >> Open the doors? >> >> >>

Re: [outages] Equinix CH1 Overheating

Open the doors? On January 14, 2024 9:52:24 PM EST, Mike Hammett via Outages wrote: > >IBX: CH1 >IBX Address: 350 E Cermak Rd, Chicago, 60616 >Number: INC0025932 >Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time > > >To view the incident and its updates, please use Service

[outages] Optus/Akamai

Off of Facebook of all places, I hear this report that Optus had a major outage in Australia last night, which is believed to have been a BGP screw up involving Akamai: "Nope: it was Akamai increasing their advertised prefix count from their cdn nodes inside the Optus network, and Optus’s

Re: [outages] Twitter API problems Fri/13

As a list administrator, I have already replied to Mr Vanino, and I consider the topic closed. You should too. Playing list police also violates mailing list rules. Cheers, -- jra On January 15, 2023 7:58:54 PM EST, TJ Martineau via Outages wrote: >Can anyone please move this to

Re: [outages] Twitter API problems Fri/13

Alex whether the list rules are written down or not, and whether you're right or not, that violates them. Please don't do it again. Cheers, -- jr '' a On January 15, 2023 7:11:06 PM EST, Alex Vanino via Outages wrote: >I'll save him the trouble. It's because he's a peace of shit. > >

Re: [outages] [ADMIN] Re: [Telnyx #1243282] Twitter API problems Fri/13

You were seeing that traffic on the list? Because I have not been seeing their recent new ticket emails actually going to the list, just coming to me when I post something, a venial sin compared to the mortal sin of the other approach. But yeah, given that I didn't hear anything from them all

[outages] AFTER ACTION; Rogers Canadawide outage

Maintenance outage; network flooding. https://globalnews.ca/news/8978590/rogers-communications-network-outage-explanation-ceo/ Comments to -discuss, please. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___

[outages] iPaws CAP feeds flapping

I'm seeing reports from my broadcast radio interests that the iPAWS CAP feeds from FEMA which supply emergency alert info are flapping this afternoon, all over the world. Well okay, conus plus territories. :-) Probably two dozen stations reporting so far, many of them have seen more than one

Re: [outages] DHCP being dropped by Lumen?

An argument that has been going on on nanog for about 30 years now, yes. ;-) On June 8, 2022 11:30:44 AM EDT, Simon Lockhart wrote: >On Wed Jun 08, 2022 at 11:25:59am -0400, Jay Ashworth via Outages wrote: >> Is there any just cause for a wide area carrier like Lumen to pass a network

Re: [outages] DHCP being dropped by Lumen?

Is there any just cause for a wide area carrier like Lumen to pass a network level protocol like DHCP? Cause I can't think of one... On June 8, 2022 11:20:39 AM EDT, Kate Gerry via Outages wrote: >Yes. I noticed this same issue last month. I spoke with Lumen and it was >confirmed that they

[outages] AWS-EAST-1

Yes, everybody knows now that it is an 'underlying outage' type of problem and we should flip our reporting to 'it has been announced to be back' and otherwise we should remember to press hard: you are making 15,000 copies. ;-) -- Sent from my Android device with K-9 Mail. Please excuse my

Re: [outages] Akamai DNS outage?

I gather there were, but my 2 p.m. notification was delayed because it got stuck in the craw of my cell phone. On July 22, 2021 1:24:17 PM EDT, Seth Mattinen via Outages wrote: > > >On 7/22/21 9:45 AM, Andy Ringsmuth via Outages wrote: >> Surprised I haven’t see any mention of it here yet. > >

[outages] THU: Akamai outage

Apparently a big enough one today that first place I heard about it was my local newspaper breaking news email. Sadly, my password is not working so I can't actually post the link here. But if you're having a problem that sounds like "Akamai's broke", it probably is. Cheers, -- jra -- Sent

Re: [outages] Google DNS

You aren't out of place, you just beat me to it. :-) I think that was well put. Cheers, -- jr '' a On May 13, 2021 12:03:02 PM EDT, Charles D'Aoust via Outages wrote: >I feel like I need to say something and I may be out of place, but >please >don't use this list as part of your initial

Re: [outages] Google DNS

And, more importantly, 'Google DNS' -- specifically Google customer resolver DNS, the stuff they provide at 8.8.8.8 -- is generally IP anycast, so depending on your geographic and network location, you are hitting different servers when you go to that IP. On May 13, 2021 10:56:34 AM EDT,

[outages] After Action: Azure

Key rotation failure. https://www.zdnet.com/article/microsofts-latest-cloud-authentication-outage-what-went-wrong/ Comments to -discuss. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list

Re: [outages] Fios Long Island Issues?

This outage actually made the general press, which is where I just got it a couple minutes ago - I'm on site and haven't been following the list. On January 26, 2021 11:31:54 AM EST, David Rubin via Outages wrote: >I'm suddenly seeibng a bunch of flapping with FiOS - seems to be on >Long

Re: [outages] Explosion in Nashville

It is. So unless that message that you reposted came to you with a flag line suggesting it should not be disseminated, you're welcome to post it here. Cheers, -- jr '' a On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages wrote: >Sorry I don't follow. Isn't the point of this list to

Re: [outages] random gmail bounce issues ?

Looks like they're on it: https://www.zdnet.com/article/gmail-down-again-some-users-are-seeing-error-messages/ On December 15, 2020 4:52:59 PM EST, Chris Cherry via Outages wrote: >YouTube and YouTube TV also seem to be affected. > >Sent from

Re: [outages] AT SF Fiber - Major Packet Loss

Thank you, Ross. Yes, Dan, let's be nice. If we can't be nice, let's at least be professional. Cheers, -- jr '' a On December 9, 2020 11:00:28 PM EST, Ross Tajvar via Outages wrote: >Let's stay on-topic, please. > >On Wed, Dec 9, 2020, 10:48 PM Dan Wood via Outages > >wrote: > >> Vagrants

[outages] After Action: Thanksgiving AWS US-East-1 outage: sharding

Here is an article - with a link to the formal AWS after Action Report - about the outage last week which took down a bunch of services in the process. They were trying to expand capacity in their Kinesis cluster, and sharding got them.

Re: [outages] Cloudflare (at least Linode) DNS issues?

Well, my mail server is at linode ATL and it seems to be working okay tonight. On October 15, 2020 9:20:16 PM EDT, Chris Adams via Outages wrote: >I have several domains on ns1-5.linode.com, which appear to actually be >on Cloudflare, and they are returning SERVFAIL for most requests. I've

Re: [outages] O365: here we go again

gt;Troy, OH 45373 > > >On Thu, Oct 1, 2020 at 9:30 AM Jay Ashworth via Outages > >wrote: > >> >> >https://www.zdnet.com/article/outlook-is-down-microsoft-web-outage-hits-users-worldwide/ >> -- >> Sent from my Android device with K-9 Mail. Please excuse my &

[outages] O365: here we go again

https://www.zdnet.com/article/outlook-is-down-microsoft-web-outage-hits-users-worldwide/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

[outages] MN/CL 911 outage

In what I've heard incorrectly dubbed a nationwide outage, CenturyLink in and around Minnesota apparently had a wide area 911 outage of some type. If anybody has any specific details, it would be great to hear them, because this attached news story is pretty mediocre, as you'd expect. Cheers,

[outages] Office 365 outlook makes the news

Oops: https://www.zdnet.com/article/office-365-outage-ongoing-after-roll-back-fails/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] mysprint.sprint.com down/dead

That's a customer-facing website then? It's too soon for that to be merger-related Fallout I guess. On September 13, 2020 11:10:31 PM EDT, Andy Ringsmuth via Outages wrote: >mysprint.sprint.com appears down, which I believe is used for checking >Sprint bills and accounts on the consumer side.

[outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org

[outages] FRIDAY OUTAGE

The consensus seems to be that cloudflare is having Regional outages of some combination of their web Protection Service and their DNS/D'oH Services, if you're silly enough to have configured those anywhere. Everybody please remember to read before you write, you are making 15,000 copies. We

Re: [outages] Broad Frontier DNS outage

I haven't seen that they were blaming it on a third partying, and I wouldn't expect them to be farming out primary customer resolver service to a third party in a case, but the spike in this appears to have been around noon, and reports have been trailing off since then. This one's probably

Re: [outages] calls to T-Mobile are fialing?

Just a reminder, this is apparently not solely cellular, even messages being sent to the list from hard connections are being delayed, and that is opening the feedback loop some. On June 15, 2020 5:17:55 PM EDT, Andy Ringsmuth via Outages wrote: >outages-discuss...@outages.org PLEASE?!?!? >

[outages] Mon/15 - cell outages

Reports from multiple sources are that all four major national US cellular carriers are having wide area problems today in multiple markets. We'll treat this as the overall report for the moment until things start clearing up. :-) Cheers, -- jra -- Sent from my Android device with K-9 Mail.

Re: [outages] pool.ntp.org

Let's move this to -discuss, please, folks? Cheers, -- jr '' a On May 16, 2020 7:36:07 PM EDT, "J. Hellenthal via Outages" wrote: >Seriously ? Not an outage or disruption considered the density that >time permits? > >-- > J. Hellenthal > >The fact that there's a highway to Hell but only a

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

unds like a fiber cut is to blame. > >https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/ > > > >On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages < >outages@outages.org> wrote: > >> I was hoping somebody was going to climb up to 30

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here. On April

Re: [outages] [EXTERNAL] Re: Discord: Sever outages and Increased API errors

;Put me down for a pre-order. > >-Brad > >[A close up of a bird Description automatically generated] > > > >From: Outages on behalf of Jay Ashworth >via Outages >Reply-To: Jay Ashworth >Date: Friday, March 20, 2020 at 9:07 PM >To: Gary Josack , Gary Josack v

Re: [outages] Discord: Sever outages and Increased API errors

I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine? (For the moment, we will pass on the topic of the irony of

[outages] [admin] Tennessee Tornados

As noted in this and a couple of other replies, it is a safe bet that for the next 72 hours minimum, any outages that you see that terminate in or transit Tennessee are probably the outcome of the swath of tornadoes which hit the state on Monday night. It's probably not necessary to report

Re: [outages] CenturyLink MPLS outage in Arkansas

Or, it could be power outages at intermediate sites which have been on battery until now and the batteries failed. On October 21, 2019 3:28:31 PM EDT, Ross Tajvar via Outages wrote: >Ah, perhaps not related then. > >On Mon, Oct 21, 2019, 2:27 PM Erik Lentz wrote: > >> Everything has been fine

[outages] REMINDER: DORIAN

Hurricane Dorian is headed for the Southeast US coast, and is expected to strike somewhere between Palm Beach and Charleston sometime over the weekend, probably as a category 1 or 2 hurricane. This is likely to generate lots of outages. We've been really good lately about not piling on to the

Re: [outages] [External] Re: Google Login Outage

Brett is right, you are wrong. If you're going to be snotty, you should at least actually be a moderator. One or two yes or no's is plenty, by policy. Further traffic should be taken to -discussion. Cheers, -- jr '' a On August 19, 2019 2:51:02 PM EDT, Neil Hanlon via Outages wrote:

Re: [outages] Comcast Issues Central & East

This appears to have been a regional outage that is in the process of clearing up. I believe we've moved enough traffic on it for the day. :-) Cheers, -- jra On July 31, 2019 12:55:17 PM EDT, Randy Nieland via Outages wrote: >The issues at both of our remote locations have just resolved

Re: [outages] Twitter down

"Something is technically wrong." And we all know that technically wrong is the best kind of wrong. Here is the beginning of the general press coverage about this. https://www.cnet.com/news/twitter-is-down/#ftag=COS-05-10aaa0i Follow ups to -discussion please. And everybody's been very good

[outages] [Admin] Testing 1, 2, 3...

We had a report there has been traffic and it did not make the list. This is a test. -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org

Re: [outages] YOUTUBE DOWN GLOBALLY

At least in Florida, yes. And that's all the more anybody needs to hear. :-) On October 16, 2018 10:43:12 PM EDT, Eric Dugas wrote: >It's back on the east coast! > >On Tue, Oct 16, 2018, 22:35 Jay Ashworth via Outages > >wrote: > >> Internet losing its mind. Let's not

[outages] YOUTUBE DOWN GLOBALLY

Internet losing its mind. Let's not lose ours. Time out 2220 EDT. No further commentary is necessary on this topic until it's back. https://www.mirror.co.uk/tech/breaking-youtube-down-worldwide-furious-13429289 -- Sent from my Android device with K-9 Mail. Please excuse my

[outages] PINCH Re: Facebook Outage

As has been discussed many times in the past on topics like this, this is on topic (so as to provide a heads up to people who give end-users technical support) and it is not on topic for 14 posts in a row. One posting, one or two confirm and deny, and we're done. Lots of people send this list

[outages] L3/CL route leak for 90min, Mon/6

http://wrd.cm/2Ao2NdV -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

[outages] Skype down globally

Nothing at all, huh? :-) Microsoft acknowledging global outage; no ETR at present. More Later. - jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org

Re: [outages] Akamai Cert Issues today

- Original Message - > From: "Sean Donelan via Outages" > This is how Akamai has handled non-SSL customers for the last 15 years. > It is the same error message, and the same action. You just noticed > it. > > If you use https for a non-SSL customer on Akamai, you

[outages] Facebook firm-down*

As of about 10 minutes ago, Facebook is giving out a "Sorry, something went wrong" message to at least some subset of users which includes me. The live map at downdetector.com shows large hotspots in the NE and the UK. I get to mine via Sprint LTE, which terminates in Lenexa KS, I think; there's

[outages] CLEARED: Facebook firm-down*

Seems to be back, at least for me… > > > > > > On Sep 24, 2015, at 9:45 AM, William Byrd via Outages > > <outages@outages.org> > > wrote: > > > > We are aware and actively working on the issue. > > > > William Collier-Byrd > > w...@col

Re: [outages] Windstream Total Outage - Nashville

- Original Message - From: John Pinkerton via Outages outages@outages.org Windstream circuit dropped an hour ago. Alarm LED showing on the wall panel. It was an old Nevox line back in the day. To be clear, Windstream is totally out *for you*, not for Nashville? Cheers, -- jra -- Jay

Re: [outages] EDGE: Anyone seeing 100% CPU on Fortigate edge routers?

- Original Message - From: Jeremy Slater jeremy.sla...@gmail.com We have a large amount of 60D's deployed and we're not seeing any similar behavior. Thanks, Jeremy; it's not the answer I hoped for, but it's helpful. :-) Cheers, -- jra -- Jay R. Ashworth Baylink

[outages] EDGE: Anyone seeing 100% CPU on Fortigate edge routers?

Got a client whose FGT60D -- two of them; they have a hot spare -- is jamming to 100% continuous CPU full time, enough to impinge capacity sharply on their 90mb/s Road Runner line to Five9. Since their primary and hotspare boxes behave identically, end neither ever did before, I infer a

Re: [outages] EDGE: Anyone seeing 100% CPU on Fortigate edge routers?

- Original Message - From: Roland Dobbins via Outages outages@outages.org On 15 Jan 2015, at 23:44, Jay Ashworth via Outages wrote: Since their primary and hotspare boxes behave identically, end neither ever did before, I infer a pattern-file pooch-screwing at Fortigate

Re: [outages] EDGE: Anyone seeing 100% CPU on Fortigate edge routers?

- Original Message - From: Andy Brezinsky via Outages outages@outages.org If you see the issue again, get on the unit and run 'diag sys top' to find the process that is burning CPU. We have in the past seen ssl imd (instant messaging intercept daemon), cache etc cause issues. That

Re: [outages] Level 3 DNS 4.2.2.1/4.2.2.2 Atlanta Issues

Oh. This wasn't my client's uplink. I saw this yesterday after 1600EST, but didn't have time then to run it down. And that was over a 6 hop path from VDSL in Clearwater FL. This minute, traces to 4.2.2.1 aren't dropping any packets. 4.2.2.2 is, about 35%. Those appear to be the ATL

[outages] Craigslist DNS Hijacking last night

If you operate customer resolver servers, they may be poisoned, still; read the message below. - Forwarded Message - From: Ted Cooper ml-nanog0903...@elcsplace.com To: na...@nanog.org Sent: Sunday, November 23, 2014 11:08:51 PM Subject: Re: Craigslist hacked? On 24/11/14 13:41,

Re: [outages] unsubscribe

We're on it; thanks. - Original Message - From: Spencer Gaw via Outages outages@outages.org To: Ronnie Montgomery ronnie.montgom...@ascentcorp.com, outages@outages.org Sent: Monday, November 17, 2014 2:13:39 PM Subject: Re: [outages] unsubscribe Looks like the list web interface is

Re: [outages] Time Warner Cable through Verizon (Wash. DC Metro): Packet Loss

- Original Message - From: Michael Hertrick via Outages outages@outages.org Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in

Re: [outages] Belkin [heartbeat.belkin.com]

- Original Message - From: Michael Loftis via Outages outages@outages.org It's not just the local. 8.8.8.8 is caching the old entry still. Check the authoritative resolvers. Point of order: No resolvers are authoritative. Resolvers (or, more fully, Customer resolver servers) are

[outages] POLL: GoDaddy DNS problems

If you are having problems, still through Thursday night, with DNS resolution off of GoDaddy hosted zone servers, could you reply to *this* thread and provide succinct details of what, when, and how? A friend is on their staff and has the proper ear to whisper in. If you aren't and don't care,

Re: [outages] Facebook down

- Original Message - From: Luke Rockwell via Outages outages@outages.org Subject: [outages] Facebook down tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: Facebook was being observed not to be able to refresh web UI newsfeeds here