Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
Probably, but better to be safe than sorry ;) On Sun, Jul 21, 2024 at 10:52 PM J. Hellenthal wrote: > > On Jul 21, 2024, at 03:50, Jrandombob wrote: > > > At this juncture I'd like to humbly suggest that between this incident and > the similar 2010 cockup he presided over as CTO of McAfee (botched defs > that caused it to delete Windows XP svchost.exe), that Mr George Kurtz > should strongly consider divesting himself of CrowdStrike and refraining > from any further involvement with EDR (or similar) vendors in future, for > the good of the industry, and the world at large ;) > > > That's way too much credit to give to a CTO! > > On Fri, Jul 19, 2024 at 6:36 PM Tony Wicks wrote: > >> >> https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue >> >> >> >> *From:* AusNOG *On Behalf Of *DaZZa >> *Sent:* Friday, July 19, 2024 8:22 PM >> *To:* Luke Thompson >> *Cc:* AusNOG >> *Subject:* Re: [AusNOG] Global Outage - CrowdStrike? What's the story? >> >> >> >> Crowdstrike pushed a bad update which sends windows machine into a BSOD >> loop >> >> So far, the only way of fixing it is to manually boot *every* effected >> machine into safe mode and delete the offending file. >> >> It's broken banks, airports, TV playout systems, ServiceNSW was >> completely unable to service anything - and it's world wide *anywhere* >> Crowdstrike is used on windows - servers or workstations. >> >> It's going to be an ugly weekend for any org who uses it! >> >> D >> >> >> >> On Fri, 19 July 2024, 6:10 pm Luke Thompson, wrote: >> >> G'day, >> >> Just curious about what's amiss, a few things lined up this arvo: >> >> 1. Banks advising that there is an outage impacting payments etc. >> >> 2. Telstra MVNO Support advising that they have no internet right now. >> >> 3. There is a spike of problems reported on Down Detector, now calming: >> >> https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, >> NBN...) >> >> Wondering if anyone can share any insights into what's happening this >> afternoon BTS or otherwise? >> >> >> https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 >> >> Cheers, >> Luke >> >> > ___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
On Jul 21, 2024, at 03:50, Jrandombob wrote:At this juncture I'd like to humbly suggest that between this incident and the similar 2010 cockup he presided over as CTO of McAfee (botched defs that caused it to delete Windows XP svchost.exe), that Mr George Kurtz should strongly consider divesting himself of CrowdStrike and refraining from any further involvement with EDR (or similar) vendors in future, for the good of the industry, and the world at large ;)That's way too much credit to give to a CTO!On Fri, Jul 19, 2024 at 6:36 PM Tony Wicks <t...@wicks.co.nz> wrote:https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue From: AusNOG <ausnog-boun...@lists.ausnog.net> On Behalf Of DaZZaSent: Friday, July 19, 2024 8:22 PMTo: Luke Thompson Cc: AusNOG <ausnog@lists.ausnog.net>Subject: Re: [AusNOG] Global Outage - CrowdStrike? What's the story? Crowdstrike pushed a bad update which sends windows machine into a BSOD loop So far, the only way of fixing it is to manually boot *every* effected machine into safe mode and delete the offending file.It's broken banks, airports, TV playout systems, ServiceNSW was completely unable to service anything - and it's world wide *anywhere* Crowdstrike is used on windows - servers or workstations.It's going to be an ugly weekend for any org who uses it!D On Fri, 19 July 2024, 6:10 pm Luke Thompson, <luke.t@tnc.works> wrote:G'day,Just curious about what's amiss, a few things lined up this arvo:1. Banks advising that there is an outage impacting payments etc.2. Telstra MVNO Support advising that they have no internet right now.3. There is a spike of problems reported on Down Detector, now calming:https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, NBN...)Wondering if anyone can share any insights into what's happening this afternoon BTS or otherwise?https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960Cheers,Luke___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
At this juncture I'd like to humbly suggest that between this incident and the similar 2010 cockup he presided over as CTO of McAfee (botched defs that caused it to delete Windows XP svchost.exe), that Mr George Kurtz should strongly consider divesting himself of CrowdStrike and refraining from any further involvement with EDR (or similar) vendors in future, for the good of the industry, and the world at large ;) On Fri, Jul 19, 2024 at 6:36 PM Tony Wicks wrote: > > https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue > > > > *From:* AusNOG *On Behalf Of *DaZZa > *Sent:* Friday, July 19, 2024 8:22 PM > *To:* Luke Thompson > *Cc:* AusNOG > *Subject:* Re: [AusNOG] Global Outage - CrowdStrike? What's the story? > > > > Crowdstrike pushed a bad update which sends windows machine into a BSOD > loop > > So far, the only way of fixing it is to manually boot *every* effected > machine into safe mode and delete the offending file. > > It's broken banks, airports, TV playout systems, ServiceNSW was completely > unable to service anything - and it's world wide *anywhere* Crowdstrike is > used on windows - servers or workstations. > > It's going to be an ugly weekend for any org who uses it! > > D > > > > On Fri, 19 July 2024, 6:10 pm Luke Thompson, wrote: > > G'day, > > Just curious about what's amiss, a few things lined up this arvo: > > 1. Banks advising that there is an outage impacting payments etc. > > 2. Telstra MVNO Support advising that they have no internet right now. > > 3. There is a spike of problems reported on Down Detector, now calming: > > https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, > NBN...) > > Wondering if anyone can share any insights into what's happening this > afternoon BTS or otherwise? > > > https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 > > Cheers, > Luke > > ___ > AusNOG mailing list > AusNOG@lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > ___ > AusNOG mailing list > AusNOG@lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > ___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue From: AusNOG On Behalf Of DaZZa Sent: Friday, July 19, 2024 8:22 PM To: Luke Thompson Cc: AusNOG Subject: Re: [AusNOG] Global Outage - CrowdStrike? What's the story? Crowdstrike pushed a bad update which sends windows machine into a BSOD loop So far, the only way of fixing it is to manually boot *every* effected machine into safe mode and delete the offending file. It's broken banks, airports, TV playout systems, ServiceNSW was completely unable to service anything - and it's world wide *anywhere* Crowdstrike is used on windows - servers or workstations. It's going to be an ugly weekend for any org who uses it! D On Fri, 19 July 2024, 6:10 pm Luke Thompson, mailto:luke.t@tnc.works> > wrote: G'day, Just curious about what's amiss, a few things lined up this arvo: 1. Banks advising that there is an outage impacting payments etc. 2. Telstra MVNO Support advising that they have no internet right now. 3. There is a spike of problems reported on Down Detector, now calming: https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, NBN...) Wondering if anyone can share any insights into what's happening this afternoon BTS or otherwise? https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 Cheers, Luke ___ AusNOG mailing list AusNOG@lists.ausnog.net <mailto:AusNOG@lists.ausnog.net> https://lists.ausnog.net/mailman/listinfo/ausnog ___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
Crowdstrike pushed a bad update, which caused Windows systems to BSOD. Given the software updates automatically, it hit lots of systems in a very short period of time. I was talking to someone who had been in a checkout line at a Woolworths, and apparently the self-service terminals were just bluescreening one after the next. Crowdstrike has pulled the update, but not before the damage was done. Resolution apparently involved booting into safe mode, removing the bad files, and then rebooting normally. The issues are very much global, but hit during a period where the US and even much of Europe was asleep, so it was more impactful here than elsewhere. Scott On Fri, Jul 19, 2024 at 6:10 PM Luke Thompson wrote: > G'day, > > Just curious about what's amiss, a few things lined up this arvo: > > 1. Banks advising that there is an outage impacting payments etc. > > 2. Telstra MVNO Support advising that they have no internet right now. > > 3. There is a spike of problems reported on Down Detector, now calming: > > https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, > NBN...) > > Wondering if anyone can share any insights into what's happening this > afternoon BTS or otherwise? > > > https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 > > Cheers, > Luke > > ___ > AusNOG mailing list > AusNOG@lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > ___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
Re: [AusNOG] Global Outage - CrowdStrike? What's the story?
Crowdstrike pushed a bad update which sends windows machine into a BSOD loop So far, the only way of fixing it is to manually boot *every* effected machine into safe mode and delete the offending file. It's broken banks, airports, TV playout systems, ServiceNSW was completely unable to service anything - and it's world wide *anywhere* Crowdstrike is used on windows - servers or workstations. It's going to be an ugly weekend for any org who uses it! D On Fri, 19 July 2024, 6:10 pm Luke Thompson, wrote: > G'day, > > Just curious about what's amiss, a few things lined up this arvo: > > 1. Banks advising that there is an outage impacting payments etc. > > 2. Telstra MVNO Support advising that they have no internet right now. > > 3. There is a spike of problems reported on Down Detector, now calming: > > https://downdetector.com.au/ (Google to AWS to CBA/NAB, Xero, Uber, > NBN...) > > Wondering if anyone can share any insights into what's happening this > afternoon BTS or otherwise? > > > https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 > > Cheers, > Luke > > ___ > AusNOG mailing list > AusNOG@lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > ___ AusNOG mailing list AusNOG@lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog
