Re: [outages] East NA DNS Kabooms (was Re: Alexa Outage?)

[Bill Woodcock via Outages]

> On Jun 9, 2021, at 9:44 PM, Ross Halliday via Outages  
> wrote:
> 
> https://isitdns.com/
> 
> We’ve had customers reporting issues with Salesforce for the last few hours 
> and one of the issues we discovered was intermittent SERVFAILs. It looks like 
> they’ve fixed/worked around the issue ( see IDs 709 and 7273 at 
> https://status.salesforce.com/ )
> 
> Looking at the graphs at https://downdetector.com/ Alexa, AWS, Zoom, and 
> Salesforce graphs look almost identical, so I’m guessing this is not a 
> coincidence.

It’s what’s left of UltraDNS.  Another DDoS like the one at the beginning of 
May, but quite a bit smaller.

-Bill



signature.asc
Description: Message signed with OpenPGP
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] East NA DNS Kabooms (was Re: Alexa Outage?)

[Ross Halliday via Outages]

https://isitdns.com/

We’ve had customers reporting issues with Salesforce for the last few hours and 
one of the issues we discovered was intermittent SERVFAILs. It looks like 
they’ve fixed/worked around the issue ( see IDs 709 and 7273 at 
https://status.salesforce.com/ )

Looking at the graphs at https://downdetector.com/ Alexa, AWS, Zoom, and 
Salesforce graphs look almost identical, so I’m guessing this is not a 
coincidence.



- Ross



From: Outages  On Behalf Of Aaron C. de Bruyn via 
Outages
Sent: June 9, 2021 2:39 PM
To: Ryan Wilkins 
Cc: outages@outages.org
Subject: [EXTERNAL] Re: [outages] Alexa Outage?

I definitely think there's a problem.  It came back up for a few minutes, and 
now it's down again and occasionally responding with random nonsense.

-A

On Wed, Jun 9, 2021 at 11:37 AM Ryan Wilkins  wrote:
Reading your previous message, I said “Alexa, Hello” to my Echo and the thing 
responded with some long answer and then asked me if that answered my question. 
 No idea what it thought I said.  Maybe it’s still broke.

Ryan Wilkins


On Jun 9, 2021, at 2:29 PM, Aaron C. de Bruyn via Outages 
 wrote:

Stuff just recovered on my end.

-A

On Wed, Jun 9, 2021 at 11:15 AM Aaron C. de Bruyn  
wrote:
Calls from a handful of users that Alexa is saying "I'm having trouble 
connecting" or "I'm having trouble understanding you at the moment".

I plugged in an Alexa device in my office, and it's also having trouble.

https://downdetector.com/status/amazon-alexa/

-A
___
Outages mailing list
mailto:Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Outages-discussion Digest, Vol 136, Issue 1

[outages-discussion-request]

Send Outages-discussion mailing list submissions to
outages-discussion@outages.org

To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org

You can reach the person managing the list at
outages-discussion-ow...@outages.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."


Today's Topics:

   1. FW: [outages-discussion] Fastly CDN down? (Mike Castoro)
   2. Re: FW: [outages-discussion] Fastly CDN down? (Jeremy Chadwick)
   3. Re: FW: [outages-discussion] Fastly CDN down? (Mike Castoro)
   4. Re: [outages] Alexa Outage? (Andy Ringsmuth)


--

Message: 1
Date: Tue, 8 Jun 2021 11:36:24 +
From: Mike Castoro 
To: "outages-discussion@outages.org" 
Subject: [Outages-discussion] FW: [outages-discussion] Fastly CDN
down?
Message-ID:



Content-Type: text/plain; charset="us-ascii"

Added to Outages-Discussion.  Who knew that half the internet used the same CDN 
edge provider?   Why would AWS and GCP use them?  Our Shopify sites were down 
this morning and Shopify support scrambled to even know it was happening (plus 
their status page was down as well).

Mike Castoro | VP of IT
Adrianna Papell Group | 646-723-3885
Chat with me in Teams

-Original Message-
From: Outages  On Behalf Of Stephane Bortzmeyer 
via Outages
Sent: Tuesday, June 8, 2021 7:10 AM
To: DaZZa 
Cc: outa...@outages.org
Subject: Re: [outages] Fastly CDN down?

External Email!


On Tue, Jun 08, 2021 at 08:54:48PM +1000,  DaZZa  wrote  
a message of 45 lines which said:

> Yes. Apparently a widespread outage. Numerous sites affected, 
> including Reddit.

Seems fixed now.

https://status.fastly.com/
___
Outages mailing list
outa...@outages.org
https://puck.nether.net/mailman/listinfo/outages


--

Message: 2
Date: Tue, 8 Jun 2021 13:06:23 +
From: Jeremy Chadwick 
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] FW: [outages-discussion] Fastly CDN
down?
Message-ID:

<01010179ebbb2d4d-8f599d73-e3e2-4715-8667-f6dd41d5a04f-000...@us-west-2.amazonses.com>

Content-Type: text/plain; charset=us-ascii

https://news.ycombinator.com/item?id=27432408 has some discussion about
why certain parts of Amazon (not AWS!) uses Fastly, and what Amazon did
to mitigate (switched to Cloudfront).

As for "who knew half the internet used this same CDN", I've known this
for a while.  The number of companies doing this continue to grow.  For
some reason online service providers have still not learned to avoid
putting all their eggs in one basket; they will spend exorbitant amounts
of time and money to guarantee 5-9s but then introduce critical SPoFs
*by choice*.  Another recent example of this same mindset was the AWS
Kinesis outage in November, where AWS simply trusted their own tech too
much: https://aws.amazon.com/message/11201/

{off-topic}  There is a pervasive mentality in the online web-based
service industry that, put roughly, amounts to "Big Company X uses
Thingus, so we should use Thingus too".  This applies to not just CDNs
or other service providers, but software as well.  Us old codgers know
this isn't a good idea (or if it's deemed acceptable, that the downsides
are well-understood and other mitigation methods are available at time
of infrastructure design).  "Why do it ourselves when Derpaherpaderp.io
can do it for us?" is a dangerous question, and one that should be
approached with extreme caution.  Proper monitoring is almost always
neglected as well (Ops are told there are more important things to
address, rinse lather repeat for years -- a backlog that never gets
worked on).  {/off-topic}

That said, ~1-1.5 hours from start to fix is pretty good:
https://status.fastly.com/#past-incidents .  Respects to those in Fastly
Operations who likely got woken up + had to deal with it.

-- 
| Jeremy Chadwick j...@koitsu.org |
| UNIX Systems Administrator  PGP 0x2A389531 |
| Making life hard for others since 1977.|

On Tue, Jun 08, 2021 at 11:36:24AM +, Mike Castoro wrote:
> Added to Outages-Discussion.  Who knew that half the internet used the same 
> CDN edge provider?   Why would AWS and GCP use them?  Our Shopify sites were 
> down this morning and Shopify support scrambled to even know it was happening 
> (plus their status page was down as well).
> 
> Mike Castoro | VP of IT
> Adrianna Papell Group | 646-723-3885
> Chat with me in Teams
> 
> -Original Message-
> From: Outages  On Behalf Of Stephane Bortzmeyer 
> via Outages
> Sent: Tuesday, June 8, 2021 7:10 AM
> To: DaZZa 
> Cc: 

Re: [outages] Alexa Outage?

[Aaron C. de Bruyn via Outages]

I definitely think there's a problem.  It came back up for a few minutes,
and now it's down again and occasionally responding with random nonsense.

-A

On Wed, Jun 9, 2021 at 11:37 AM Ryan Wilkins  wrote:

> Reading your previous message, I said “Alexa, Hello” to my Echo and the
> thing responded with some long answer and then asked me if that answered my
> question.  No idea what it thought I said.  Maybe it’s still broke.
>
> Ryan Wilkins
>
> On Jun 9, 2021, at 2:29 PM, Aaron C. de Bruyn via Outages <
> outages@outages.org> wrote:
>
> Stuff just recovered on my end.
>
> -A
>
> On Wed, Jun 9, 2021 at 11:15 AM Aaron C. de Bruyn 
> wrote:
>
>> Calls from a handful of users that Alexa is saying "I'm having trouble
>> connecting" or "I'm having trouble understanding you at the moment".
>>
>> I plugged in an Alexa device in my office, and it's also having trouble.
>>
>> https://downdetector.com/status/amazon-alexa/
>>
>> -A
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Alexa Outage?

[Ryan Wilkins via Outages]

Reading your previous message, I said “Alexa, Hello” to my Echo and the thing 
responded with some long answer and then asked me if that answered my question. 
 No idea what it thought I said.  Maybe it’s still broke.

Ryan Wilkins

> On Jun 9, 2021, at 2:29 PM, Aaron C. de Bruyn via Outages 
>  wrote:
> 
> Stuff just recovered on my end.
> 
> -A
> 
> On Wed, Jun 9, 2021 at 11:15 AM Aaron C. de Bruyn  > wrote:
> Calls from a handful of users that Alexa is saying "I'm having trouble 
> connecting" or "I'm having trouble understanding you at the moment".
> 
> I plugged in an Alexa device in my office, and it's also having trouble.
> 
> https://downdetector.com/status/amazon-alexa/ 
> 
> 
> -A
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Alexa Outage?

[Aaron C. de Bruyn via Outages]

Stuff just recovered on my end.

-A

On Wed, Jun 9, 2021 at 11:15 AM Aaron C. de Bruyn 
wrote:

> Calls from a handful of users that Alexa is saying "I'm having trouble
> connecting" or "I'm having trouble understanding you at the moment".
>
> I plugged in an Alexa device in my office, and it's also having trouble.
>
> https://downdetector.com/status/amazon-alexa/
>
> -A
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Alexa Outage?

[Aaron C. de Bruyn via Outages]

Calls from a handful of users that Alexa is saying "I'm having trouble
connecting" or "I'm having trouble understanding you at the moment".

I plugged in an Alexa device in my office, and it's also having trouble.

https://downdetector.com/status/amazon-alexa/

-A
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages