Re: [VoiceOps] Bandwidth East Coast Issues

BW Considers the incident resolved. Permalink to Status Page for the outage: https://status.bandwidth.com/incidents/d09cdj4qljc1 Final Update: Resolved - With continued network stability, we consider this incident resolved. Sep 25, 2021 - 19:45 EDT On Sat, 25 Sep 2021, Peter

Re: [VoiceOps] Bandwidth East Coast Issues

BW Status Page is showing 100% operational now. Our monitoring confirms their supposition. Monitoring - Bandwidth teams have seen overall network stability with expected traffic behavior. We will continue to monitor the network. Sep 25, 19:29 EDT On Sat, 25 Sep 2021,

Re: [VoiceOps] Bandwidth East Coast Issues

Another Update Update - Bandwidth continues to see portal and network improvements. At this time, 911 & messaging traffic is processing as expected. We continue to work to resolve other impacted services. Sep 25, 19:07 EDT The Status Page ( https://status.bandwidth.com/ )

Re: [VoiceOps] Bandwidth East Coast Issues

According to BW TAC it was a DDoS. The outages list posted this: Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and

Re: [VoiceOps] Bandwidth East Coast Issues

Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.

Re: [VoiceOps] Bandwidth East Coast Issues

Looks like things are recovering. Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago. Latest Update Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior. Sep 25, 18:18 EDT

Re: [VoiceOps] Bandwidth East Coast Issues

Hmmm... DoS? -Original Message- From: VoiceOps On Behalf Of Peter Beckman via VoiceOps Sent: Saturday, September 25, 2021 4:49 PM To: VoiceOps Subject: Re: [VoiceOps] Bandwidth East Coast Issues Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these

Re: [VoiceOps] Bandwidth East Coast Issues

Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs 67.231.9.112 67.231.13.113 On Sat, 25 Sep 2021, Peter Beckman wrote: Another Update Update - Bandwidth teams continue to investigate Voice and Messaging

Re: [VoiceOps] Bandwidth East Coast Issues

Another Update Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures. Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness. All teams are working

Re: [VoiceOps] Bandwidth East Coast Issues

Agreed. But remote now ... Calls are completing now on bandwidth. But we will remove them from the deck. On Sat, Sep 25, 2021, 10:34 AM Peter Beckman wrote: > Just remove them from your deck. > > On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote: > > > Can someone ask bandwidth to stop

Re: [VoiceOps] Bandwidth East Coast Issues

Just remove them from your deck. On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote: Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.? On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: As of 3:31pm EDT

Re: [VoiceOps] Bandwidth East Coast Issues

After a long silence, no updates other than "we're working on it." Update - Bandwidth teams continue to investigate Voice and Messaging Failures. At this time, 911 calls are not impacted. Customers may also experience portal and portal API time-outs, slowness, or

Re: [VoiceOps] Bandwidth East Coast Issues

Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.? On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > As of 3:31pm EDT Bandwidth has acknowledged the issue: > > Investigating - Bandwidth is investigating an

Re: [VoiceOps] Bandwidth East Coast Issues

Confirmed. All our calls, bandwidth is taking the call / sip invite but not passing the call. On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > As of 3:31pm EDT Bandwidth has acknowledged the issue: > > Investigating - Bandwidth is investigating an

Re: [VoiceOps] Bandwidth East Coast Issues

third. and our monitoring picked up packet loss to their IPs. Hopefully not a DDOS or if it is, they are better prepared to mitigate it than voip.ms. [image: Star Telecom - Cloud Communications and Customer Experience Solutions] *Ivan Kovacevic* *Co-Founder and

Re: [VoiceOps] Bandwidth East Coast Issues

We are seeing similar issues starting about the same time, Jeff On Sat, Sep 25, 2021, 3:33 PM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS > starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has

Re: [VoiceOps] Bandwidth East Coast Issues

As of 3:31pm EDT Bandwidth has acknowledged the issue: Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged. Sep 25, 15:31 EDT I will update here

[VoiceOps] Bandwidth East Coast Issues

We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated. I'm seeing outbound calls time out, inbound calls to BW DIDs receive a