Re: [VoiceOps] Bandwidth East Coast Issues

-ix.com > > > > From: "Ross Tajvar" > To: "Mike Hammett" > Cc: "Paul Timmins" , "VoiceOps" > Sent: Monday, September 27, 2021 1:38:32 AM > Subject: Re: [VoiceOps] Bandwidth East Coast Issues > > On Sun, Sep 26, 2021 at

Re: [VoiceOps] Bandwidth East Coast Issues

://www.midwest-ix.com - Original Message - From: "Ross Tajvar" To: "Mike Hammett" Cc: "Paul Timmins" , "VoiceOps" Sent: Monday, September 27, 2021 1:38:32 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues On Sun, Sep 26, 2021 at 5:24 PM Mi

Re: [VoiceOps] Bandwidth East Coast Issues

On Sun, Sep 26, 2021 at 5:24 PM Mike Hammett wrote: > I have IXes in areas where Inteliquent and Peerless have POPs. They > haven't had any interest. > > I don't understand. > I understand. Serious customers will pay for a PNI. Industry partners (other large telcos with whom they exchange a

Re: [VoiceOps] Bandwidth East Coast Issues

: "Paul Timmins" To: "Mike Hammett" Cc: "Pete Mundy" , "VoiceOps" Sent: Sunday, September 26, 2021 3:53:47 PM Subject: Re: [VoiceOps] Bandwidth East Coast Issues I just looked into all the big wholesalers, and they all have terrible peering policies,

Re: [VoiceOps] Bandwidth East Coast Issues

t; Midwest Internet Exchange > http://www.midwest-ix.com <http://www.midwest-ix.com/> > > > > From: "Pete Mundy via VoiceOps" > To: "VoiceOps" > Sent: Sunday, September 26, 2021 1:47:18 AM > Subject: Re: [VoiceOps] Bandwidth East Coast Issues >

Re: [VoiceOps] Bandwidth East Coast Issues

quot; Sent: Sunday, September 26, 2021 3:02:45 PM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Right, yes, sorry - I should have been more explanative :) I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, a

Re: [VoiceOps] Bandwidth East Coast Issues

Right, yes, sorry - I should have been more explanative :) I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, and perhaps it's not quite over just yet..." > On 27/09/2021, at 3:33 AM, Mike Hammett wrote: > > That

Re: [VoiceOps] Bandwidth East Coast Issues

://www.midwest-ix.com - Original Message - From: "Pete Mundy via VoiceOps" To: "VoiceOps" Sent: Sunday, September 26, 2021 1:47:18 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Not sure if Slashdot does... https://tech.slashdot.org/story/21/09/26/0014216

Re: [VoiceOps] Bandwidth East Coast Issues

Not sure if Slashdot does... https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-sustained-ddos-for-ransom-attack > On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps > wrote: > > BW Considers the incident resolved. ___

Re: [VoiceOps] Bandwidth East Coast Issues

BW Considers the incident resolved. Permalink to Status Page for the outage: https://status.bandwidth.com/incidents/d09cdj4qljc1 Final Update: Resolved - With continued network stability, we consider this incident resolved. Sep 25, 2021 - 19:45 EDT On Sat, 25 Sep 2021, Peter

Re: [VoiceOps] Bandwidth East Coast Issues

BW Status Page is showing 100% operational now. Our monitoring confirms their supposition. Monitoring - Bandwidth teams have seen overall network stability with expected traffic behavior. We will continue to monitor the network. Sep 25, 19:29 EDT On Sat, 25 Sep 2021,

Re: [VoiceOps] Bandwidth East Coast Issues

Another Update Update - Bandwidth continues to see portal and network improvements. At this time, 911 & messaging traffic is processing as expected. We continue to work to resolve other impacted services. Sep 25, 19:07 EDT The Status Page ( https://status.bandwidth.com/ )

Re: [VoiceOps] Bandwidth East Coast Issues

According to BW TAC it was a DDoS. The outages list posted this: Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and

Re: [VoiceOps] Bandwidth East Coast Issues

Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.

Re: [VoiceOps] Bandwidth East Coast Issues

Looks like things are recovering. Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago. Latest Update Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior. Sep 25, 18:18 EDT

Re: [VoiceOps] Bandwidth East Coast Issues

Hmmm... DoS? -Original Message- From: VoiceOps On Behalf Of Peter Beckman via VoiceOps Sent: Saturday, September 25, 2021 4:49 PM To: VoiceOps Subject: Re: [VoiceOps] Bandwidth East Coast Issues Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring

Re: [VoiceOps] Bandwidth East Coast Issues

Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs 67.231.9.112 67.231.13.113 On Sat, 25 Sep 2021, Peter Beckman wrote: Another Update Update - Bandwidth teams continue to investigate Voice and Messaging

Re: [VoiceOps] Bandwidth East Coast Issues

Another Update Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures. Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness. All teams are working

Re: [VoiceOps] Bandwidth East Coast Issues

Agreed. But remote now ... Calls are completing now on bandwidth. But we will remove them from the deck. On Sat, Sep 25, 2021, 10:34 AM Peter Beckman wrote: > Just remove them from your deck. > > On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote: > > > Can someone ask bandwidth to stop

Re: [VoiceOps] Bandwidth East Coast Issues

Just remove them from your deck. On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote: Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.? On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: As of 3:31pm EDT

Re: [VoiceOps] Bandwidth East Coast Issues

After a long silence, no updates other than "we're working on it." Update - Bandwidth teams continue to investigate Voice and Messaging Failures. At this time, 911 calls are not impacted. Customers may also experience portal and portal API time-outs, slowness, or

Re: [VoiceOps] Bandwidth East Coast Issues

Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.? On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > As of 3:31pm EDT Bandwidth has acknowledged the issue: > > Investigating - Bandwidth is investigating an

Re: [VoiceOps] Bandwidth East Coast Issues

Confirmed. All our calls, bandwidth is taking the call / sip invite but not passing the call. On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > As of 3:31pm EDT Bandwidth has acknowledged the issue: > > Investigating - Bandwidth is investigating an

Re: [VoiceOps] Bandwidth East Coast Issues

third. and our monitoring picked up packet loss to their IPs. Hopefully not a DDOS or if it is, they are better prepared to mitigate it than voip.ms. [image: Star Telecom - Cloud Communications and Customer Experience Solutions] *Ivan Kovacevic* *Co-Founder and

Re: [VoiceOps] Bandwidth East Coast Issues

We are seeing similar issues starting about the same time, Jeff On Sat, Sep 25, 2021, 3:33 PM Peter Beckman via VoiceOps < voiceops@voiceops.org> wrote: > We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS > starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has

Re: [VoiceOps] Bandwidth East Coast Issues

As of 3:31pm EDT Bandwidth has acknowledged the issue: Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged. Sep 25, 15:31 EDT I will update here

[VoiceOps] Bandwidth East Coast Issues

We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated. I'm seeing outbound calls time out, inbound calls to BW DIDs receive a