Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is
observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound
and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
Monitoring these two IPs
67.231.9.112
67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging
Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs,
slowness, or
unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT)
as they are now responding at an acceptable rate. I just tested outbound
and got a call through to a Verizon Mobile, though the PDD was about 5-6
seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging
Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs,
slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered
faster and we have not seen any missed SMS messages. Worst SMS delay was
9m
37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5
were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice
and Messaging Services. Calls and Messages may experience unexpected
failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS
messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and
SMS
starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
acknowledged
issues are detected but their status page as of yet has not been
updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a
"call cannot be completed" error on Verizon, and our SMS tests every 10
minutes that usually complete in under 2 seconds are taking 15 seconds
to
3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
https://status.bandwidth.com/
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
---------------------------------------------------------------------------
Peter Beckman Internet
Guy
beck...@angryox.com
https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beck...@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
