I was sent this: I believe they are still working on a more in depth root cause
We had a service disruption today. As of a few moments ago, our services are
actively being restored and, with few exceptions, the outage we had today is
First and foremost, this email is an apology for this widespread outage. I
know that as your business phone system provider, you depend upon the telephone
for sales and service, and HUD to help you run your company, and that what
happened today probably destroyed a lot of the trust that we built to date.
Despite our best efforts, we didn’t deliver the reliability we’ve been known
for, and I’m sorry.
Now, on to the details, which I hope is the first step toward rebuilding that
What we know right now, which isn’t much but I expect more details later
tonight, is that we had a failure of our DNS services. DNS, or Domain Name
System, is what translates names (like fonality.com) to the internet’s IP
addresses, which are numbers that identify each computer, phone, and other
device everywhere in the world. Despite having multiple data centers, and
multiple layers of redundancy within and between our hosting locations,
something common to all of them - our DNS service - failed. And as a result,
just about everything else went down with it, because they all depend upon DNS
so they can be “found” by the various components of our system. This includes
our voice carriers, our control panel, our HUD service, and even our own office
communications (website, phones, HUD, and email) so it was difficult to even
respond to you if you tried to reach us.
Something so critical as DNS certainly has redundancies, but in this case and
for a reason we’re still investigating, both the primary and redundant DNS
systems failed. You can be sure we’re going to do everything in our power to
ensure that this type of global event can’t ever happen again - we will learn
from this incident and will be better for it.
In the future, if you’d like to receive proactive updates, please visit our
public system status site at http://trust.fonality.com or
http://fonality.statuspage.io and sign up for alerts. That way you can be kept
up to date proactively.
Please accept my most sincere apologies for the trouble today, and know that
all of us here at Fonality are going to work hard to ensure this type of
incident never happens again.
Sent from my iPhone
> On Sep 17, 2016, at 6:08 PM, chris via Outages <firstname.lastname@example.org> wrote:
> Just curious, did anyone get an RFO or any explanation of what happened?
>> On Fri, Sep 16, 2016 at 1:30 PM, Aryn Nakaoka 808.356.2901
>> <anaka...@trinet-hi.com> wrote:
>> Aryn H. K. Nakaoka
>> Direct: 808.356.2901
>> Fax : 808.356.2919
>> Tri-net Solutions
>> 518 Holokahana Lane Suite #200
>> Honolulu, HI 96817
>> Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU
>> Aloha Tone (HA) High Availability http://youtu.be/rJsr4k0RBH8
>> CONFIDENTIALITY NOTICE: The information contained in this email and any
>> attachments may be privileged, confidential and protected from disclosure.
>> Any disclosure, distribution or copying of this email or any attachments by
>> persons or entities other than the intended recipient is prohibited. If you
>> have received this email in error, please notify the sender immediately by
>> replying to the message and deleting this email and any attachments from
>> your system. Thank you for your cooperation.
>>> On Fri, Sep 16, 2016 at 7:08 AM, Adam Moss <am...@flowroute.com> wrote:
>>> Getting tons of reports from customers that their Fonality systems have
>>> stopped working.
>>> Anyone got any news on this? Looks like their twitter is blowing up too.
>>> ADAM MOSS
>>> SUPPORT ENGINEER
>>> VoiceOps mailing list
>> VoiceOps mailing list
> Outages mailing list
Outages mailing list