I was sent this: I believe they are still working on a more in depth root cause

Hello everyone,

We had a service disruption today.  As of a few moments ago, our services are 
actively being restored and, with few exceptions, the outage we had today is 

First and foremost, this email is an apology for this widespread outage.  I 
know that as your business phone system provider, you depend upon the telephone 
for sales and service, and HUD to help you run your company, and that what 
happened today probably destroyed a lot of the trust that we built to date.  
Despite our best efforts, we didn’t deliver the reliability we’ve been known 
for, and I’m sorry.

Now, on to the details, which I hope is the first step toward rebuilding that 

What we know right now, which isn’t much but I expect more details later 
tonight, is that we had a failure of our DNS services.  DNS, or Domain Name 
System, is what translates names (like fonality.com) to the internet’s IP 
addresses, which are numbers that identify each computer, phone, and other 
device everywhere in the world.  Despite having multiple data centers, and 
multiple layers of redundancy within and between our hosting locations, 
something common to all of them - our DNS service - failed.  And as a result, 
just about everything else went down with it, because they all depend upon DNS 
so they can be “found” by the various components of our system.  This includes 
our voice carriers, our control panel, our HUD service, and even our own office 
communications (website, phones, HUD, and email) so it was difficult to even 
respond to you if you tried to reach us.    

Something so critical as DNS certainly has redundancies, but in this case and 
for a reason we’re still investigating, both the primary and redundant DNS 
systems failed.  You can be sure we’re going to do everything in our power to 
ensure that this type of global event can’t ever happen again - we will learn 
from this incident and will be better for it.

In the future, if you’d like to receive proactive updates, please visit our 
public system status site at http://trust.fonality.com or 
http://fonality.statuspage.io and sign up for alerts. That way you can be kept 
up to date proactively.

Please accept my most sincere apologies for the trouble today, and know that 
all of us here at Fonality are going to work hard to ensure this type of 
incident never happens again.

- David


Rich Walz
Sent from my iPhone

> On Sep 17, 2016, at 6:08 PM, chris via Outages <outages@outages.org> wrote:
> Just curious, did anyone get an RFO or any explanation of what happened?
> chris
>> On Fri, Sep 16, 2016 at 1:30 PM, Aryn Nakaoka 808.356.2901 
>> <anaka...@trinet-hi.com> wrote:
>> http://www.isitdownrightnow.com/fonality.com.html
>> http://downdetector.com/status/fonality
>> Aryn H. K. Nakaoka
>> anaka...@trinet-hi.com
>> Direct: 808.356.2901
>> Fax : 808.356.2919
>> Tri-net Solutions
>> 518 Holokahana Lane Suite #200
>> Honolulu, HI 96817
>> http://www.trinet-hi.com
>> https://twitter.com/AlohaTone
>> Aloha Tone PBX  https://www.youtube.com/watch?v=96YWPY9wCeU
>> Aloha Tone (HA) High Availability http://youtu.be/rJsr4k0RBH8
>> CONFIDENTIALITY NOTICE:  The information contained in this email and any 
>> attachments may be privileged, confidential and protected from disclosure.  
>> Any disclosure, distribution or copying of this email or any attachments by 
>> persons or entities other than the intended recipient is prohibited. If you 
>> have received this email in error, please notify the sender immediately by 
>> replying to the message and deleting this email and any attachments from 
>> your system. Thank you for your cooperation.
>>> On Fri, Sep 16, 2016 at 7:08 AM, Adam Moss <am...@flowroute.com> wrote:
>>> Getting tons of reports from customers that their Fonality systems have 
>>> stopped working.
>>> Anyone got any news on this?  Looks like their twitter is blowing up too.
>>> -- 
>>> +1.206.641.8034      
>>> _______________________________________________
>>> VoiceOps mailing list
>>> voice...@voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>> _______________________________________________
>> VoiceOps mailing list
>> voice...@voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
Outages mailing list

Reply via email to