Hi All, We had a major issue with our Extreme Networks stack in our core network in Equinix SY1, as the service was flapping, it was not reporting as a hard down, therefore transit was not failing over correctly to other data centres for some customers.
When a hard down in a network happens, redundancy kicks in within the core, todays issues we saw OSPF flapping within the switch stack, meaning that the redundant sites and paths did not consider it down long enough to re route traffic. This subsequently caused a heap more load on the switch stack and caused it to reload, multiple times, after crashing. When it was reloading, people were seeing devices come back as the second data centres were taking over as designed. This is not something that should have happened and we are looking into this, the main goal was to get the stack restored as fast as possible. It has been a very long time since we have had a major outage of this size, and I am disappointed that it has happened and I apologise to anyone who was affected. We will be working with the switch vendors to see what happened and will issue a Post Incident Report to anyone that has a ticket open and linked to this incident. For those not able to get through on the phones, we had too many calls and the system could not cope. We are already implementing a plan for that now. We put a status message up on status.mysau.com.au and the phones to alleviate he call volume and this did help, but we need to be able to handle larger volumes of calls. If anyone is still affected, please call or email us on 1300 788 862 or [email protected]<mailto:[email protected]> Kind Regards, Nick Pratley From: AusNOG <[email protected]> on behalf of Benjamin Ricardo <[email protected]> Date: Monday, 4 September 2017 at 7:40 am To: Burt Mascareigne <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [AusNOG] VOCUS or Server Australia We've got connection failures to a provider at SA this morning too. Sent on the go with Vodafone -------- Original message -------- From: Burt Mascareigne <[email protected]> Date: 4/09/2017 7:37 AM (GMT+10:00) To: ausnog <[email protected]> Subject: [AusNOG] VOCUS or Server Australia Hi Guys Is anyone using VOCUS down? Apart from Sea-Me-We 3 I see nothing on their Status page. VOCUS NOC mentioned Kent St is down. A bit vague, anyone know more? Regards, Burt Mascareigne _______________________________________________ AusNOG mailing list [email protected] http://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________ AusNOG mailing list [email protected] http://lists.ausnog.net/mailman/listinfo/ausnog
