Post-mortem on Crown Castle / Lightower issue from 10/9... I would have sent to "outages-discussion" list but I was never added to the list.
"On October 9, 2018, Crown Castle Fiber experienced two significant network events on our core backbone that impacted IP and Ethernet services. The root cause of the issue was a hardware issue on a Crown Castle Fiber backbone switch in the New York Metro region. As a result of the hardware issue, Crown Castle Fiber had (3) backbone trunks down. In an effort to restore full capacity, Crown Castle Fiber Network Engineering worked with the hardware vendor support team and it was recommended to reset the faulty hardware. Services began to restore; however, the hardware did not fully recover resulting in unidirectional routing issues. Crown Castle Fiber then disabled the hardware, forcing traffic to alternate paths to restore connectivity. At approximately 2:30pm eastern time, the faulty hardware was inadvertently re-enabled, causing a repeat of the morning's issue. The hardware was subsequently disabled and routing was stabilized. Steps have since been taken to prevent any opportunity for reoccurrence. Any further action taken on this issue will be done in a controlled environment with no impact to customer service. A complete trouble resolution report will be sent out once both Crown Castle and our hardware vendor have an opportunity to complete a full investigation into the issue. Crown Castle is committed to improving network performance and operational processes to reduce downtime. We apologize for the adverse impact this issue may have caused. If additional testing and verification is required, please contact our Network Operations Center at (855) 93-FIBER." Mike Castoro | Director of IT Adrianna Papell Group | 646-723-3885 From: Mike Castoro Sent: Tuesday, October 09, 2018 9:22 AM To: Sulaman, Muhammad <[email protected]>; Brian Weir <[email protected]>; Neil Hanlon <[email protected]>; Daniel Hoye <[email protected]> Cc: [email protected] Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area Still having issues. Going up and down in NYC. Kearny NJ is up. Mike Castoro Director of IT (646) 723-3885 [email protected]<mailto:[email protected]> ________________________________ From: Outages <[email protected]<mailto:[email protected]>> on behalf of Daniel Hoye via Outages <[email protected]<mailto:[email protected]>> Sent: Tuesday, October 9, 2018 7:44:48 AM To: Sulaman, Muhammad; Brian Weir; Neil Hanlon Cc: [email protected]<mailto:[email protected]> Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area External Email! Can confirm issues at PHL colo for Lightower/Crown Castle. Back up now though. From: Outages <[email protected]<mailto:[email protected]>> On Behalf Of Sulaman, Muhammad via Outages Sent: Tuesday, October 9, 2018 7:41 AM To: Brian Weir <[email protected]<mailto:[email protected]>>; Neil Hanlon <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area [Image removed by sender.]This Message was Encrypted. We had issues this morning connecting to our VPN concentrator located in upstate NY. After reading this last message I tried again and was able to connect. Cheers, M From: Brian Weir via Outages <[email protected]<mailto:[email protected]>> Sent: Tuesday, October 9, 2018 7:37 AM To: Neil Hanlon <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area Downtown Manhattan DIA circuit was down as well. However monitoring just reported it as coming back online. On Oct 9, 2018, at 7:32 AM, Neil Hanlon via Outages <[email protected]<mailto:[email protected]>> wrote: I can confirm there's a cut in NJ. Somehow our branch offices in disparate locations is affected... Had to call our account team and the NOC manager to get a straight answer. On Oct 9, 2018, at 07:29, Mike Castoro via Outages <[email protected]<mailto:[email protected]>> wrote: Yes, I lost my DIA lines in Midtown NYC and Kearny NJ around the same time. Mike Castoro Director of IT (646) 723-3885 [email protected]<mailto:[email protected]> ________________________________ From: Outages <[email protected]<mailto:[email protected]>> on behalf of Matthew Huff via Outages <[email protected]<mailto:[email protected]>> Sent: Tuesday, October 9, 2018 7:21:58 AM To: [email protected]<mailto:[email protected]> Subject: [outages] CrownCastle / Lightower issue in NY/NJ area External Email! Around 3:45 AM EST we lost our 10GB WAN circuit from NY to NJ. Around 6:00 AM we started having issue passing traffic via our separate Lightower internet circuit (circuit & BGP peering was up, but packets were getting lost somewhere in their ASN). I opened a case with CrownCastle about 10 10GB circuit about 5:30 AM, but when I called in to report the internet issue, their support line I get in the queue, but never get a tech. I imagine they are getting slammed with calls. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://puck.nether.net/mailman/listinfo/outages ________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://puck.nether.net/mailman/listinfo/outages ________________________________ This email and its contents are confidential. If you have received this email in error, please notify the sender and delete this message. No confidentiality is waived or lost by any transmission errors. Any unauthorized dissemination, distribution or copying of this email is strictly forbidden. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, neither Breckinridge nor the sender will accept liability for any errors or omissions in the contents of this message which arise as a result of email transmission. Breckinridge reserves the right to archive, monitor and review all emails within its network. Nothing contained in this email should be relied upon or construed as financial, legal or tax advice. This email is not a solicitation or offer for the purchase or sale of any interest in any securities or as an official confirmation of any transaction. All investments involve risks, including the potential loss of principal. Any clients mentioned in this email may or may not maintain similar investment objectives as those expressed by your organization or may or may not be familiar to your organization. It is not known whether these clients approve or disapprove of Breckinridge or its advisory services. Past performance is no guarantee of future results. This message may contain the views and opinions of Breckinridge; these are subject to change.
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
