Someone didn't have proper protections in place it looks like. On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages < [email protected]> wrote:
> So another BGP highjacking... > > - Mike Bolitho > > On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages < > [email protected]> wrote: > >> Found this on Reddit as an initial RFO, so keep in mind the source >> (although r/sysadmin is usually pretty good): >> >> >> >> “A 3rd party network provider was advertising routes which resulted in >> our WorldWide traffic becoming severely impeded. This led to IBM Cloud >> clients being unable to log-in to their accounts, greatly limited >> internet/DC connectivity and other significant network route related >> impacts. Network Specialists have made adjustments to route policies to >> restore network access, and alleviate the impacts. The overall incident >> lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed >> Customer Incident Report/Root Cause Analysis as soon as possible” >> >> >> >> Greg Dickinson >> >> Network Engineer >> >> >> >> [image: cid:[email protected]] >> >> 234 Goodwin Crest Drive, Suite 500 >> >> Birmingham, AL 35209 >> >> Office: 205.917.2407 | Mobile: 205.234.6427 >> >> Fax: 205.945.0515 | >> >> [email protected] | BryantBank.com <http://bryantbank.com/> >> >> Unbeatable Service. Legendary Results. >> >> Facebook <https://facebook.com/bryantbank/> | Twitter >> <https://twitter.com/bryantbank/> >> >> >> >> *From:* Outages <[email protected]> *On Behalf Of *Jason Kuehl >> via Outages >> *Sent:* Tuesday, June 9, 2020 8:38 PM >> *To:* James Brown <[email protected]> >> *Cc:* [email protected] >> *Subject:* Re: [outages] IBM Cloud / NetworkLayer Outage >> >> >> >> >> >> This Message originates from* outside* Bryant Bank. Please use >> *caution* when opening this correspondence, attachments or hyperlinks >> (URLs). If you have questions, please contact IT Support. Thank you. >> >> I can't wait for this post mortem. >> >> >> >> On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages < >> [email protected]> wrote: >> >> As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look >> normal to me. >> >> >> >> On Tue, Jun 9, 2020 at 5:27 PM James Brown <[email protected]> wrote: >> >> Just wanted to share updates: >> >> >> >> I got through to someone at IBM who confirmed the level of chaos over >> there. It looks like they've started deploying some mitigations because I'm >> seeing packet loss drop off but latency is now awful for all US locations. >> It looks like IBM has failed over all of their peering to go through their >> European facilities because every return packet from every POP is coming >> from lon03.networklayer.com >> <http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com> >> or ams02.networklayer.com >> <http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com> >> (via Telia). >> >> >> >> On Tue, Jun 9, 2020 at 4:12 PM James Brown <[email protected]> wrote: >> >> It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global >> outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / >> CenturyLink, and a bunch of other providers. Has anyone been able to get >> more details? >> >> >> >> -- >> >> James Brown >> >> Network Engineer >> >> >> >> >> -- >> >> James Brown >> >> Engineer >> >> >> >> >> -- >> >> James Brown >> >> Engineer >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> <https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages> >> >> >> >> >> -- >> >> Sincerely, >> >> Jason W Kuehl >> Cell 920-419-8983 >> [email protected] >> >> >> NOTICE: This electronic mail message and any files transmitted with it >> are intended exclusively for the individual or entity to which it is >> addressed. The message, together with any attachment, may contain >> confidential and/or privileged information. Any unauthorized review, use, >> print, save, copy, disclosure or distribution is strictly prohibited. If >> you have received this message in error, please immediately advise the >> sender by reply email and delete copies. Thank you. >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > -- Sincerely, Jason W Kuehl Cell 920-419-8983 [email protected]
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
