Re: Global issues @ Telia - doing a "FB/hold my beer" move?

On 10/7/21 20:46, Max Tulyev wrote: We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York (USA). I have seen both ports simultaneously dropped traffic volume for about one hour today. Our traffic across Telia dipped at 1600hrs UTC yesterday, and recovered 2hrs later. No

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

I've just sent this update over the Outages ML: >Dear Customer, > >We regret to inform you that your services were affected by an incident >occurred at 16:00 UTC during a routine update of a routing policy for >aggregated prefixes in Telia Carrier IP Core network. This caused traffic to

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

Really it depends on the problem source. BGP do not know either route really reachable or not. This time we was just lucky. 07.10.21 22:36, Ca By пише: On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev > wrote: We have 2 ports from Telia, one in Kiev (Ukraine) and

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev wrote: > We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York > (USA). I have seen both ports simultaneously dropped traffic volume for > about one hour today. > > It was not critical (for us), as traffic was shifted to another links, >

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York (USA). I have seen both ports simultaneously dropped traffic volume for about one hour today. It was not critical (for us), as traffic was shifted to another links, and there was no unreachable destinations like BGP

Global issues @ Telia - doing a "FB/hold my beer" move?

Hi everyone, Looks like the season for outages is on. Does anyone has more details regarding the issues at Telia? I didn't found any public available information. They say it's over but this is clearly not the case. Best regards, Vincentz signature.asc Description: Message signed with

Re: FB?

ANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Selphie > Keller > *Sent:* Thursday, March 14, 2019 4:06 PM > *To:* Mike Hammett > *Cc:* NANOG list > *Subject:* Re: FB? > > > > I did see this article indicating they had somehow invalidated their cache > in a

Re: FB?

The cache invalidation thing is incorrect according to an Facebook SWE I talked to. He wouldn't tell me what it actually was though, basically saying "you have to know our infrastructure to understand and I can't tell you that." On Thu, Mar 14, 2019, 5:28 PM cosmo wrote: > Yes, evidently

Re: FB?

Yes, evidently someone screenshotted it and it was making the rounds on social media this morning, sans the date. So now back to other theories. On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz wrote: > The date on that is 2010. > > On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller > wrote: > > > > I

Re: FB?

; > --srs > > -- > *From:* NANOG on behalf of cosmo < > clinton.mie...@gmail.com> > *Sent:* Thursday, March 14, 2019 9:50 PM > *To:* Bryan Holloway > *Cc:* nanog@nanog.org > *Subject:* Re: FB? > > Facebook pushed an update to their code

RE: FB?

, March 14, 2019 2:09 PM To: Selphie Keller ; Mike Hammett Cc: NANOG list Subject: RE: FB? That’s old. By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM Luke Ns From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller Sent: Thursday, March 14, 2019 4:06 PM

Re: FB?

...@nanog.org] *On Behalf Of *Selphie > Keller > *Sent:* Thursday, March 14, 2019 4:06 PM > *To:* Mike Hammett > *Cc:* NANOG list > *Subject:* Re: FB? > > > > I did see this article indicating they had somehow invalidated their cache > in a botched deployment of chan

Re: FB?

The date on that is 2010. On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller wrote: > > I did see this article indicating they had somehow invalidated their cache in > a botched deployment of changes - >

Re: FB?

Ah-ha, that is indeed the write-up I saw. 8 years old! On Thu, Mar 14, 2019, 2:07 PM Selphie Keller wrote: > I did see this article indicating they had somehow invalidated their cache > in a botched deployment of changes - >

RE: FB?

That’s old. By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM Luke Ns From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller Sent: Thursday, March 14, 2019 4:06 PM To: Mike Hammett Cc: NANOG list Subject: Re: FB? I did see this article indicating they had

Re: FB?

I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/ On Thu, 14 Mar 2019 at 06:18, Mike Hammett wrote: > So what happened at Facebook today?

Re: FB?

That's a 2010 outage that someone dug out and was doing the rounds as a new one --srs From: NANOG on behalf of cosmo Sent: Thursday, March 14, 2019 9:50 PM To: Bryan Holloway Cc: nanog@nanog.org Subject: Re: FB? Facebook pushed an update to their code

Re: FB?

but cannot seem to find it now! Grrr. Did I dream it? On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway wrote: > > On 3/14/19 9:06 AM, Tom Beecher wrote: > > As much as I wanted to crack jokes because I cannot stand Facebook (the > > product), much love to all you FB engineers

Re: FB?

On 3/14/19 9:06 AM, Tom Beecher wrote: As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell. +1 on both counts. We've all been there; no bueno.

Re: FB? / AS 200020 leak

Hi, On Thu, Mar 14, 2019 at 02:04:39PM +, Jeroen Wunnink wrote: > The route-leak was something different that seems to have mainly hit > west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in > the *NOG communities still digging at the complete details of that > right now, but it

Re: FB? - route leak AS200020

e timing was very similar between the > start of the FB outage and the leak. Hello, I’m an engineer at AS200020. We indeed had a route leak yesterday between 16:45 and 17:07 UTC. This was caused by a mistake in the configuration of one of ou core routers by one of our engineers. We leaked full

Re: FB?

As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell. On Thu, Mar 14, 2019 at 9:58 AM Jason Suter wrote: > > I found this article &g

Re: FB?

upstreams for a short period of time. So unless this leak caused a catastrophic cascade in FB’s network somehow, it seems to be unrelated. It looked like a valid suspect because timing was very similar between the start of the FB outage and the leak. Jeroen Wunnink Sr. Manager - Integration

Re: FB?

tt > *Sent:* Thursday, March 14, 2019 8:29 AM > *To:* Roland Dobbins > *Cc:* nanog@nanog.org > *Subject:* Re: FB? > > > > Do you have a link to the clarification? With the high jitter of news, all > I'm finding is people parroting the original statement. > > > > -

RE: FB?

So what happened yesterday? From: NANOG On Behalf Of Mike Hammett Sent: Thursday, March 14, 2019 8:29 AM To: Roland Dobbins Cc: nanog@nanog.org Subject: Re: FB? Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement

Re: FB?

oland Dobbins" To: nanog@nanog.org Sent: Thursday, March 14, 2019 7:23:00 AM Subject: Re: FB? On 14 Mar 2019, at 19:17, Mike Hammett wrote: > I saw one article quoting Roland saying it was a route leak, but I > haven't seen any other sources that aren't just quoting Roland. Tha

Re: FB?

On 14 Mar 2019, at 19:17, Mike Hammett wrote: > I saw one article quoting Roland saying it was a route leak, but I > haven't seen any other sources that aren't just quoting Roland. That was the result of a miscommunication; a clarification has been issued, FYI.

FB?

So what happened at Facebook today ? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now. - Mike Hammett Intelligent Computing Solutions