Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Mark Tinka]

On 10/7/21 20:46, Max Tulyev wrote:

We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York 
(USA). I have seen both ports simultaneously dropped traffic volume 
for about one hour today.


Our traffic across Telia dipped at 1600hrs UTC yesterday, and recovered 
2hrs later.


No impact, as I'm certain others with resiliency also saw.

Mark.

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Eric Dugas via NANOG]

I've just sent this update over the Outages ML:

>Dear Customer,
>
>We regret to inform you that your services were affected by an incident 
>occurred at 16:00 UTC during a routine update of a routing policy for 
>aggregated prefixes in Telia Carrier IP Core network. This caused traffic to 
>prefixes contained within the aggregates >to be blackholed, resulting in a 
>impact on some parts of the network.
>
>When the underlying problem source was traced, the configuration was rolled 
>back to the earlier working version of the routing policy (17:05 UTC). 
>Affected services started to recover gradually after this operation was 
>applied. No further disturbances >related to this incident are expected.
>
>Due to the wide impact on several customer services, a number of complaints 
>was received by our Customer Support Centre, resulting in delays in the 
>communication with customers via email and/or phone. We apologize for any 
>inconvenience this incident >has caused to your services.

On Thu, Oct 7, 2021 at 3:48 PM Max Tulyev  wrote:
>
> Really it depends on the problem source. BGP do not know either route
> really reachable or not. This time we was just lucky.
>
> 07.10.21 22:36, Ca By пише:
> >
> >
> > On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev  > > wrote:
> >
> > We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York
> > (USA). I have seen both ports simultaneously dropped traffic volume for
> > about one hour today.
> >
> > It was not critical (for us), as traffic was shifted to another links,
> > and there was no unreachable destinations like BGP announces with
> > traffic blackholed. But looks strange.
> >
> >
> > Thats why it is called the bridging gap protocol
> >
> > While Telia barfed, it bridged the gap.
> >
> > See, bgp is not all bad. There are no bad routes, only bad days
> >
> >
> >
> > 07.10.21 21:23, Vincentz Petzholtz пише:
> >  > Hi everyone,
> >  >
> >  > Looks like the season for outages is on. Does anyone has more
> > details regarding the issues at Telia? I didn't found any public
> > available information. They say it's over but this is clearly not
> > the case.
> >  >
> >  > Best regards,
> >  > Vincentz
> >  >
> >

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Max Tulyev]

Really it depends on the problem source. BGP do not know either route 
really reachable or not. This time we was just lucky.


07.10.21 22:36, Ca By пише:



On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev > wrote:


We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York
(USA). I have seen both ports simultaneously dropped traffic volume for
about one hour today.

It was not critical (for us), as traffic was shifted to another links,
and there was no unreachable destinations like BGP announces with
traffic blackholed. But looks strange.


Thats why it is called the bridging gap protocol

While Telia barfed, it bridged the gap.

See, bgp is not all bad. There are no bad routes, only bad days



07.10.21 21:23, Vincentz Petzholtz пише:
 > Hi everyone,
 >
 > Looks like the season for outages is on. Does anyone has more
details regarding the issues at Telia? I didn't found any public
available information. They say it's over but this is clearly not
the case.
 >
 > Best regards,
 > Vincentz
 >

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Ca By]

On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev  wrote:

> We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York
> (USA). I have seen both ports simultaneously dropped traffic volume for
> about one hour today.
>
> It was not critical (for us), as traffic was shifted to another links,
> and there was no unreachable destinations like BGP announces with
> traffic blackholed. But looks strange.
>

Thats why it is called the bridging gap protocol

While Telia barfed, it bridged the gap.

See, bgp is not all bad. There are no bad routes, only bad days



> 07.10.21 21:23, Vincentz Petzholtz пише:
> > Hi everyone,
> >
> > Looks like the season for outages is on. Does anyone has more details
> regarding the issues at Telia? I didn't found any public available
> information. They say it's over but this is clearly not the case.
> >
> > Best regards,
> > Vincentz
> >
>

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Max Tulyev]

We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York 
(USA). I have seen both ports simultaneously dropped traffic volume for 
about one hour today.


It was not critical (for us), as traffic was shifted to another links, 
and there was no unreachable destinations like BGP announces with 
traffic blackholed. But looks strange.


07.10.21 21:23, Vincentz Petzholtz пише:

Hi everyone,

Looks like the season for outages is on. Does anyone has more details regarding 
the issues at Telia? I didn't found any public available information. They say 
it's over but this is clearly not the case.

Best regards,
Vincentz

Global issues @ Telia - doing a "FB/hold my beer" move?

[Vincentz Petzholtz]

Hi everyone,

Looks like the season for outages is on. Does anyone has more details regarding 
the issues at Telia? I didn't found any public available information. They say 
it's over but this is clearly not the case.

Best regards,
Vincentz


signature.asc
Description: Message signed with OpenPGP

Re: FB?

[Robert Webb]

No one looks at dates on Facebook posts.

On Thu, Mar 14, 2019, 17:10 Luke Guillory  wrote:

> That’s old.
>
>
>
> By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
>
>
>
>
>
> Luke
>
>
>
> Ns
>
>
>
>
>
>
>
> *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Selphie
> Keller
> *Sent:* Thursday, March 14, 2019 4:06 PM
> *To:* Mike Hammett
> *Cc:* NANOG list
> *Subject:* Re: FB?
>
>
>
> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
>
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
>
>

Re: FB?

[Ross Tajvar]

The cache invalidation thing is incorrect according to an Facebook SWE I
talked to. He wouldn't tell me what it actually was though, basically
saying "you have to know our infrastructure to understand and I can't tell
you that."

On Thu, Mar 14, 2019, 5:28 PM cosmo  wrote:

> Yes, evidently someone screenshotted it and it was making the rounds on
> social media this morning, sans the date.
>
> So now back to other theories.
>
> On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz 
> wrote:
>
>> The date on that is 2010.
>>
>> On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller 
>> wrote:
>> >
>> > I did see this article indicating they had somehow invalidated their
>> cache in a botched deployment of changes -
>> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>> >
>> > On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>> >>
>> >> So what happened at Facebook today? I saw one article quoting Roland
>> saying it was a route leak, but I haven't seen any other sources that
>> aren't just quoting Roland. Usually there are a few independent posts out
>> there by now.
>> >>
>> >>
>> >>
>> >> -
>> >> Mike Hammett
>> >> Intelligent Computing Solutions
>> >> http://www.ics-il.com
>> >>
>> >> Midwest-IX
>> >> http://www.midwest-ix.com
>> >>
>>
>>
>> --
>> Jeff Shultz
>> Central Office Technician
>> SCTC
>> (503) 769-2125
>> Go Big  Ask for Gig
>>
>> --
>> Like us on Social Media for News, Promotions, and other information!!
>>
>>
>> 
>> 
>> 
>> 
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _ This message
>> contains confidential information and is intended only for the individual
>> named. If you are not the named addressee you should not disseminate,
>> distribute or copy this e-mail. Please notify the sender immediately by
>> e-mail if you have received this e-mail by mistake and delete this e-mail
>> from your system. E-mail transmission cannot be guaranteed to be secure
>> or
>> error-free as information could be intercepted, corrupted, lost,
>> destroyed,
>> arrive late or incomplete, or contain viruses. The sender therefore does
>> not accept liability for any errors or omissions in the contents of this
>> message, which arise as a result of e-mail transmission. _
>>
>>

Re: FB?

[cosmo]

Yes, evidently someone screenshotted it and it was making the rounds on
social media this morning, sans the date.

So now back to other theories.

On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz  wrote:

> The date on that is 2010.
>
> On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller 
> wrote:
> >
> > I did see this article indicating they had somehow invalidated their
> cache in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
> >
> > On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
> >>
> >> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
> >>
> >>
> >>
> >> -
> >> Mike Hammett
> >> Intelligent Computing Solutions
> >> http://www.ics-il.com
> >>
> >> Midwest-IX
> >> http://www.midwest-ix.com
> >>
>
>
> --
> Jeff Shultz
> Central Office Technician
> SCTC
> (503) 769-2125
> Go Big  Ask for Gig
>
> --
> Like us on Social Media for News, Promotions, and other information!!
>
>
> 
> 
> 
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
> _ This message
> contains confidential information and is intended only for the individual
> named. If you are not the named addressee you should not disseminate,
> distribute or copy this e-mail. Please notify the sender immediately by
> e-mail if you have received this e-mail by mistake and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be secure or
> error-free as information could be intercepted, corrupted, lost,
> destroyed,
> arrive late or incomplete, or contain viruses. The sender therefore does
> not accept liability for any errors or omissions in the contents of this
> message, which arise as a result of e-mail transmission. _
>
>

Re: FB?

[cosmo]

Looks like Google recently posted their post-mortem of their outage on the
12th
https://status.cloud.google.com/incident/storage/19002

On Thu, Mar 14, 2019 at 1:21 PM Suresh Ramasubramanian 
wrote:

> That's a 2010 outage that someone dug out and was doing the rounds as a
> new one
>
> --srs
>
> --
> *From:* NANOG  on behalf of cosmo <
> clinton.mie...@gmail.com>
> *Sent:* Thursday, March 14, 2019 9:50 PM
> *To:* Bryan Holloway
> *Cc:* nanog@nanog.org
> *Subject:* Re: FB?
>
> Facebook pushed an update to their code that manages cookies, that had a
> rather severe bug in it that resulted in a large flood of requests to their
> database servers. To deal with this load, they had to prevent all writes
> and then slowly allow people back on.
>
> I saw the writeup for it last night but cannot seem to find it now! Grrr.
> Did I dream it?
>
> On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway  wrote:
>
>>
>> On 3/14/19 9:06 AM, Tom Beecher wrote:
>> > As much as I wanted to crack jokes because I cannot stand Facebook (the
>> > product), much love to all you FB engineers that went through (and are
>> > probably still going through) much hell.
>> >
>>
>> +1 on both counts.
>>
>> We've all been there; no bueno.
>>
>

RE: FB?

[Ray Van Dolson]

https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_facebook_status_1106229690069442560=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=zrKUWVShQdFllKTGbJE5kITG87q7KNJHo0bD6aETBBk=

From: NANOG  On Behalf Of Luke Guillory
Sent: Thursday, March 14, 2019 2:09 PM
To: Selphie Keller ; Mike Hammett 
Cc: NANOG list 
Subject: RE: FB?

That’s old.

By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM


Luke

Ns




From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller
Sent: Thursday, March 14, 2019 4:06 PM
To: Mike Hammett
Cc: NANOG list
Subject: Re: FB?

I did see this article indicating they had somehow invalidated their cache in a 
botched deployment of changes - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=1EmIo8GEivgILxC4jZzEdBpYWt5R9CZ5cXhtr6i55rc=<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_=DwMGaQ=n6-cguzQvX_tUIrZOS_4Og=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY=ryKQO6pOvAkwCIA0hQDtc2YA6zayXLUHgF1cQYftw6s=>

On Thu, 14 Mar 2019 at 06:18, Mike Hammett 
mailto:na...@ics-il.net>> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now.


-
Mike Hammett
Intelligent Computing Solutions
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=EAZZC6r_-2rdFCKgq9XpQy30F7OH79M6sZPNvXq0FPA=<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com=DwMGaQ=n6-cguzQvX_tUIrZOS_4Og=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY=mqEeLmJWERmxCoZUEBneCbzo6adoCrpBOoe6fg-HoIk=>

Midwest-IX
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=fEoBWpTXgY7eXczzc8vo7VHbvopKqDWk6Xz2XYutL0k=<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com=DwMGaQ=n6-cguzQvX_tUIrZOS_4Og=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY=F-6Dzwg1bC3hEWn0_nXjswSlbrgbBSq3DLpcpMtdh-Q=>

Re: FB?

[Selphie Keller]

Yeah I just saw that date and that is odd, I got the link yesterday from
somewhere and didn't notice the date was old.

They do mention the configuration change issue in this one though that is
dated today 14th -
https://www.cbsnews.com/news/facebook-instagram-down-wednesday-facebook-blames-server-configuration-for-longest-ever-outage/

>From my understanding yesterday is they invalidated their world cache and
all the traffic hit their backend quickly overwhelming their servers. It
was strange that they didn't really talk about the issue just some brief
messages on their twitter saying it wasn't ddos.




On Thu, 14 Mar 2019 at 15:08, Luke Guillory 
wrote:

> That’s old.
>
>
>
> By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
>
>
>
>
>
> Luke
>
>
>
> Ns
>
>
>
>
>
>
>
> *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Selphie
> Keller
> *Sent:* Thursday, March 14, 2019 4:06 PM
> *To:* Mike Hammett
> *Cc:* NANOG list
> *Subject:* Re: FB?
>
>
>
> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
>
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
>
>

Re: FB?

[Jeff Shultz]

The date on that is 2010.

On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller  wrote:
>
> I did see this article indicating they had somehow invalidated their cache in 
> a botched deployment of changes - 
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>>
>> So what happened at Facebook today? I saw one article quoting Roland saying 
>> it was a route leak, but I haven't seen any other sources that aren't just 
>> quoting Roland. Usually there are a few independent posts out there by now.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>


-- 
Jeff Shultz
Central Office Technician
SCTC
(503) 769-2125
Go Big  Ask for Gig

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: FB?

[cosmo]

Ah-ha, that is indeed the write-up I saw. 8 years old!


On Thu, Mar 14, 2019, 2:07 PM Selphie Keller 
wrote:

> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
>> So what happened at Facebook today? I saw one article quoting Roland
>> saying it was a route leak, but I haven't seen any other sources that
>> aren't just quoting Roland. Usually there are a few independent posts out
>> there by now.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>>

RE: FB?

[Luke Guillory]

That’s old.

By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM


Luke

Ns




From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller
Sent: Thursday, March 14, 2019 4:06 PM
To: Mike Hammett
Cc: NANOG list
Subject: Re: FB?

I did see this article indicating they had somehow invalidated their cache in a 
botched deployment of changes - 
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/

On Thu, 14 Mar 2019 at 06:18, Mike Hammett 
mailto:na...@ics-il.net>> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: FB?

[Selphie Keller]

I did see this article indicating they had somehow invalidated their cache
in a botched deployment of changes -
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/

On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:

> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>

Re: FB?

[Suresh Ramasubramanian]

That's a 2010 outage that someone dug out and was doing the rounds as a new one

--srs


From: NANOG  on behalf of cosmo 

Sent: Thursday, March 14, 2019 9:50 PM
To: Bryan Holloway
Cc: nanog@nanog.org
Subject: Re: FB?

Facebook pushed an update to their code that manages cookies, that had a rather 
severe bug in it that resulted in a large flood of requests to their database 
servers. To deal with this load, they had to prevent all writes and then slowly 
allow people back on.

I saw the writeup for it last night but cannot seem to find it now! Grrr. Did I 
dream it?

On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway 
mailto:br...@shout.net>> wrote:

On 3/14/19 9:06 AM, Tom Beecher wrote:
> As much as I wanted to crack jokes because I cannot stand Facebook (the
> product), much love to all you FB engineers that went through (and are
> probably still going through) much hell.
>

+1 on both counts.

We've all been there; no bueno.

Re: FB?

[cosmo]

Facebook pushed an update to their code that manages cookies, that had a
rather severe bug in it that resulted in a large flood of requests to their
database servers. To deal with this load, they had to prevent all writes
and then slowly allow people back on.

I saw the writeup for it last night but cannot seem to find it now! Grrr.
Did I dream it?

On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway  wrote:

>
> On 3/14/19 9:06 AM, Tom Beecher wrote:
> > As much as I wanted to crack jokes because I cannot stand Facebook (the
> > product), much love to all you FB engineers that went through (and are
> > probably still going through) much hell.
> >
>
> +1 on both counts.
>
> We've all been there; no bueno.
>

Re: FB?

[Bryan Holloway]

On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the 
product), much love to all you FB engineers that went through (and are 
probably still going through) much hell.




+1 on both counts.

We've all been there; no bueno.

Re: FB? / AS 200020 leak

[Job Snijders]

Hi,

On Thu, Mar 14, 2019 at 02:04:39PM +, Jeroen Wunnink wrote:
> The route-leak was something different that seems to have mainly hit
> west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in
> the *NOG communities still digging at the complete details of that
> right now, but it currently points to have originated from AS200020,
> impacting a few large upstreams for a short period of time.

Here are some details of prefixes affected (courtesy of Doug Madory).
The percent at the beginning is the percentage of the peering sources
that saw each prefix leaked. Last column is the AS_PATH

March 14th, 2019 - 16:43 UTC was the start of the BGP leak incident.

The leak was very serious in terms of negative impact, it affected many
West European access providers (for instance AS 1136 has over 50% of
Dutch access market).

Kind regards,

Job

70.6% 92.68.0.0/14 KPN B.V. NL ... 200020 1136
70.6% 92.64.0.0/14 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 1136
70.4% 93.154.64.0/18 KPN B.V. NL ... 200020 1136
70.4% 93.154.0.0/18 KPN B.V. NL ... 200020 1136
70.4% 86.88.0.0/13 KPN B.V. NL ... 200020 1136
70.4% 86.80.0.0/13 KPN B.V. NL ... 200020 1136
70.4% 84.84.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 84.80.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 81.206.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 81.204.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 80.61.0.0/16 Customers NL ... 200020 1136
70.4% 80.60.0.0/16 Customers NL ... 200020 1136
70.4% 77.62.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.60.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.170.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.168.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.164.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 77.160.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 62.12.0.0/20 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 1136
70.4% 46.145.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 46.144.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 31.161.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 31.160.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 145.7.128.0/17 KPN B.V. NL ... 200020 1136
70.4% 145.133.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 145.132.0.0/16 KPN B.V. NL ... 200020 1136
70.1% 89.200.64.0/18 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 89.200.0.0/18 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.32.0/19 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 83.232.0.0/19 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.0.0/17 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 82.171.96.0/19 Customers NL ... 200020 1136
70.1% 82.171.64.0/19 Customers NL ... 200020 1136
70.1% 82.171.32.0/19 Customers NL ... 200020 1136
70.1% 82.171.192.0/18 Customers NL ... 200020 1136
70.1% 82.171.128.0/18 Customers NL ... 200020 1136
70.1% 82.171.0.0/19 Customers NL ... 200020 1136
70.1% 82.170.128.0/17 Customers NL ... 200020 1136
70.1% 82.170.0.0/17 Customers NL ... 200020 1136
70.1% 82.169.96.0/20 Customers NL ... 200020 1136
70.1% 82.169.80.0/20 Customers NL ... 200020 1136
70.1% 82.169.64.0/20 Customers NL ... 200020 1136
70.1% 82.169.32.0/19 Customers NL ... 200020 1136
70.1% 82.169.224.0/19 Customers NL ... 200020 1136
70.1% 82.169.192.0/19 Customers NL ... 200020 1136
70.1% 82.169.176.0/20 Customers NL ... 200020 1136
70.1% 82.169.160.0/20 Customers NL ... 200020 1136
70.1% 82.169.144.0/20 Customers NL ... 200020 1136
70.1% 82.169.128.0/20 Customers NL ... 200020 1136
70.1% 82.169.112.0/20 Customers NL ... 200020 1136
70.1% 82.169.0.0/19 Customers NL ... 200020 1136
70.1% 82.168.64.0/18 Customers NL ... 200020 1136
70.1% 82.168.240.0/20 Customers NL ... 200020 1136
70.1% 82.168.224.0/20 Customers NL ... 200020 1136
70.1% 82.168.208.0/20 Customers NL ... 200020 1136
70.1% 82.168.192.0/20 Customers NL ... 200020 1136
70.1% 82.168.160.0/19 Customers NL ... 200020 1136
70.1% 82.168.128.0/19 Customers NL ... 200020 1136
70.1% 82.168.0.0/18 Customers NL ... 200020 1136
70.1% 82.136.224.0/19 KPN B.V. NL ... 200020 1136
70.1% 82.136.192.0/19 KPN B.V. NL ... 200020 1136
70.1% 80.60.224.0/20 Customers Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 80.60.224.0/19 Customers Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 77.173.128.0/17 Customers NL ... 200020 1136
70.1% 77.173.0.0/17 Customers NL ... 200020 1136
70.1% 77.172.128.0/17 Customers NL ... 200020 1136
70.1% 77.172.0.0/17 Customers NL ... 200020 1136
70.1% 62.41.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.41.0.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.25.32.0/19 KPN B.V. NL ... 200020 1136
70.1% 62.25.0.0/19 KPN B.V. NL ... 200020 1136
70.1% 62.21.192.0/18 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 62.21.128.0/18 KPN B.V. NL ... 200020 1136
70.1% 62.207.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.207.0.0/17 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 62.133.96.0/19 KPN Mobile The Netherlands B.V. NL ... 

Re: FB? - route leak AS200020

[Pim van Stam]

> On 14 Mar 2019, at 15:04, Jeroen Wunnink  wrote:
> 
> The route-leak was something different that seems to have mainly hit 
> west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG 
> communities still digging at the complete details of that right now, but it 
> currently points to have originated from AS200020, impacting a few large 
> upstreams for a short period of time.
>  
> So unless this leak caused a catastrophic cascade in FB’s network somehow, it 
> seems to be unrelated.
> It looked like a valid suspect because timing was very similar between the 
> start of the FB outage and the leak.

Hello,

I’m an engineer at AS200020. We indeed had a route leak yesterday between 16:45 
and 17:07 UTC.
This was caused by a mistake in the configuration of one of ou core routers by 
one of our engineers.
We leaked full table to one of our transits, which is apparently accepted. 
We’ve seen some prefixes from a couple of AS numbers coming in. Facebook was 
not one of these AS’es. As far as I can tell it’s not related at all to the 
major incidents at Facebook.

We are very sorry for any inconvenience people could have experienced.
If you received complaints in the mentioned time period and want to know if it 
could be related, please let me know.

Best regards,

Pim van Stam
NBIP-naWas / AS200020

Re: FB?

[Tom Beecher]

As much as I wanted to crack jokes because I cannot stand Facebook (the
product), much love to all you FB engineers that went through (and are
probably still going through) much hell.

On Thu, Mar 14, 2019 at 9:58 AM Jason Suter  wrote:

>
> I found this article
> <https://www.datacenterdynamics.com/news/facebook-instagram-whatsapp-suffer-global-outage/>
>  but
> no real answers.
>
> On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.)  wrote:
>
>> So what happened yesterday?
>>
>>
>>
>> *From:* NANOG  *On Behalf Of *Mike Hammett
>> *Sent:* Thursday, March 14, 2019 8:29 AM
>> *To:* Roland Dobbins 
>> *Cc:* nanog@nanog.org
>> *Subject:* Re: FB?
>>
>>
>>
>> Do you have a link to the clarification? With the high jitter of news,
>> all I'm finding is people parroting the original statement.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>>
>> --
>>
>> *From: *"Roland Dobbins" 
>> *To: *nanog@nanog.org
>> *Sent: *Thursday, March 14, 2019 7:23:00 AM
>> *Subject: *Re: FB?
>>
>> On 14 Mar 2019, at 19:17, Mike Hammett wrote:
>>
>> > I saw one article quoting Roland saying it was a route leak, but I
>> > haven't seen any other sources that aren't just quoting Roland.
>>
>> That was the result of a miscommunication; a clarification has been
>> issued, FYI.
>>
>> 
>> Roland Dobbins 
>>
>>
>>
>

Re: FB?

[Jeroen Wunnink]

The route-leak was something different that seems to have mainly hit 
west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG 
communities still digging at the complete details of that right now, but it 
currently points to have originated from AS200020, impacting a few large 
upstreams for a short period of time.

So unless this leak caused a catastrophic cascade in FB’s network somehow, it 
seems to be unrelated.
It looked like a valid suspect because timing was very similar between the 
start of the FB outage and the leak.




Jeroen Wunnink
Sr. Manager - Integration Engineering

www.gtt.net<http://www.gtt.net/>

[id:image001.png@01D37331.D1301F60]


From: NANOG  on behalf of "Kain, Rebecca (.)" 

Date: Thursday, 14 March 2019 at 14:36
To: Mike Hammett , Roland Dobbins 

Cc: "nanog@nanog.org" 
Subject: RE: FB?

So what happened yesterday?

From: NANOG  On Behalf Of Mike Hammett
Sent: Thursday, March 14, 2019 8:29 AM
To: Roland Dobbins 
Cc: nanog@nanog.org
Subject: Re: FB?

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com<https://url.emailprotection.link/?b6x9-Fnx1LAfWyqRtESstZgT_vbCd3MONOqCVFZ0R_BHO80Ox-i-8zIm9qQ1soVeoZrxi8iA3iwJ_b5GpLUD9bw~~>

Midwest-IX
http://www.midwest-ix.com<https://url.emailprotection.link/?bTe_C0izNVY8kqDiMqBi3hZOjTZ-zNYEmkKhYlGBamvFdtMk4Ad_MAPQonzhIUmKh8G8FAwXEtjMYejM3PlLz6A~~>


From: "Roland Dobbins" 
mailto:roland.dobb...@netscout.com>>
To: nanog@nanog.org<mailto:nanog@nanog.org>
Sent: Thursday, March 14, 2019 7:23:00 AM
Subject: Re: FB?

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been
issued, FYI.


Roland Dobbins mailto:roland.dobb...@netscout.com>>

Re: FB?

[Jason Suter]

I found this article
<https://www.datacenterdynamics.com/news/facebook-instagram-whatsapp-suffer-global-outage/>
but
no real answers.

On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.)  wrote:

> So what happened yesterday?
>
>
>
> *From:* NANOG  *On Behalf Of *Mike Hammett
> *Sent:* Thursday, March 14, 2019 8:29 AM
> *To:* Roland Dobbins 
> *Cc:* nanog@nanog.org
> *Subject:* Re: FB?
>
>
>
> Do you have a link to the clarification? With the high jitter of news, all
> I'm finding is people parroting the original statement.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
> --
>
> *From: *"Roland Dobbins" 
> *To: *nanog@nanog.org
> *Sent: *Thursday, March 14, 2019 7:23:00 AM
> *Subject: *Re: FB?
>
> On 14 Mar 2019, at 19:17, Mike Hammett wrote:
>
> > I saw one article quoting Roland saying it was a route leak, but I
> > haven't seen any other sources that aren't just quoting Roland.
>
> That was the result of a miscommunication; a clarification has been
> issued, FYI.
>
> 
> Roland Dobbins 
>
>
>

RE: FB?

[Kain, Rebecca (.)]

So what happened yesterday?

From: NANOG  On Behalf Of Mike Hammett
Sent: Thursday, March 14, 2019 8:29 AM
To: Roland Dobbins 
Cc: nanog@nanog.org
Subject: Re: FB?

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Roland Dobbins" 
mailto:roland.dobb...@netscout.com>>
To: nanog@nanog.org<mailto:nanog@nanog.org>
Sent: Thursday, March 14, 2019 7:23:00 AM
Subject: Re: FB?

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been
issued, FYI.


Roland Dobbins mailto:roland.dobb...@netscout.com>>

Re: FB?

[Mike Hammett]

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Roland Dobbins"  
To: nanog@nanog.org 
Sent: Thursday, March 14, 2019 7:23:00 AM 
Subject: Re: FB? 

On 14 Mar 2019, at 19:17, Mike Hammett wrote: 

> I saw one article quoting Roland saying it was a route leak, but I 
> haven't seen any other sources that aren't just quoting Roland. 

That was the result of a miscommunication; a clarification has been 
issued, FYI. 

 
Roland Dobbins  

Re: FB?

[Dobbins, Roland]

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I 
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been 
issued, FYI.


Roland Dobbins 

FB?

[Mike Hammett]

So what happened at Facebook today ? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com