Re: Global issues @ Telia - doing a "FB/hold my beer" move?

On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev wrote: > We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York > (USA). I have seen both ports simultaneously dropped traffic volume for > about one hour today. > > It was not critical (for us), as traffic was shifted to another links, >

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

Really it depends on the problem source. BGP do not know either route really reachable or not. This time we was just lucky. 07.10.21 22:36, Ca By пише: On Thu, Oct 7, 2021 at 11:47 AM Max Tulyev > wrote: We have 2 ports from Telia, one in Kiev (Ukraine) and

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

I've just sent this update over the Outages ML: >Dear Customer, > >We regret to inform you that your services were affected by an incident >occurred at 16:00 UTC during a routine update of a routing policy for >aggregated prefixes in Telia Carrier IP Core network. This caused traffic to

Re: DNS pulling BGP routes?

On Thu, Oct 7, 2021 at 10:23 AM Masataka Ohta wrote: > William Herrin wrote: > > Facebook's _internal_ DNS, while not anycasted, followed a similar > > logic: if the data center is isolated and their data goes stale, they > > stop serving potentially wrong answers. > > As I already wrote, that is

Re: DNS pulling BGP routes?

William Herrin wrote: Facebook's _internal_ DNS, while not anycasted, followed a similar logic: if the data center is isolated and their data goes stale, they stop serving potentially wrong answers. As I already wrote, that is a standard mechanism of DNS with SOA expiration period as is

Re: DNS pulling BGP routes?

- On Oct 7, 2021, at 9:03 PM, Masataka Ohta mo...@necom830.hpcl.titech.ac.jp wrote: Hi, > It means DNS management of facebook is poor. Whenever there is an aviation incident, the keyboard warriors at pprune.org are always the first to start speculating about root causes, and complain how

Re: DNS pulling BGP routes?

On 10/8/21 07:25, Sabri Berisha wrote: Whenever there is an aviation incident, the keyboard warriors at pprune.org are always the first to start speculating about root causes, and complain how the air crew made mistakes. They, the keyboard warriors, of course know how best to fly an aircraft

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

On 10/7/21 20:46, Max Tulyev wrote: We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York (USA). I have seen both ports simultaneously dropped traffic volume for about one hour today. Our traffic across Telia dipped at 1600hrs UTC yesterday, and recovered 2hrs later. No

Re: BGP communities, was: Re: Facebook post-mortems... - Update!

There are also a bunch at http://bgp.community (linked to the source where possible instead of keeping a stale copy). On Tue, Oct 5, 2021, 1:17 PM Jay Hennigan wrote: > On 10/5/21 09:49, Warren Kumari wrote: > > > Can someone explain to me, preferably in baby words, why so many > > providers

Re: DNS pulling BGP routes?

On 10/7/21 08:26, Hank Nussbacher wrote: Better question is why do we not see any FB netadmins on NANOG? I'm not talking about October 2021 but rather over the past 3-5 years how many FB techies have posted here like we see people from Google, Cloudflare, Akamai, etc.? They are likely

Re: DNS pulling BGP routes?

On 06/10/2021 22:38, Jon Lewis wrote: But I just don't understand why this is a good idea at all. Network topology is not DNS's bailiwick so using it as a trigger to withdraw routes seems Everything I've seen posted about this (whether from Facebook directly, or others) is so vague as to

Re: DNS pulling BGP routes?

On 10/7/21 13:18, Jean St-Laurent wrote: Something public that we know now, is that it's possible to totally shut down facebook and restart it. Can we shutdown the full internet one day and see if it will restart properly without too much hack here and there? I think one thing that I

Re: DNS pulling BGP routes?

> > But, the reality is that it is impossible to correctly > recognize server is unavailable or to correctly withdraw > routes only when server is unavailable. > Not true at all. On Thu, Oct 7, 2021 at 9:50 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > William Herrin wrote: > >

Re: DNS pulling BGP routes?

Masataka Ohta writes: > William Herrin wrote: > This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. >>> >>> Yes, that is a commonly seen mistake with anycast. >> You don't know what you're talking about. > > I do but

Re: IRR for IX peers

Randy Bush wrote on 04/10/2021 21:15: i was hoping that, if 3130 said it is peering with martha, artemis would get a clue and stfu right. This was klunked around using the export-via and import-via rpsl constructions (draft-snijders-rpsl-via), which never quite made it to ietf wg adoption

Re: IRR for IX peers

>> i was hoping that, if 3130 said it is peering with martha, artemis >> would get a clue and stfu > > right. This was klunked around using the export-via and import-via > rpsl constructions (draft-snijders-rpsl-via), which never quite made > it to ietf wg adoption status. It did, however, point

Re: IRR for IX peers

Randy Bush wrote on 07/10/2021 15:26: it was sabatoged there was more to it than that. The grammar was too complicated to easily describe common policies and too limited to describe complex policies. The structure was difficult to extend when the routing became more complicated (e.g.

Re: DNS pulling BGP routes?

William Herrin wrote: This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. Yes, that is a commonly seen mistake with anycast. You don't know what you're talking about. I do but you don't. If your anycast node stops

Re: DNS pulling BGP routes?

On Wed, Oct 6, 2021 at 10:44 PM Masataka Ohta wrote: > Jared Mauch wrote: > > This is quite common to tie an underlying service announcement to BGP > > announcements in an Anycast or similar environment. > > Yes, that is a commonly seen mistake with anycast. You don't know what you're talking

RE: DNS pulling BGP routes?

Something public that we know now, is that it's possible to totally shut down facebook and restart it. Can we shutdown the full internet one day and see if it will restart properly without too much hack here and there? Jean -Original Message- From: NANOG On Behalf Of Mark Tinka

Re: IRR for IX peers

On 10/7/21 16:33, Nick Hilliard wrote: there was more to it than that.  The grammar was too complicated to easily describe common policies and too limited to describe complex policies.  The structure was difficult to extend when the routing became more complicated (e.g. mpls, route

Re: DNS pulling BGP routes?

Bjørn Mork wrote: This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. Yes, that is a commonly seen mistake with anycast. You don't know what you're talking about. I do but you don't.

Re: DNS pulling BGP routes?

On Thu, Oct 7, 2021 at 8:28 AM Masataka Ohta wrote: > My comment on the rfc is that it is simply wrong. > > See also: > > https://datatracker.ietf.org/doc/html/rfc3258 > While it would be > possible to have some process withdraw the route for a specific > server instance when it

RE: DNS pulling BGP routes?

Nice document. In section 2.5 Routing, this is written: Distributing Authoritative Name Servers via Shared Unicast Addresses... organizations implementing these practices should always provide at least one authoritative server which is not a participant in any shared unicast mesh. Could

Re: DNS pulling BGP routes?

On 10/7/21 18:21, William Herrin wrote: It wasn't forgotten. Folks gained a lot of experience with anycast DNS between 2002 and 2006. Not withdrawing the routes when the servers are deemed malfunctioning turned out not to be an operationally sound practice. The theory offered in 3258 was

Re: DNS pulling BGP routes?

On Thu, Oct 7, 2021 at 9:52 AM Masataka Ohta wrote: > But, this time, the reality strikes back. Not really. Or at all. Facebook the external service was down hard as soon as the cross-datacenter connections all failed. Whether or not the BGP routes for the external DNS were withdrawn had no

Re: DNS pulling BGP routes?

> On Oct 7, 2021, at 6:25 PM, Jean St-Laurent via NANOG wrote: > > Nice document. > > In section 2.5 Routing, this is written: > > Distributing Authoritative Name Servers via Shared Unicast Addresses... > > organizations implementing these practices should > always provide at least one

Re: DNS pulling BGP routes?

William Herrin wrote: It wasn't forgotten. Folks gained a lot of experience with anycast DNS between 2002 and 2006. Not withdrawing the routes when the servers are deemed malfunctioning turned out not to be an operationally sound practice. The theory offered in 3258 was wrong. So, from

RE: DNS pulling BGP routes?

Well said Bill. I agree with you about having all your tech/adm records + registrar on the same NS... especially for your OOB domain. Probably what killed them. They lost access to their fb-00b-net-mgmt.io cool dns name network. It just went from bad to worst when they realized that they

Re: DNS pulling BGP routes?

William Herrin wrote: Facebook's _internal_ DNS, while not anycasted, followed a similar logic: if the data center is isolated and their data goes stale, they stop serving potentially wrong answers. As I already wrote, that is a standard mechanism of DNS with SOA expiration period as is

Global issues @ Telia - doing a "FB/hold my beer" move?

Hi everyone, Looks like the season for outages is on. Does anyone has more details regarding the issues at Telia? I didn't found any public available information. They say it's over but this is clearly not the case. Best regards, Vincentz signature.asc Description: Message signed with

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York (USA). I have seen both ports simultaneously dropped traffic volume for about one hour today. It was not critical (for us), as traffic was shifted to another links, and there was no unreachable destinations like BGP