Re: DNS pulling BGP routes?
Baldur Norddahl wrote: Around here there are certain expectations if you sell a product called IP Transit and other expectations if you call the product paid peering. That some word is used for marketing hype with an intentional self-contradicting definition is not my problem, at all. The so-called "tier" of a company is a meaningless term. I have been using that term properly, which means it is meaningful if people use it properly. So is "peering". Sabri Berisha wrote: > The term "network neutrality" was invented by people who want to > control a network owned and paid for by someone else. Excellent theory. Masataka Ohta
Re: DNS pulling BGP routes?
On Mon, Oct 18, 2021 at 1:47 PM Matthew Petach wrote: > On Mon, Oct 18, 2021 at 1:17 PM William Herrin wrote: >> Since peering customers can only reach transit customers, it follows >> that one of the customers in the equation is a fully-paid transit >> customer. That fully paid customer's service is degraded or denied >> unless the peering customer also pays. Hence the conflict of interest. > > Customer A is full transit paying customer. > in case 2, Customer B is a paid peering customer. > > Can you explain what it is I'm missing here? ^_^; The part where customer A is paying for a connection to "the Internet" at some data rate, which includes the network run by customer B. REGARDLESS of whether B pays the same service provider. If the service rendered to A is changed by B's payment (or lack), that's a conflict of interest. To remove the conflict of interest, you either have to fiddle the definition of what customer A is buying, turning it into something that would not be obvious to an ordinary person or you have or you have to allow B to engage in settlement free peering if they want to. Or, counterintuitively, pay your own transit provider enough to handle any capacity A and B together care to consume. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
On 10/18/21 1:51 PM, Sabri Berisha wrote: I know that there are a lot of risks with hamfisted gubbermint regulations. But even when StarLink turns the sky into perpetual daylight and we get another provider, there are going to still be painfully few choices, and too often the response to $EVIL is not "oh great, more customers for us!" but "oh great, let's do that too!". That's the point where MBAs take over from engineering to squeeze every last penny out of the customer. And that usually happens when a company gets large. So what's the counter? I mean, MSO's already pull that kind of shitty behavior with their "fees" cloaked as taxes. Maybe a better argument is that this is all theoretical since to my knowledge it's not being done on any large scale, so let's not fix theoretical problems. This is obviously complicated and one of the complications is QoS in the last mile. DOCSIS has a lot of QoS machinery so that MSO's could get CBR like flows for voice back in the day. I'm not sure whether this ever got deployed because as is often the case, brute force and ignorance (ie, make the wire faster) wins, mooting the need. Is there even a constructive use of QoS in the last mile these days that isn't niche? Maybe gaming? Would any sizable set of customers buy it if it were offered? It's been a few years since I've worked for a residential service provider, but to the best of my memory, congestion was rarely found in the last mile. That's what I figured. I remember talking to some Sprint architect types around the same time when I told them all of their insistence on AAL2 was useless because voice was going to be drop in the bucket. They looked at me as if I was completely insane. Mike
Re: DNS pulling BGP routes?
- On Oct 18, 2021, at 12:40 PM, Michael Thomas m...@mtcc.com wrote: > On 10/18/21 12:22 PM, Sabri Berisha wrote: >> I totally agree. 100%. Now we just have to agree on the regulation that >> we're talking about. >> >> My idea of regulation in this context is to get rid of the monopoly/duopoly >> so that users actually do have a way out and can vote with their feet. From >> that perspective, the NBN model isn't that bad (not trying to start an NBN >> flamewar here). > I know that there are a lot of risks with hamfisted gubbermint > regulations. But even when StarLink turns the sky into perpetual > daylight and we get another provider, there are going to still be > painfully few choices, and too often the response to $EVIL is not "oh > great, more customers for us!" but "oh great, let's do that too!". That's the point where MBAs take over from engineering to squeeze every last penny out of the customer. And that usually happens when a company gets large. > Witness airlines and the race to the bottom with various fees -- and > that's in a field where there is plenty of competition. For the most part: yes. But, that's also where the success of Southwest comes from. They generally don't take part in that kind of bovine manure. > This is obviously complicated and one of the complications is QoS in the > last mile. DOCSIS has a lot of QoS machinery so that MSO's could get CBR > like flows for voice back in the day. I'm not sure whether this ever got > deployed because as is often the case, brute force and ignorance (ie, > make the wire faster) wins, mooting the need. Is there even a > constructive use of QoS in the last mile these days that isn't niche? > Maybe gaming? Would any sizable set of customers buy it if it were offered? It's been a few years since I've worked for a residential service provider, but to the best of my memory, congestion was rarely found in the last mile. > If there isn't, a regulation that just says "don't cut deals to > prioritize one traffic source at the expense of others" seems pretty > reasonable, and probably reflects the status quo anyway. But again, now you are interfering in how I operate my network. Let's say I have two options: 1. Accept one million from Netflix to prioritize their traffic and set my residential internet pricing to $50; or 2. Be subjected to government regulations that prohibit me from accepting said funds and set my residential internet pricing to $100 to cover costs; Isn't it up to me to make that decision? The government should not need to have any say in this matter. And note my careful wording, because in the current market, they do need to have a say. My point is: the market should be open enough that if a sub disagrees with their ISP's technical choices, they should be able to switch. It's government regulation that makes that extremely difficult, if not impossible. But, I don't want to pollute the list any further and I've made my points so I shall grant you the last word publically :) Thanks, Sabri
Re: DNS pulling BGP routes?
On Mon, Oct 18, 2021 at 1:17 PM William Herrin wrote: > On Mon, Oct 18, 2021 at 11:47 AM Matthew Petach > wrote: > > On Mon, Oct 18, 2021 at 11:16 AM William Herrin wrote: > >> On Mon, Oct 18, 2021 at 10:30 AM Baldur Norddahl > >> wrote: > >> > Around here there are certain expectations if you sell a product > called IP Transit and other expectations if you call the product paid > peering. The latter is not providing the whole internet and is cheaper. > >> > >> The problem with paid peering is that it creates a conflict of > >> interest which corruptly influences the company's behavior. Two > >> customers are paying you in full for a service but if one elects not > >> to pay you will also deny or degrade the service to the other one who > >> has, in fact, paid you. > > > > > > The phrase "paying you in full" is the stumbling point with your > > claim. > > > > As Baldur noted, "paid peer [...] is not providing the whole > > internet and is cheaper." > > Since peering customers can only reach transit customers, it follows > that one of the customers in the equation is a fully-paid transit > customer. That fully paid customer's service is degraded or denied > unless the peering customer also pays. Hence the conflict of interest. > I'm sorry. :( I'm feeling particularly dense this morning, so I'm going to work through the two cases very slowly to make sure I understand. Customer A is full transit paying customer. In case 1, Customer B is a full transit paying customer also. Customer A announces their prefixes to ISP; as a transit customer, ISP promises to announce those prefixes to everyone they have a BGP relationship with, including customer B. Likewise, ISP provides a full BGP table, including default if requested, to Customer A, ensuring Customer A can reach Customer B, and Customer B can reach Customer A. in case 2, Customer B is a paid peering customer. Customer A announces their prefixes to ISP; as a transit customer, the ISP promises to announce those prefixes to everyone they have a BGP relationship with, including Customer B. Likewise, ISP provides a full BGP table, including default if requested, to Customer A, ensuring Customer A can reach Customer B, and Customer B can reach Customer A. I'm not seeing how Customer B's status as paid peer versus transit customer changes either the set of prefixes Customer A sees, or the spread of Customer A's prefixes to the rest of the Internet. In short--the amount Customer B is paying or not paying, does not change the view of prefixes that Customer A sees, nor does it change the propagation scope of Customer A's prefixes. As neither of those two things change, I'm completely failing to see how Customer A's service is being degraded or denied based on Customer B's choices. Can you explain what it is I'm missing here? ^_^; Regards, > Bill Herrin > Thanks! Matt
Re: DNS pulling BGP routes?
On Mon, Oct 18, 2021 at 11:47 AM Matthew Petach wrote: > On Mon, Oct 18, 2021 at 11:16 AM William Herrin wrote: >> On Mon, Oct 18, 2021 at 10:30 AM Baldur Norddahl >> wrote: >> > Around here there are certain expectations if you sell a product called IP >> > Transit and other expectations if you call the product paid peering. The >> > latter is not providing the whole internet and is cheaper. >> >> The problem with paid peering is that it creates a conflict of >> interest which corruptly influences the company's behavior. Two >> customers are paying you in full for a service but if one elects not >> to pay you will also deny or degrade the service to the other one who >> has, in fact, paid you. > > > The phrase "paying you in full" is the stumbling point with your > claim. > > As Baldur noted, "paid peer [...] is not providing the whole > internet and is cheaper." Since peering customers can only reach transit customers, it follows that one of the customers in the equation is a fully-paid transit customer. That fully paid customer's service is degraded or denied unless the peering customer also pays. Hence the conflict of interest. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
On 10/18/21 12:22 PM, Sabri Berisha wrote: - On Oct 18, 2021, at 11:51 AM, Michael Thomas m...@mtcc.com wrote: Hi, On 10/18/21 11:09 AM, Sabri Berisha wrote: The term "network neutrality" was invented by people who want to control a network owned and paid for by someone else. Your version of "unreasonable" and my version of "unreasonable" are on the opposite end of the spectrum. I think it is unreasonable for you to tell me how to run configure my routers, and you think it is unreasonable for me to configure my routers that I pay for the way that I want to. Yeahbut, for the last mile that network is often a monopoly or maybe a duopoly if you're lucky. If streaming provider 1 pays ISP to give priority over streaming provider 2 -- maybe by severely rate limiting provider 2 -- the people who get screwed are end users without a way to vote with their feet. That sort of monopolistic behavior is bad for end users. Mostly I want ISP's to be dumb bit providers and stay out of shady deals that enrich ISP's at my expense. And if it takes regulation to do that, bring it. I totally agree. 100%. Now we just have to agree on the regulation that we're talking about. My idea of regulation in this context is to get rid of the monopoly/duopoly so that users actually do have a way out and can vote with their feet. From that perspective, the NBN model isn't that bad (not trying to start an NBN flamewar here). But, I would be opposed to regulation that prevents a network operator from going into enable mode. There are more reasons than "government intervention into a privately owned network" / "network neutrality" to want more competition. Lower prices and better service, for example. Have you ever tried calling Comcast/Spectrum? I'd love to get involved (privately, not professionally) in a municipal broadband project where I live. We have 1 fiber duct for the entire town. That got cut last year, and literally everyone was without internet access for many hours. We don't need net neutrality. We need competition. The FCC sucks, and so does the CPUC. I know that there are a lot of risks with hamfisted gubbermint regulations. But even when StarLink turns the sky into perpetual daylight and we get another provider, there are going to still be painfully few choices, and too often the response to $EVIL is not "oh great, more customers for us!" but "oh great, let's do that too!". Witness airlines and the race to the bottom with various fees -- and that's in a field where there is plenty of competition. This is obviously complicated and one of the complications is QoS in the last mile. DOCSIS has a lot of QoS machinery so that MSO's could get CBR like flows for voice back in the day. I'm not sure whether this ever got deployed because as is often the case, brute force and ignorance (ie, make the wire faster) wins, mooting the need. Is there even a constructive use of QoS in the last mile these days that isn't niche? Maybe gaming? Would any sizable set of customers buy it if it were offered? If there isn't, a regulation that just says "don't cut deals to prioritize one traffic source at the expense of others" seems pretty reasonable, and probably reflects the status quo anyway. Mike
Re: DNS pulling BGP routes?
- On Oct 18, 2021, at 11:51 AM, Michael Thomas m...@mtcc.com wrote: Hi, > On 10/18/21 11:09 AM, Sabri Berisha wrote: >> >> The term "network neutrality" was invented by people who want to control >> a network owned and paid for by someone else. >> >> Your version of "unreasonable" and my version of "unreasonable" are on the >> opposite end of the spectrum. I think it is unreasonable for you to tell me >> how to run configure my routers, and you think it is unreasonable for me >> to configure my routers that I pay for the way that I want to. > > Yeahbut, for the last mile that network is often a monopoly or maybe a > duopoly if you're lucky. If streaming provider 1 pays ISP to give > priority over streaming provider 2 -- maybe by severely rate limiting > provider 2 -- the people who get screwed are end users without a way to > vote with their feet. That sort of monopolistic behavior is bad for end > users. Mostly I want ISP's to be dumb bit providers and stay out of > shady deals that enrich ISP's at my expense. And if it takes regulation > to do that, bring it. I totally agree. 100%. Now we just have to agree on the regulation that we're talking about. My idea of regulation in this context is to get rid of the monopoly/duopoly so that users actually do have a way out and can vote with their feet. From that perspective, the NBN model isn't that bad (not trying to start an NBN flamewar here). But, I would be opposed to regulation that prevents a network operator from going into enable mode. There are more reasons than "government intervention into a privately owned network" / "network neutrality" to want more competition. Lower prices and better service, for example. Have you ever tried calling Comcast/Spectrum? I'd love to get involved (privately, not professionally) in a municipal broadband project where I live. We have 1 fiber duct for the entire town. That got cut last year, and literally everyone was without internet access for many hours. We don't need net neutrality. We need competition. The FCC sucks, and so does the CPUC. Thanks, Sabri
Re: DNS pulling BGP routes?
" to give priority" Assuming priority is given. It's going to be very rare for their to be both only one ISP and no other ISPs able to be motivated to be present. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Michael Thomas" To: nanog@nanog.org Sent: Monday, October 18, 2021 1:51:50 PM Subject: Re: DNS pulling BGP routes? On 10/18/21 11:09 AM, Sabri Berisha wrote: > > The term "network neutrality" was invented by people who want to control > a network owned and paid for by someone else. > > Your version of "unreasonable" and my version of "unreasonable" are on the > opposite end of the spectrum. I think it is unreasonable for you to tell me > how to run configure my routers, and you think it is unreasonable for me > to configure my routers that I pay for the way that I want to. Yeahbut, for the last mile that network is often a monopoly or maybe a duopoly if you're lucky. If streaming provider 1 pays ISP to give priority over streaming provider 2 -- maybe by severely rate limiting provider 2 -- the people who get screwed are end users without a way to vote with their feet. That sort of monopolistic behavior is bad for end users. Mostly I want ISP's to be dumb bit providers and stay out of shady deals that enrich ISP's at my expense. And if it takes regulation to do that, bring it. Mike
Re: DNS pulling BGP routes?
On 10/18/21 11:09 AM, Sabri Berisha wrote: The term "network neutrality" was invented by people who want to control a network owned and paid for by someone else. Your version of "unreasonable" and my version of "unreasonable" are on the opposite end of the spectrum. I think it is unreasonable for you to tell me how to run configure my routers, and you think it is unreasonable for me to configure my routers that I pay for the way that I want to. Yeahbut, for the last mile that network is often a monopoly or maybe a duopoly if you're lucky. If streaming provider 1 pays ISP to give priority over streaming provider 2 -- maybe by severely rate limiting provider 2 -- the people who get screwed are end users without a way to vote with their feet. That sort of monopolistic behavior is bad for end users. Mostly I want ISP's to be dumb bit providers and stay out of shady deals that enrich ISP's at my expense. And if it takes regulation to do that, bring it. Mike
Re: DNS pulling BGP routes?
On Mon, Oct 18, 2021 at 11:16 AM William Herrin wrote: > On Mon, Oct 18, 2021 at 10:30 AM Baldur Norddahl > wrote: > > Around here there are certain expectations if you sell a product called > IP Transit and other expectations if you call the product paid peering. The > latter is not providing the whole internet and is cheaper. > > The problem with paid peering is that it creates a conflict of > interest which corruptly influences the company's behavior. Two > customers are paying you in full for a service but if one elects not > to pay you will also deny or degrade the service to the other one who > has, in fact, paid you. > The phrase "paying you in full" is the stumbling point with your claim. As Baldur noted, "paid peer [...] is not providing the whole internet and is cheaper." If the two customers are "paying you in full", then they're paying you for transit, and as such, they get a copy of the full tables, regardless of how you learn those routes, whether through a paid relationship or a settlement free relationship. If the two customers are *not* paying full price, but are instead paying the reduced price for "paid peering", then they each recognize that the set of prefixes they are receiving, and the spread of their prefixes in return are inherently limited, *and will change over time as the customer relationships on each side change." Nobody buying "paid peering" expects the list of prefixes sent and received across those sessions to remain constant forever. That would imply no new customers are ever added, and would imply no customers ever leave, which is clearly unreasonable in the real world. If you, as the customer paying for paid peering, see the list of prefixes decreasing over time, when the contract comes up for renewal, you are likely to argue for a lower price, or may decide it's no longer worth it, and decide to not renew the relationship. On the other hand, if you, as the provider, are increasing the number of prefixes being seen across those paid peerings at a substantial rate, when the next renewal cycle comes up, you may decide the price for paid peering should go up, because you're providing more value across those sessions. Each side evaluates the then-present set of prefixes being exchanged when the contract comes up for renewal, to decide if it's still worth it or not. But if you're "paying in full" for IP transit, then the sessions should include as much of the full BGP table as possible, potentially including a default route, and the promise of that session is to make your prefixes as visible to the entire rest of the Internet as possible. (This is, as a small aside, why I don't think Cogent should be allowed to label their product "IP transit" so long as they are willfully refusing to propagate their customer's prefixes to *all* of the rest of the Internet. So long as they are choosing to cherry-pick out certain networks that they will *not* propagate their customers routes to, they are *not* providing true IP transit, and should not label it as such.) > > Regards, > Bill Herrin > Thanks! Matt
Re: DNS pulling BGP routes?
On Sun, Oct 17, 2021 at 4:54 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Matthew Petach wrote: > > > I'd like to take a moment to point out the other problem with this > > sentence, which is "antitrust agencies". > > > > One of the key aspects to both CDN providers and transit > > providers is they tend to be multi-national organizations with > > infrastructure in multiple countries on multiple continents. > > Your theory that multi-national entities can not be > targets of anti-trust agencies of individual countries > and can enjoy world wide oligopoly is totally against > the reality. > *facepalm* No, the point I was making wasn't that they can't be the target of antitrust agencies, the point was that there's so many conflicting jurisdictions that consistent enforcement in a coordinated fashion is impossible. We can't even get countries to agree on what a copyright or a trademark means, or even what privacy rights a person should have. I know one content distribution company that was originally thinking of putting a site in country X; however, after taking a closer look at the laws in country X, decided instead to put the site in a nearby country with more favourable laws and to interconnect with the network providers just outside country X, thus putting them outside the reach of those laws. It's really, *really* hard to "regulate" global infrastructure because it crosses over/under/through so many different jurisdictions; if one country decides to put considerably stronger restrictions in place, the reaction by and large is to 'route around the damage' so to speak. The lack of success from Brasil's efforts are a good indication of just how successful per-country regulation of internet providers tends to be: https://www.networkworld.com/article/2175352/brazil-to-drop-requirement-that-internet-firms-store-data-locally.html The GDPR is probably the most successful effort at reining in global internet companies in recent years, and even there, when companies ignore it, the resulting fines are a small slap on the wrist at best, hardly causing them to change their behaviours: https://secureprivacy.ai/blog/gdpr-the-6-biggest-fines-enforced-by-regulators-so-far Even the $5 billion fine Facebook paid to the FTC after the Cambridge Analytica was really only a $106M fine, with an extra $4.9B thrown in to make the personal lawsuit go away: https://www.politico.com/news/2021/09/21/facebook-paid-billions-extra-to-the-ftc-to-spare-zuckerberg-in-data-suit-shareholders-allege-513456 When companies can afford to throw an extra 50x the money at a regulatory agency to make a problem go away, it's pretty clear that thinking that regulatory agencies are going to have enough teeth to fundamentally change the way of life of those businesses is optimistic at best. Looking at the top 15 antitrust cases in the US, you can see how in many cases, the antitrust action was minimally effective in the long term, as the companies that were split up often ended up rejoining again, years down the line: https://stacker.com/stories/3604/15-companies-us-government-tried-break-monopolies > > Masataka Ohta > Matt
Re: DNS pulling BGP routes?
On Mon, Oct 18, 2021 at 10:30 AM Baldur Norddahl wrote: > Around here there are certain expectations if you sell a product called IP > Transit and other expectations if you call the product paid peering. The > latter is not providing the whole internet and is cheaper. The problem with paid peering is that it creates a conflict of interest which corruptly influences the company's behavior. Two customers are paying you in full for a service but if one elects not to pay you will also deny or degrade the service to the other one who has, in fact, paid you. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
- On Oct 18, 2021, at 1:40 AM, Masataka Ohta mo...@necom830.hpcl.titech.ac.jp wrote: > Sabri Berisha wrote: > >> Therefore, anti-trust intervention is only considered in markets >> where there are a relatively small amount of competitors and this >> lack of competition harms the consumer, or when one or more dominant >> parties use their position to force smaller companies into >> unreasonable compliance with their wishes. > > Didn't network neutrality become an issue because "one or more > dominant parties use their position to force smaller companies > into unreasonable compliance with their wishes"? The term "network neutrality" was invented by people who want to control a network owned and paid for by someone else. Your version of "unreasonable" and my version of "unreasonable" are on the opposite end of the spectrum. I think it is unreasonable for you to tell me how to run configure my routers, and you think it is unreasonable for me to configure my routers that I pay for the way that I want to. Net neutrality is just a fancy word for "I don't like the fifth"*. >> The CDN market has multiple competitors, and the barrier to entry the >> market is relatively low as you don't have any last-mile issues or >> difficult-to-get government license requirements. > > To enter the market competitively, you must have large number > of servers at many locations, I think. Hence the "relatively low". It is far easier to start a CDN than it is to start a residential internet service. At least here in the U.S. Thanks, Sabri * The fifth, besides the right to remain silent, also contains the takings clause.
Re: DNS pulling BGP routes?
On Mon, 18 Oct 2021 at 09:51, Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > But, with settlement free peering between tier 1 ISPs, tier 2 > ISPs having transit/paid peering with a tier 1 ISP will receive > routes from peers of the tier 1 ISP. There is transit traffic > exchanged between tier 1 ISPs over settlement free peering. > > So, I don't think distinguishing transit from peering > meaningful for precise discussions. > Around here there are certain expectations if you sell a product called IP Transit and other expectations if you call the product paid peering. The latter is not providing the whole internet and is cheaper. The so-called "tier" of a company is a meaningless term. Traffic will never traverse two settlement free peering links and this is true for "tier 1" ISPs as well. Paid peering is understood to be the same as a settlement free peering except for not being settlement free. Therefore a paid peering with an "tier 1" ISP will not provide any traffic that traverses their settlement free peering links with other "tier 1" ISPs. It is quite possible some "tier 1" ISPs do not see the point in providing such a product but then they just won't offer paid peering - only IP transit. In more technical terms, no peering link, settlement free or for pay, has routes for the whole internet. If the peering had routes for the whole internet it would be IP transit. This is achieved by only announcing own customer routes on the peering links and _not_ announcing routes received from other peering links. You get access to their customers but you need to make other arrangements to get access to the rest of the internet. > > > For smaller ISPs it works the other way around. An evil CDN could > > attempt to charge us, the small ISP. I am happy that is not > > happening. > > Because of natural monopoly and PON, most access/retail ISPs > enjoy their domination in their own area regardless of their > sizes. > This is not true in our part of the world. The regulator is requiring all major last mile infrastructure owners to give access to reseller ISPs breaking that monopoly. My own company both owns infrastructure (FTTH and FTTB / apartment networks) and resell using FTTH / DSL owned by other companies. Plus we have three 5G networks providing an alternative and also breaking the monopoly. Regards, Baldur
Re: DNS pulling BGP routes?
Mark Tinka wrote: Yes, but nobody cares about Layer 1 or Layer 2. As you wrote: You can't tell me that US$700 million being spent to build a > submarine cable around a continent is something to scoff at. you do care. Look, I'm not saying the ITU are bad FYI, I'm not arguing especially for ITU. But, it do have some regulatory influence for its Members. FYI, IS-IS is part of OSI, which was jointly developed by ISO and ITU, not by IETF at all. You might be forgetting that the IETF adapted IS-IS to IP networks: Just as RIP was imported from XNS world, which does not deny Xerox and ITU/ISO primarily contributed to develop the protocols. I have zero interest in being the profit police. Who am I tell anyone that they are earning too much? Anti-trust agencies, of course. Access networks are subject to regional monopoly unless unbundling is forced by regulatory bodies. Worse, with PON, such unbundling is hard (not impossible, see https://ieeexplore.ieee.org/document/5616389). Submarine cables are usually either owned by one party, or a small club. Submarine cables are for backbone. That's why you must distinguish access and backbone. Masataka Ohta
Re: DNS pulling BGP routes?
> > Otherwise, CDN providers with their own backbone are free riders > ignoring access costs. > I think the Pointy Hairs and Bean Counters would love it if they could ignore all the monthly bills for the access costs that we generate. On Sat, Oct 16, 2021 at 9:46 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Mark Tinka wrote: > > >> Remember that CDN providers are not neutral at all. > > > > Well, the purpose of a network is whatever its proprietor deems it to > > be, and makes no false advertising about it. > > What? > > > A private enterprise network that carries a company's internal traffic - > > which may or may not interface with an external network that is > > interested in some or all of that traffic - would, in your eyes, be > > classified as not neutral, because it chooses not to use its network to > > provide global IP Transit? > Unless they directly reach their end users, yes, of course. > > The fundamental problem of networking is the last mile problem > that access costs alot more than backbone. > > As such, long distance carriers may peer with access providers > only when they are neutral or pay some of there revenue share > to access providers. > > > In my mind, the word "transit" refers to carriage between two > > non-homogeneous points. So network A (customer) will talk to network C > > (content) via my network B (transit). If the traffic originates either > > from A or C, BUT terminates/ends inside of B, I do not consider that > > transit. > > With your definition, as CDN providers with their own backbone are > not "transit", they can not request access providers (and, ultimately, > end users) peering without paying some as compensation for access > network cost. > > Otherwise, CDN providers with their own backbone are free riders > ignoring access costs. > > Masataka Ohta >
Re: DNS pulling BGP routes?
On 10/18/21 14:16, Masataka Ohta wrote: As copper and optical fiber for access politically belongs to ITU, DSL and optical fiber standards of ITU are followed by the IETF world. Yes, but nobody cares about Layer 1 or Layer 2. Once the road is built, all anyone remembers is the car I drove across it, not whether the tar used to build the road was mixed well :-). I actually joined an ITU meeting at Geneva, when I was actively acting for DSL in Japan. Good for you. Look, I'm not saying the ITU are bad - I am saying that they are "more structured and rigid", than Internet-land. And that is okay. There is a reason we TCP/IP became dominant. FYI, IS-IS is part of OSI, which was jointly developed by ISO and ITU, not by IETF at all. You might be forgetting that the IETF adapted IS-IS to IP networks: https://datatracker.ietf.org/doc/html/rfc1195 I'm not sure anyone running IS-IS in an ISP environment, today, is running it for CLNS. But we thank the ISO, immensely :-). Are you agreeing with me that they are earning a lot more than they should? I have zero interest in being the profit police. Who am I tell anyone that they are earning too much? If you make something people find value in, the billions will automatically flow your way - you can't stop it. Is it a perfect system, probably not, but it's what we've got. Access networks are subject to regional monopoly unless unbundling is forced by regulatory bodies. Worse, with PON, such unbundling is hard (not impossible, see https://ieeexplore.ieee.org/document/5616389). Submarine cables are usually either owned by one party, or a small club. It's no different - and trying to be a member of the club can be just as demoralizing as local regulation on terrestrial builds. That said, different markets have different policies on access networks. So a single policy for what we think is best is not practical. Moreover, if access networks are expensive due to backward regulation and monopolistic promotion, then that is an artificial problem that can be removed, but the actors choose not to. You can't blame a content operator for that market position. So, you are a neo-liberalist. Good luck. I also like the one where whole gubbermints shutdown the Internet for elections, or to hush voices. I discriminate equally :-). Though precise definition of "tier 1" is a rat hole, that there are entities called tier 1, which are the primary elements of the Internet backbone, is a common concept shared by most of us, maybe excluding you. I know many here that have moved on from the "tier" terminologies. But it's unnecessary for them to chime in. There hasn't been "a core of the Internet" for a long while, and anyone still believing that either in reality or words is living in a fantasy world long gone, which is partially why infrastructure finds itself becoming less and less relevant, and being swallowed up by BigContent. I mean, if you missed the fact that Facebook went down, and people thought the Internet had stopped, then maybe Facebook are a Tier 1... Mark.
Re: DNS pulling BGP routes?
Mark Tinka wrote: As you are seemingly requesting international legal formality, let me point out there are "International Telecommunication Regulations", based on which network neutrality is discussed by ITU. And since when does the IETF world follow the ITU standards? As copper and optical fiber for access politically belongs to ITU, DSL and optical fiber standards of ITU are followed by the IETF world. I actually joined an ITU meeting at Geneva, when I was actively acting for DSL in Japan. Even though ITU heads don't think much of IETF heads, you can't find an SDH or DWDM port in a laptop. On the other hand, GMPLS is based on OSPF, IS-IS and RSVP-TE :-). FYI, IS-IS is part of OSI, which was jointly developed by ISO and ITU, not by IETF at all. Well, I'll be asking my bank to sell me some IP Transit or DIA, then, since they are running an IP network. Feel free to do so. It may be, it may not be. The reason is only one or a small handful of folk are investing US$700 million into a submarine cable. Are you agreeing with me that they are earning a lot more than they should? On the other hand, access networks are built by several operators, all competing. Access networks are subject to regional monopoly unless unbundling is forced by regulatory bodies. Worse, with PON, such unbundling is hard (not impossible, see https://ieeexplore.ieee.org/document/5616389). Willing buyer, willing seller. That's all that's needed. If the seller doesn't like the buyer, they move on. If the buyer doesn't like the seller, they move on. So, you are a neo-liberalist. Good luck. Are you saying that there is no such thing as tier 1 ISPs? Hehe, let's not go down that rat hole. > > But no, I don't believe in "tiers" for service provider networks. > Haven't done so in nearly 15 years. Though precise definition of "tier 1" is a rat hole, that there are entities called tier 1, which are the primary elements of the Internet backbone, is a common concept shared by most of us, maybe excluding you. Masataka Ohta
Re: DNS pulling BGP routes?
On 10/18/21 10:11, Masataka Ohta wrote: As you are seemingly requesting international legal formality, let me point out there are "International Telecommunication Regulations", based on which network neutrality is discussed by ITU. And since when does the IETF world follow the ITU standards? Even though ITU heads don't think much of IETF heads, you can't find an SDH or DWDM port in a laptop. On the other hand, GMPLS is based on OSPF, IS-IS and RSVP-TE :-). No, of course. So? Well, I'll be asking my bank to sell me some IP Transit or DIA, then, since they are running an IP network. That cost is negligible compared to the cost to prepare access network all over the continent, I'm afraid. It may be, it may not be. The reason is only one or a small handful of folk are investing US$700 million into a submarine cable. On the other hand, access networks are built by several operators, all competing. So no single operator building an access network is spending more than the content folk laying pipe in the Atlantic and Indian oceans. The essential difference is whether they are neutral or not. Well, the issue is you want to label things. I can see this is what is causing your confusion. Willing buyer, willing seller. That's all that's needed. If the seller doesn't like the buyer, they move on. If the buyer doesn't like the seller, they move on. Are you saying that there is no such thing as tier 1 ISPs? Hehe, let's not go down that rat hole. But no, I don't believe in "tiers" for service provider networks. Haven't done so in nearly 15 years. Heck, my marketing team are always asking if we can identify ourselves as "Tier 1" because we own and operate a submarine cable. I'm sure you can guess my answer to them... Personally, I don't care whether you are "transit-free" or not. You cannot provide an Internet service to the entire world from just one operator (network or content). So trying to be "bigger" than the other guy is a pointless exercise. Jane + Thatho don't care about your measuring contest. Mark.
Re: DNS pulling BGP routes?
Sabri Berisha wrote: Therefore, anti-trust intervention is only considered in markets where there are a relatively small amount of competitors and this lack of competition harms the consumer, or when one or more dominant parties use their position to force smaller companies into unreasonable compliance with their wishes. Didn't network neutrality become an issue because "one or more dominant parties use their position to force smaller companies into unreasonable compliance with their wishes"? The CDN market has multiple competitors, and the barrier to entry the market is relatively low as you don't have any last-mile issues or difficult-to-get government license requirements. To enter the market competitively, you must have large number of servers at many locations, I think. Masataka Ohta
Re: DNS pulling BGP routes?
Mark Tinka wrote: What? I will use my network for what I want my network to do for me. There are no international rules about why a network must be built. As you are seemingly requesting international legal formality, let me point out there are "International Telecommunication Regulations", based on which network neutrality is discussed by ITU. Unless they directly reach their end users, yes, of course. So by your logic, a bank's internal network used to drive its ATM machines is not neutral because one cannot use that network for global IP Transit? No, of course. So? You can't tell me that US$700 million being spent to build a submarine cable around a continent is something to scoff at. That cost is negligible compared to the cost to prepare access network all over the continent, I'm afraid. > Okay, so by your logic, "access providers" should pay CDN's for > peering, because the CDN's have spent millions building submarine > cables and data centres around the world to bring their service to > the access providers. After all, why give the access providers a free > ride either? The essential difference is whether they are neutral or not. > In case it's not clear, that last paragraph was sarcastic. It's 2021 > - long distance, access, backbone, metro, e.t.c. Those are boxes that > don't exist anymore. Are you saying that there is no such thing as tier 1 ISPs? Masataka Ohta
Re: DNS pulling BGP routes?
Baldur Norddahl wrote: Neutral backbone providers don't peer with access/retail ISPs. They sell transit to them. FYI, that is called paid peering. > Paid peering is not the same product as IP Transit. In general a > packet never traverse two peering links because that would mean the > middle man is not getting paid to move the traffic. So, there is terminology confusion because, these days, many people distinguish transit from peering without precise understanding on the peering situations. Paid peering with a backbone provider will get you routes from their paying customers but not from their peers. That argument may be applicable to the simplest cases of, so called, peering between leaf ISPs and transit peering (here, "transit peering" seems to be a proper terminology accepted by most) between leaf ISPs and upper level ISPs. But, with settlement free peering between tier 1 ISPs, tier 2 ISPs having transit/paid peering with a tier 1 ISP will receive routes from peers of the tier 1 ISP. There is transit traffic exchanged between tier 1 ISPs over settlement free peering. So, I don't think distinguishing transit from peering meaningful for precise discussions. I do not want Netflix to pay me. You are so generous. Let me tell you the point. Large ISP can exploit their domination of the marked to double dip, which means they want to be paid twice. That happens to be not neutral and is a way to make the customer pay a hidden fee. For smaller ISPs it works the other way around. An evil CDN could attempt to charge us, the small ISP. I am happy that is not happening. Because of natural monopoly and PON, most access/retail ISPs enjoy their domination in their own area regardless of their sizes. Masataka Ohta
Re: DNS pulling BGP routes?
On 10/16/21 15:44, Masataka Ohta wrote: What? I will use my network for what I want my network to do for me. There are no international rules about why a network must be built. Provided that I am clear to those whom I want to connect to my network, I can do what I want with it and not be a bad actor. Unless they directly reach their end users, yes, of course. So by your logic, a bank's internal network used to drive its ATM machines is not neutral because one cannot use that network for global IP Transit? The fundamental problem of networking is the last mile problem that access costs alot more than backbone. Well, yes and no. While it is true that one of the biggest problems of the Internet is the last mile, it is vital to not be forced into the mistake of classification. For some operators, the "last mile" is the biggest cost. For other networks, the "backbone" is the biggest cost. You can't tell me that US$700 million being spent to build a submarine cable around a continent is something to scoff at. For me, I don't want to hold myself back by classifying "access", "backbone", "metro", e.t.c. Your business model will determine what is costly to you, and what isn't. As such, long distance carriers may peer with access providers only when they are neutral or pay some of there revenue share to access providers. Again, you are trying to keep the old Internet (and the classic telephone company model) alive in 2021. That is not how operators work anymore. There are networks that have neither a "backbone" nor an "access network" that do very well, and don't cause anyone else pain, because they are clear in what their model is. With your definition, as CDN providers with their own backbone are not "transit", they can not request access providers (and, ultimately, end users) peering without paying some as compensation for access network cost. Otherwise, CDN providers with their own backbone are free riders ignoring access costs. Okay, so by your logic, "access providers" should pay CDN's for peering, because the CDN's have spent millions building submarine cables and data centres around the world to bring their service to the access providers. After all, why give the access providers a free ride either? In case it's not clear, that last paragraph was sarcastic. It's 2021 - long distance, access, backbone, metro, e.t.c. Those are boxes that don't exist anymore. Let's not refuse the advancement of the model because we can't find a way to make it fit in our old box. Mark.
Re: DNS pulling BGP routes?
søn. 17. okt. 2021 11.16 skrev Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp>: > Jay Hennigan wrote: > > >> Access/retail ISPs have no problem by peering with neutral > >> backbone providers. > > > > Neutral backbone providers don't peer with access/retail ISPs. They > > sell transit to them. > > FYI, that is called paid peering. > Paid peering is not the same product as IP Transit. In general a packet never traverse two peering links because that would mean the middle man is not getting paid to move the traffic. Paid peering with a backbone provider will get you routes from their paying customers but not from their peers. The same as you would have from a settlement free peering. > >> CDN provided backbone only reduces costs of other backbone > >> providers without reducing costs of access/retail ISPs. > > > > Access/retail ISPs that peer with CDNs eliminate the cost of paying > > for transit for the content delivered by the CDN. That's what the > > initials CDN stand for. > > But, it does not mean both parties of the peer are equally > benefited. As such peering may be paid one, though it > may not be the current practice. > > Given the observed profitability of CDN providers, CDN > providers are, seemingly, more benefited (because they > are not neutral), in which case, CDN providers should > pay to access/retail ISPs. > > Masataka Ohta > I do not want Netflix to pay me. I get paid by my customers, some of which also happens to be Netflix customers. If Netflix had to pay me, they would need to get that money from the same people who are already paying me directly. What is the point of that? Let me tell you the point. Large ISP can exploit their domination of the marked to double dip, which means they want to be paid twice. That happens to be not neutral and is a way to make the customer pay a hidden fee. For smaller ISPs it works the other way around. An evil CDN could attempt to charge us, the small ISP. I am happy that is not happening. Regards Baldur
Re: DNS pulling BGP routes?
- On Oct 17, 2021, at 4:50 AM, Masataka Ohta mo...@necom830.hpcl.titech.ac.jp wrote: Hi, > Matthew Petach wrote: >> One of the key aspects to both CDN providers and transit >> providers is they tend to be multi-national organizations with >> infrastructure in multiple countries on multiple continents. > > Your theory that multi-national entities can not be > targets of anti-trust agencies of individual countries > and can enjoy world wide oligopoly is totally against > the reality. At face value, your statement is correct. In context, it is unrealistic. Government anti-trust intervention is nothing less than the (a) government interfering in private business. In most civilized countries, that requires a strong legal basis as the government is essentially infringing on private property which is protected in most Constitutions. Therefore, anti-trust intervention is only considered in markets where there are a relatively small amount of competitors and this lack of competition harms the consumer, or when one or more dominant parties use their position to force smaller companies into unreasonable compliance with their wishes. The CDN market has multiple competitors, and the barrier to entry the market is relatively low as you don't have any last-mile issues or difficult-to-get government license requirements. And let's not even begin to talk about anti-trust for content providers; on just my Roku I have Netflix, Disney+, Hulu, Amazon Prime, Discovery+, FandangoNow (although they moved into something else I think), NatGeo+, Sling TV, Nickelodeon, and a bunch more that I can't even remember. Plenty of competition there. Thanks, Sabri
Re: DNS pulling BGP routes?
* mo...@necom830.hpcl.titech.ac.jp (Masataka Ohta) [Sun 17 Oct 2021, 11:17 CEST]: Jay Hennigan wrote: Neutral backbone providers don't peer with access/retail ISPs. They sell transit to them. FYI, that is called paid peering. Can you please please please stop posting nonsense? -- Niels.
Re: DNS pulling BGP routes?
Matthew Petach wrote: I'd like to take a moment to point out the other problem with this sentence, which is "antitrust agencies". One of the key aspects to both CDN providers and transit providers is they tend to be multi-national organizations with infrastructure in multiple countries on multiple continents. Your theory that multi-national entities can not be targets of anti-trust agencies of individual countries and can enjoy world wide oligopoly is totally against the reality. Masataka Ohta
Re: DNS pulling BGP routes?
Jay Hennigan wrote: Access/retail ISPs have no problem by peering with neutral backbone providers. Neutral backbone providers don't peer with access/retail ISPs. They sell transit to them. FYI, that is called paid peering. CDN provided backbone only reduces costs of other backbone providers without reducing costs of access/retail ISPs. Access/retail ISPs that peer with CDNs eliminate the cost of paying for transit for the content delivered by the CDN. That's what the initials CDN stand for. But, it does not mean both parties of the peer are equally benefited. As such peering may be paid one, though it may not be the current practice. Given the observed profitability of CDN providers, CDN providers are, seemingly, more benefited (because they are not neutral), in which case, CDN providers should pay to access/retail ISPs. Masataka Ohta
Re: DNS pulling BGP routes?
On Wed, Oct 13, 2021 at 6:26 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Matthew Petach wrote: > > >>> With an anycast setup using the same IP addresses in every > >>> location, returning SERVFAIL doesn't have the same effect, > >>> however, because failing over from anycast address 1 to > >>> anycast address 2 is likely to be routed to the same pop > >>> location, where the same result will occur. > >> > >> That's why that is a bad idea. Alternative name servers with > >> different IP addresses should be provided at separate locations. > > > Sure. But that doesn't do anything to help prevent the > > type of outage that hit Facebook, which was the point I > > was trying to make in my response. Facebook did use > different IP > addresses, and it didn't matter, because the > > underlying health of the network is what was at issue, > > not the health of the nameservers. > > A possible solution is to force unbundling of CDN providers and > transit providers by antitrust agencies. > Other people have already spoken to the misunderstanding or misuse of the terms "CDN provider" and "transit provider" in this case. I'd like to take a moment to point out the other problem with this sentence, which is "antitrust agencies". One of the key aspects to both CDN providers and transit providers is they tend to be multi-national organizations with infrastructure in multiple countries on multiple continents. A CDN provider that only exists in one city is a hosting company, not a CDN. A transit provider that only provides network connectivity in one city, or one state, isn't a very valuable transit provider, since the implicit (and sometimes explicit) promise the transit network is making to their customers is that they will carry their IP traffic to the rest of the world, ensuring as best as they can that their prefixes are visible to others, and that their packets are carried to other networks, wherever they may be. You won't be terribly successful as a transit provider if your business model is to "carry traffic for your customers all the way to the edges of the city", or "carry your traffic anywhere within the country it needs to go, but discard it if it needs to go outside the country." So, given that both our CDN provider and our transit network provider operate in more than one country, what "antitrust agency" would have jurisdiction over the CDN provider and the transit provider that could force unbundling of their services? What if every country the CDN provider and the transit provider operate in has a different definition of what it means to "unbundle" the services? Then, CDN providers can't pursue efficiency only to kill > fundamental redundancy of DNS. > > For network neutrality, backbone providers *MUST* be neutral > for contents they carry. > Nothing at all requires backbone providers to be neutral. Backbone networks are free to restrict what traffic or content passes across their networks. Indeed, many backbone providers include in their terms of service lists of traffic that they reserve the right to block or discard. Most of the time, those clauses are focused on traffic which may be injurious to the backbone network or the systems that support it; but even DDoS traffic which isn't itself injurious to the backbone, but does impact other customers, may be dropped at the backbone providers' discretion. We should recognize the fundamental difference between > independent, thus neutral, backbone providers and > CDN providers with anti-neutral backbone of their own. Others have, I think, already addressed more directly their fundamental disagreement with that statement. ^_^; > Masataka Ohta > > Thanks! :) Matt
Re: DNS pulling BGP routes?
On 10/16/21 06:48, Masataka Ohta wrote: Jay Hennigan wrote: Access/retail ISPs should want to peer with CDNs as it greatly reduces their transport costs. Not at all. Access/retail ISPs have no problem by peering with neutral backbone providers. Neutral backbone providers don't peer with access/retail ISPs. They sell transit to them. CDN provided backbone only reduces costs of other backbone providers without reducing costs of access/retail ISPs. Access/retail ISPs that peer with CDNs eliminate the cost of paying for transit for the content delivered by the CDN. That's what the initials CDN stand for. Access/retail ISPs that peer with CDNs don't reduce the costs of backbone providers, they reduce their profits. Those backbone providers no longer are charging to deliver the content provided by the CDNs. The retail/access ISPs are getting it direct at no charge from the CDN by peering. It also reduces the cost to the content provider as they no longer are paying a transit provider to deliver it. It also often increases the reliability of the Internet experience by creating a more direct path. Worse, peering beyond neutral providers costs more for access/retail providers. I think you are mistaken. Every gigabyte delivered by peering is a gigabyte that the access/retail ISP isn't paying a transit provider to deliver. -- Jay Hennigan - j...@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Re: DNS pulling BGP routes?
* mo...@necom830.hpcl.titech.ac.jp (Masataka Ohta) [Sat 16 Oct 2021, 15:50 CEST]: Access/retail ISPs have no problem by peering with neutral backbone providers. Getting strong Marie-Antoinette vibes here. -- Niels.
Re: DNS pulling BGP routes?
Jay Hennigan wrote: Access/retail ISPs should want to peer with CDNs as it greatly reduces their transport costs. Not at all. Access/retail ISPs have no problem by peering with neutral backbone providers. CDN provided backbone only reduces costs of other backbone providers without reducing costs of access/retail ISPs. Worse, peering beyond neutral providers costs more for access/retail providers. Masataka Ohta
Re: DNS pulling BGP routes?
Mark Tinka wrote: Remember that CDN providers are not neutral at all. Well, the purpose of a network is whatever its proprietor deems it to be, and makes no false advertising about it. What? A private enterprise network that carries a company's internal traffic - which may or may not interface with an external network that is interested in some or all of that traffic - would, in your eyes, be classified as not neutral, because it chooses not to use its network to provide global IP Transit? Unless they directly reach their end users, yes, of course. The fundamental problem of networking is the last mile problem that access costs alot more than backbone. As such, long distance carriers may peer with access providers only when they are neutral or pay some of there revenue share to access providers. In my mind, the word "transit" refers to carriage between two non-homogeneous points. So network A (customer) will talk to network C (content) via my network B (transit). If the traffic originates either from A or C, BUT terminates/ends inside of B, I do not consider that transit. With your definition, as CDN providers with their own backbone are not "transit", they can not request access providers (and, ultimately, end users) peering without paying some as compensation for access network cost. Otherwise, CDN providers with their own backbone are free riders ignoring access costs. Masataka Ohta
Re: DNS pulling BGP routes?
On 10/13/21 07:34, Masataka Ohta wrote: But, I certainly mean that CDN operators should not request peering directly to access/retail ISPs merely because they have their own transit, because the transit is not at all neutral. I'm not sure that I understand this. Peering is rarely if ever neutral. It's almost always "My network and customers only to your network and customers only." CDNs and their customers (content providers) peering with ISPs and their customers (eyeballs) seems to me to be a win-win. Access/retail ISPs should want to peer with CDNs as it greatly reduces their transport costs. CDNs will want to peer with access/retail ISPs for the same reason. Specifically what is the objection to CDNs peering with access ISPs? -- Jay Hennigan - j...@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Re: DNS pulling BGP routes?
On 10/13/21 17:24, Masataka Ohta wrote: The problem is that, unlike neutral transit providers, "the bits" is biased by the CDN provider. Then, access/retail ISPs who also want to supply their own contents, even though they must be neutral to contents provided by neutral transit providers, naturally refuse peering with the anti-neutral CDN providers. Remember that CDN providers are not neutral at all. Well, the purpose of a network is whatever its proprietor deems it to be, and makes no false advertising about it. A private enterprise network that carries a company's internal traffic - which may or may not interface with an external network that is interested in some or all of that traffic - would, in your eyes, be classified as not neutral, because it chooses not to use its network to provide global IP Transit? In my mind, the word "transit" refers to carriage between two non-homogeneous points. So network A (customer) will talk to network C (content) via my network B (transit). If the traffic originates either from A or C, BUT terminates/ends inside of B, I do not consider that transit. I'm unaware of content operators who run their own network and (promise to) provide connectivity between A and C. Mark.
Re: DNS pulling BGP routes?
Tom Beecher wrote: But, I certainly mean that CDN operators should not request peering directly to access/retail ISPs merely because they have their own transit, because the transit is not at all neutral. I'm still confused. Let's say I have a CDN network, with a datacenter somewhere, an edge site somewhere else. I carry my bits from my datacenter, across my internal network, to my edge site. This is where I intend to hand the bits over to someone else to carry them to the end user. The problem is that, unlike neutral transit providers, "the bits" is biased by the CDN provider. Then, access/retail ISPs who also want to supply their own contents, even though they must be neutral to contents provided by neutral transit providers, naturally refuse peering with the anti-neutral CDN providers. Remember that CDN providers are not neutral at all. Masataka Ohta
Re: DNS pulling BGP routes?
On Wed, Oct 13, 2021 at 10:56 AM Tom Beecher wrote: > But, I certainly mean that CDN operators should not request >> peering directly to access/retail ISPs merely because they have >> their own transit, because the transit is not at all neutral. >> > > I'm still confused. > > Let's say I have a CDN network, with a datacenter somewhere, an edge site > somewhere else. I carry my bits from my datacenter, across my internal > network, to my edge site. This is where I intend to hand the bits over to > someone else to carry them to the end user. > > Let's say in this site, I have a paid transit connection , and a peering > session directly with the end user's ISP. Where is anything related to > neutrality being 'violated', regardless of which path I choose to send the > bits out? > > It sounds like masataka is saying that the network between your 'datacenter' and 'cdn node' is a 'transit network'. I think 'transit network' is a sentence fragment much like: "bgp peer" .. it's overloaded (in this conversation at least) so probably some more clarity is required in the conversation to progress in a meaningful manner. > On Wed, Oct 13, 2021 at 10:36 AM Masataka Ohta < > mo...@necom830.hpcl.titech.ac.jp> wrote: > >> Tom Beecher wrote: >> >> >> For network neutrality, backbone providers *MUST* be neutral >> >> for contents they carry. >> >> >> >> However, CDN providers having their own backbone are using >> >> their backbone for contents they prefer, which is *NOT* >> >> neutral at all. >> >> >> >> As such, access/retail providers may pay for peering with >> >> neutral backbone providers for their customers but should >> >> reject direct peering request from, actively behaving against >> >> neutrality, CDN providers. >> >> > If I am understanding you correctly, are you arguing that anyone with a >> > network MUST be forced to become a transit provider for anyone else, in >> the >> > name of "neutrality"? >> >> No, not at all. >> >> For example, CDN (N stands for a network) operators may rely on >> neutral transit providers to connect their CDN to access/retail >> providers. >> >> But, I certainly mean that CDN operators should not request >> peering directly to access/retail ISPs merely because they have >> their own transit, because the transit is not at all neutral. >> >> Masataka Ohta >> >
Re: DNS pulling BGP routes?
> > But, I certainly mean that CDN operators should not request > peering directly to access/retail ISPs merely because they have > their own transit, because the transit is not at all neutral. > I'm still confused. Let's say I have a CDN network, with a datacenter somewhere, an edge site somewhere else. I carry my bits from my datacenter, across my internal network, to my edge site. This is where I intend to hand the bits over to someone else to carry them to the end user. Let's say in this site, I have a paid transit connection , and a peering session directly with the end user's ISP. Where is anything related to neutrality being 'violated', regardless of which path I choose to send the bits out? On Wed, Oct 13, 2021 at 10:36 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Tom Beecher wrote: > > >> For network neutrality, backbone providers *MUST* be neutral > >> for contents they carry. > >> > >> However, CDN providers having their own backbone are using > >> their backbone for contents they prefer, which is *NOT* > >> neutral at all. > >> > >> As such, access/retail providers may pay for peering with > >> neutral backbone providers for their customers but should > >> reject direct peering request from, actively behaving against > >> neutrality, CDN providers. > > > If I am understanding you correctly, are you arguing that anyone with a > > network MUST be forced to become a transit provider for anyone else, in > the > > name of "neutrality"? > > No, not at all. > > For example, CDN (N stands for a network) operators may rely on > neutral transit providers to connect their CDN to access/retail > providers. > > But, I certainly mean that CDN operators should not request > peering directly to access/retail ISPs merely because they have > their own transit, because the transit is not at all neutral. > > Masataka Ohta >
Re: DNS pulling BGP routes?
Tom Beecher wrote: For network neutrality, backbone providers *MUST* be neutral for contents they carry. However, CDN providers having their own backbone are using their backbone for contents they prefer, which is *NOT* neutral at all. As such, access/retail providers may pay for peering with neutral backbone providers for their customers but should reject direct peering request from, actively behaving against neutrality, CDN providers. If I am understanding you correctly, are you arguing that anyone with a network MUST be forced to become a transit provider for anyone else, in the name of "neutrality"? No, not at all. For example, CDN (N stands for a network) operators may rely on neutral transit providers to connect their CDN to access/retail providers. But, I certainly mean that CDN operators should not request peering directly to access/retail ISPs merely because they have their own transit, because the transit is not at all neutral. Masataka Ohta
Re: DNS pulling BGP routes?
> > For network neutrality, backbone providers *MUST* be neutral > for contents they carry. > > However, CDN providers having their own backbone are using > their backbone for contents they prefer, which is *NOT* > neutral at all. > > As such, access/retail providers may pay for peering with > neutral backbone providers for their customers but should > reject direct peering request from, actively behaving against > neutrality, CDN providers. > If I am understanding you correctly, are you arguing that anyone with a network MUST be forced to become a transit provider for anyone else, in the name of "neutrality"? I'll reserve further comment until I make sure I have grasped your point. On Wed, Oct 13, 2021 at 9:28 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Matthew Petach wrote: > > >>> With an anycast setup using the same IP addresses in every > >>> location, returning SERVFAIL doesn't have the same effect, > >>> however, because failing over from anycast address 1 to > >>> anycast address 2 is likely to be routed to the same pop > >>> location, where the same result will occur. > >> > >> That's why that is a bad idea. Alternative name servers with > >> different IP addresses should be provided at separate locations. > > > Sure. But that doesn't do anything to help prevent the > > type of outage that hit Facebook, which was the point I > > was trying to make in my response. Facebook did use > different IP > addresses, and it didn't matter, because the > > underlying health of the network is what was at issue, > > not the health of the nameservers. > > A possible solution is to force unbundling of CDN providers and > transit providers by antitrust agencies. > > Then, CDN providers can't pursue efficiency only to kill > fundamental redundancy of DNS. > > For network neutrality, backbone providers *MUST* be neutral > for contents they carry. > > However, CDN providers having their own backbone are using > their backbone for contents they prefer, which is *NOT* > neutral at all. > > As such, access/retail providers may pay for peering with > neutral backbone providers for their customers but should > reject direct peering request from, actively behaving against > neutrality, CDN providers. > > > I agree with you--different IP addresses should be > > used in different geographic locations, even with > > anycast setups. > > > > But people need to also recognize that's not a > > panacea that solves everything, and that it wouldn't > > have changed the nature of the outage last week. > > We should recognize the fundamental difference between > independent, thus neutral, backbone providers and > CDN providers with anti-neutral backbone of their own. > > Masataka Ohta >
Re: DNS pulling BGP routes?
Matthew Petach wrote: With an anycast setup using the same IP addresses in every location, returning SERVFAIL doesn't have the same effect, however, because failing over from anycast address 1 to anycast address 2 is likely to be routed to the same pop location, where the same result will occur. That's why that is a bad idea. Alternative name servers with different IP addresses should be provided at separate locations. Sure. But that doesn't do anything to help prevent the type of outage that hit Facebook, which was the point I was trying to make in my response. Facebook did use > different IP addresses, and it didn't matter, because the > underlying health of the network is what was at issue, > not the health of the nameservers. A possible solution is to force unbundling of CDN providers and transit providers by antitrust agencies. Then, CDN providers can't pursue efficiency only to kill fundamental redundancy of DNS. For network neutrality, backbone providers *MUST* be neutral for contents they carry. However, CDN providers having their own backbone are using their backbone for contents they prefer, which is *NOT* neutral at all. As such, access/retail providers may pay for peering with neutral backbone providers for their customers but should reject direct peering request from, actively behaving against neutrality, CDN providers. I agree with you--different IP addresses should be used in different geographic locations, even with anycast setups. But people need to also recognize that's not a panacea that solves everything, and that it wouldn't have changed the nature of the outage last week. We should recognize the fundamental difference between independent, thus neutral, backbone providers and CDN providers with anti-neutral backbone of their own. Masataka Ohta
Re: DNS pulling BGP routes?
On Tue, Oct 12, 2021 at 8:41 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Matthew Petach wrote: > > > With an anycast setup using the same IP addresses in every > > location, returning SERVFAIL doesn't have the same effect, > > however, because failing over from anycast address 1 to > > anycast address 2 is likely to be routed to the same pop > > location, where the same result will occur. > > That's why that is a bad idea. Alternative name servers with > different IP addresses should be provided at separate locations. > > Masataka Ohta > > Sure. But that doesn't do anything to help prevent the type of outage that hit Facebook, which was the point I was trying to make in my response. Facebook did use different IP addresses, and it didn't matter, because the underlying health of the network is what was at issue, not the health of the nameservers. I agree with you--different IP addresses should be used in different geographic locations, even with anycast setups. But people need to also recognize that's not a panacea that solves everything, and that it wouldn't have changed the nature of the outage last week. Thanks! :) Matt
Re: DNS pulling BGP routes?
Matthew Petach wrote: With an anycast setup using the same IP addresses in every location, returning SERVFAIL doesn't have the same effect, however, because failing over from anycast address 1 to anycast address 2 is likely to be routed to the same pop location, where the same result will occur. That's why that is a bad idea. Alternative name servers with different IP addresses should be provided at separate locations. Masataka Ohta
Re: DNS pulling BGP routes?
Christopher Morrow wrote: To be fair, it looks like FB has 4 /32's (and 4 /128's) for their DNS authoritatives. All from different /24's or /48's, so they should have decent routing diversity. They could choose to announce half/half from alternate pops, or other games such as this. Yup. I don't know that that would have solved any of the problems last week nor any problems in the future. There are various solutions. For example, if FB had relied on, instead of route withdrawal, standard DNS expire mechanism, FB should have noticed that FB needed another zone for stable data for maintenance servers, I think. > I think Bill's slide 30 is pretty much what FB has/had deployed: It seems to me that he assumes transit providers and cloud providers are different entities. FB, instead, operate their own transit network and clouds within its domain and clouds are connected only by FB transit (there aren't multiple (red and green) transit). it's also not clear that FB is connecting their CDN to single points in any provider... I'd guess there are some cases of that, That is bad enough, if FB wants to "optimize" their traffic for the cases by killing DNS redundancy to put all the name servers in single POP, which is my concern. Masataka Ohta
Re: DNS pulling BGP routes?
On Mon, Oct 11, 2021 at 8:07 AM Christopher Morrow wrote: > On Sat, Oct 9, 2021 at 11:16 AM Masataka Ohta < > mo...@necom830.hpcl.titech.ac.jp> wrote: > >> Bill Woodcock wrote: >> > [...] > > it seems that the problem FB ran into was really that there wasn't either: >"secondary path to communicate: "You are the last one standing, do not > die" (to an edge node) > or: > "maintain a very long/less-preferred path to a core location(s) to > maintain service in case the CDN disappears" > > There are almost certainly more complexities which FB is not discussion in > their design/deployment which > affected their services last week, but it doesn't look like they were very > far off on their deployment, if they > need to maintain back-end connectivity to serve customers from the CDN > locales. > > -chris > Having worked on trying to solve health-checking situations in large production complexes in the past, I can definitely say that is is an exponentially difficult problem for a single site to determine whether it is "safe" for it to fail out, or if doing so will result in an entire service going offline, short of having a central controller which tracks every edge site's health, and can determine "no, we're below $magic_threshold number of sites, you can't fail yourself out no matter how unhealthy you think you are". Which of course you can't really have, without undoing one of the key reasons for distributing your serving sites to geographically distant places in different buildings on different providers--namely to eliminate single points of failure in your serving infrastructure. Doing the equivalent of "no router bgp" on your core backbone is going to make things suck, no matter how you slice it, and I don't think any amount of tweaking the anycast setup or DNS values would have made a whit of difference to the underlying outage. I think the only question we can armchair quarterback at this point is whether there were prudent steps that could go into a design to shorten the recovery interval. So far, we seem to have collected a few key points: 1) make sure your disaster recovery plan doesn't depend on your production DNS servers being usable; have key nodes in /etc/hosts files that are periodically updated via $automation_tool, but ONLY for non-production, out-of-band recovery nodes; don't static any of your production-facing entries. 2) Have a working out-of-band that exists entirely independent of your production network. Dial, frame relay, SMDS, LTE modems, starlink dishes on the roof; pick your poison, but budget it in for every production site. Test it monthly to ensure connectivity to all sites works. Audit regularly to ensure no dependencies on the production infrastructure have crept in. 3) Ensure you have a good "oh sh**" physical access plan for key personnel. Some of you at a recent virtual happy hour heard me talk about the time I isolated the credit card payment center for a $dayjob, which also cut off access for the card readers to get into it to restore the network. Use of a fire axe was granted to on-site personnel during that. Take the time to think through how physical access is controlled for every key site in your network, think about failure scenarios, and have a "in case of emergency, break glass to get the key" plan in place to shorten recovery times. 4) Have a dependency map/graph of your production network. a) if everything dies, and you have to restart, what has to come up first? b) what dependencies are there that have to be done in the right order c) what services are independent that can be brought up in parallel to speed up recovery? d) does every team supporting services on the critical, dependent pathway have 24x7 on-call coverage, and do they know where in the recovery graph they're needed? It doesn't help to have teams that can't start back up until step 9 crowding around asking "are you ready for us yet?" when you still can't raise the team needed for step 1 on the dependency graph. ^_^; 5) do you know how close the nearest personnel are to each POP/CDN node, in case you have to do emergency "drive over with a laptop, hop on the console, and issue the following commands" rousting in the middle of the night? If someone lives.3 miles from the CDN node, it's good to know that, so you don't call the person who is on-call but is 2 hours away without first checking if the person 3 miles away can do it faster. I'm sure others have even better experiences than I, who can contribute and add to the list. If nothing else, perhaps collectively we can help other companies prepare a bit better, so that when the next big "ooops" happens, the recovery time can be a little bit shorter. :) Thanks! Matt
Re: DNS pulling BGP routes?
On Sat, Oct 9, 2021 at 1:40 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Christopher Morrow wrote: > >> means their DNS servers were serving the zone, even after they > >> recognize their zone data were too old, that is, expired. > > > that's not what this means. I think Mr. Petach previously described > > this, > > He wrote: > > > So, the idea is that if the edge CDN node loses connectivity to > > the core datacenters, the DNS servers should stop answering > > queries for A records with the local CDN node's address, and > > let a different site respond back to the client's DNS request. > > which may be performed by standard DNS with short expire period, > after which name servers will return SERVFAIL and other name > servers in other edge node with different IP addresses are tried. > (Apologies for the delayed response--I had back-to-back board meetings the past two days which had me completely tied up.) That is one way in which it *could* be done--but is by no means the ONLY way in which it can be done. With an anycast setup using the same IP addresses in every location, returning SERVFAIL doesn't have the same effect, however, because failing over from anycast address 1 to anycast address 2 is likely to be routed to the same pop location, where the same result will occur. You don't really want to hunt among different *IP addresses*, you want to hunt to a different *location*. This is why withdrawing the BGP announcement from that location works more effectively, because it allows the clients to continue querying the same IP address, but get routed to the next most proximal location. If you simply return SERVFAIL and have the client pick a different IP address from the list of NS entries, it falls into one of two situations: a) the new IP address is also anycasted, and is therefore likely to pick the same pop that is unhealthy, with similar results, or b) the new IP address is *not* anycasted, but is served from a single geographical location, which means answers given back by that DNS server are unlikely to be geolocated with any accuracy, and therefore the content served is also unlikely to be geographically relevant or correct. > > It may be that facebook uses all the four name server IP addresses > in each edge node. But, it effectively kills essential redundancy > of DNS to have two or more name servers (at separate locations) > and the natural consequence is, as you can see, mass disaster. > Even if the four anycasted nameserver IP addresses weren't completely overlapping (let's assume as a hypothetical that a.ns is served out of EU pops, b.ns is served out of NA pops, c.ns is served out of SA pops, and d.ns is served out of APAC pops), if all sites run the same healthcheck code, then if the underlying healthcheck fails, *every site* will decide it is unhealthy, and stop answering requests; so, all the EU sites fail health check and stop serving a.ns; all the North America sites fail health check, and stop serving b.ns...and so forth. You followed the best practices, you had different NS entries that were on different subnets, that were geographically dispersed around the globe, that were redundant for each other. But because they all used the same fundamental health check, they all *independently* decided they were unhealthy and needed to stop giving out DNS answers, and instead let one of the other healthier sites take over. > > > but: 1) dns server in pop serves some content (ttls aren't > > important right now) > > You MUST distinguish TTL and EXPIRE. They are different. > TTL and EXPIRE are irrelevant here. The only thing changing those values would do is change how long it took for caching resolvers to reflect the loss of connectivity at the DNS layer. Once the underlying layer 3 connectivity had broken, DNS answers became meaningless. No matter what records were returned, or cached, you couldn't reach the servers. Yes, yes, as an academic exercise you can point out that there's a difference in how and when those DNS records stop being used, and you're right about that--but in terms of this particular failure, this particular post-mortem we're beating to a horse-shaped pulp, it's entirely meaningless. ^_^; > > > there's not a lot of magic here... and it's not about the zone data > > really at all. > > Statement of Petach: "the edge CDN node loses connectivity to > the core datacenters, the DNS servers should stop answering" > means, with DNS terminology, zone data is expired, which has > nothing to do with TTL. > As you're using my words, I'm going to have to point out that "the DNS servers should stop answering" does not require that any change happens *at the DNS layer* -- in this case, the change can happen at the routing layer, ensuring that even if some caching resolver out there is completely defiant of your expire time, you *will not answer* because the query packets can never reach you in the first place. >
Re: DNS pulling BGP routes?
On Sat, Oct 9, 2021 at 11:16 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Bill Woodcock wrote: > > >> It may be that facebook uses all the four name server IP addresses > >> in each edge node. But, it effectively kills essential redundancy > >> of DNS to have two or more name servers (at separate locations) > >> and the natural consequence is, as you can see, mass disaster. > > > > Yep. I think we even had a NANOG talk on exactly that specific topic a > long time ago. > > > > > https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf > > Yes, having separate sets of anycast addresses by two or more pops > should be fine. > > To be fair, it looks like FB has 4 /32's (and 4 /128's) for their DNS authoritatives. All from different /24's or /48's, so they should have decent routing diversity. They could choose to announce half/half from alternate pops, or other games such as this. I don't know that that would have solved any of the problems last week nor any problems in the future. I think Bill's slide 30 is pretty much what FB has/had deployed: 1) I would think the a/b cloud is really 'as similar a set of paths from like deployments as possible 2) redundant pairs of servers in the same transit/network 3) hidden masters (almost certainly these are in the depths of the FB datacenter network) (though also this part isn't important for the conversation) 4) control/sync traffic on a different topology than the customer serving one > However, if CDN provider has their own transit backbone, which is, > seemingly, not assumed by your slides, and retail ISPs are tightly > I think it is, actually, in slide 30 ? "We need a network topology to carry control and synchronization traffic between the nodes" connected to only one pop of the CDN provider, the CDN provider > it's also not clear that FB is connecting their CDN to single points in any provider... I'd guess there are some cases of that, but for larger networks I would imagine there are multiple CDN deployments per network. I can't imagine that it's safe to deploy 1 CDN node for all of 7018 or 3320... for instance. > may be motivated to let users access only one pop killing essential > redundancy of DNS, which should be overengineering, which is my > concern of the paragraph quoted by you. > > it seems that the problem FB ran into was really that there wasn't either: "secondary path to communicate: "You are the last one standing, do not die" (to an edge node) or: "maintain a very long/less-preferred path to a core location(s) to maintain service in case the CDN disappears" There are almost certainly more complexities which FB is not discussion in their design/deployment which affected their services last week, but it doesn't look like they were very far off on their deployment, if they need to maintain back-end connectivity to serve customers from the CDN locales. -chris
Re: DNS pulling BGP routes?
Owen DeLong wrote: But, it has nothing specifically to do with anycast. As there are other name servers with different IP addresses, there is no reason to withdraw routes. So? WRONG. First, assuming that there are non-anycast name servers assumes facts not in evidence. There is no such assumption. Second, if you are a participant in an anycast name server network, there are good reasons to withdraw your announcement of that prefix You completely misunderstand my points. See other posts of mine. Masataka Ohta
Re: DNS pulling BGP routes?
> On Oct 7, 2021, at 06:49 , Masataka Ohta > wrote: > > William Herrin wrote: > This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. >>> >>> Yes, that is a commonly seen mistake with anycast. >> You don't know what you're talking about. > > I do but you don't. > >> If your anycast node stops >> receiving updated data and you can't reach any of the other nodes to >> check whether they're online, 99 times out of 100 this means a local >> failure of some sort. > > Yes. In case of DNS, if expiration period of a zone is passed > without successful check of the current most zone version, > unicast or anycast name servers stop responding requests for > the zone. > > But, it has nothing specifically to do with anycast. As there > are other name servers with different IP addresses, there is > no reason to withdraw routes. So? WRONG. First, assuming that there are non-anycast name servers assumes facts not in evidence. Second, if you are a participant in an anycast name server network, there are good reasons to withdraw your announcement of that prefix in order to avoid users having to wait for timeouts (which in some cases might be even worse than serving stale data). >> You withdraw the node's announcement so that you >> don't serve bad data to the end user. > > That will only introduce new failure modes of mismatches between > server availability and server reachability and is a bad idea. No, if the server is available, it should announce the anycast prefix. If it i snot available, it should withdraw it. That’s the best way to make anycast work and it’s what virtually every anycast DNS server network does. If the server is unavailable, but doesn’t withdraw, then you have the failure mode of the server being reachable, but unavailable and it becomes a black hole for traffic that should otherwise flow to other available anycast nodes. >> That's what happened here - > > Yes, facebook did wrong thing to actively withdraw routes. No, facebook did the right thing for 99+% of situations that would trigger this withdraw. The problem was that they withdrew EVERY server when the failure wasn’t local instead of having some way to recognize the failure for what it was, global in nature and continue serving DNS. >> Simply >> turning themselves off, instead of withdrawing the routes, would >> result in suboptimal performance. > > This time, facebook is saying that they could not reach their > name servers even though the servers were perfectly working. Because their servers couldn’t verify that they were working and thus thought that they had stale data. Thus, the servers were “perfectly working” with stale data and the safe thing to do if you can’t confirm that your reason for believing you have stale data is erroneous, is to stop serving what you have. If you’re not going to serve what you have, then you shouldn’t announce the anycast prefix, either. > How much performance, do you think, facebook enjoyed? A lot > less than "suboptimal", I'm afraid. As noted, this was that 1% failure that isn’t anticipated. The behavior of the system was correct for 99% of failures and the number of years facebook has operated without a significant or noticeable DNS outage is testament to that fact. > > > And 99 times out of 100, not doing > > one or the other would cause rather than prevent an outage. > > That is a commonly seen misconception wrongly assuming that > server routes were withdrawn if and only if the server is > unavailable. The servers withdrew their routes because the servers had no ability to verify that they were serving valid data. If you can’t verify your data is valid, it’s better (in most cases) to not serve the data you have. If you’re not going to serve, the best thing to do is withdraw the anycast prefix that claims you are a server for the data. > But, the reality is that it is impossible to correctly > recognize server is unavailable or to correctly withdraw > routes only when server is unavailable. Yes… So you go with something that works 99% of the time and you get an event like this in that 1% of cases where the failure in question was not one of the failure modes that was previously anticipated. I’m betting that facebook is quickly figuring out changes that will mitigate this type of failure in the future and their DNS will likely stay up until the next 1 in 100 (or will it be 1 in 10,000 this time?) events pops up that surprised them again. That’s the nature of operations. Owen
Re: DNS pulling BGP routes?
Bill Woodcock wrote: It may be that facebook uses all the four name server IP addresses in each edge node. But, it effectively kills essential redundancy of DNS to have two or more name servers (at separate locations) and the natural consequence is, as you can see, mass disaster. Yep. I think we even had a NANOG talk on exactly that specific topic a long time ago. https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf Yes, having separate sets of anycast addresses by two or more pops should be fine. However, if CDN provider has their own transit backbone, which is, seemingly, not assumed by your slides, and retail ISPs are tightly connected to only one pop of the CDN provider, the CDN provider may be motivated to let users access only one pop killing essential redundancy of DNS, which should be overengineering, which is my concern of the paragraph quoted by you. Masataka Ohta
Re: DNS pulling BGP routes?
> On Oct 9, 2021, at 10:37 AM, Masataka Ohta > wrote: > It may be that facebook uses all the four name server IP addresses > in each edge node. But, it effectively kills essential redundancy > of DNS to have two or more name servers (at separate locations) > and the natural consequence is, as you can see, mass disaster. Yep. I think we even had a NANOG talk on exactly that specific topic a long time ago. https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf -Bill signature.asc Description: Message signed with OpenPGP
Re: DNS pulling BGP routes?
On Fri, Oct 8, 2021 at 10:04 AM Christopher Morrow wrote: > On Fri, Oct 8, 2021 at 10:22 AM Masataka Ohta > wrote: >> The end result was that our DNS servers became unreachable >> even though they were still operational. >> >> means their DNS servers were serving the zone, even after >> they recognize their zone data were too old, that is, expired. > > that's not what this means. Give it up man. Masataka knows more about how Facebook implemented DNS than people who actually worked there. He will tell them (and us) what their public statements really mean. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
Christopher Morrow wrote: means their DNS servers were serving the zone, even after they recognize their zone data were too old, that is, expired. that's not what this means. I think Mr. Petach previously described this, He wrote: So, the idea is that if the edge CDN node loses connectivity to the core datacenters, the DNS servers should stop answering queries for A records with the local CDN node's address, and let a different site respond back to the client's DNS request. which may be performed by standard DNS with short expire period, after which name servers will return SERVFAIL and other name servers in other edge node with different IP addresses are tried. It may be that facebook uses all the four name server IP addresses in each edge node. But, it effectively kills essential redundancy of DNS to have two or more name servers (at separate locations) and the natural consequence is, as you can see, mass disaster. but: 1) dns server in pop serves some content (ttls aren't important right now) You MUST distinguish TTL and EXPIRE. They are different. > there's not a lot of magic here... and it's not about the zone data > really at all. Statement of Petach: "the edge CDN node loses connectivity to the core datacenters, the DNS servers should stop answering" means, with DNS terminology, zone data is expired, which has nothing to do with TTL. Masataka Ohta
Re: DNS pulling BGP routes?
(I'm going to hate myself in the morning, but) On Fri, Oct 8, 2021 at 10:22 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > William Herrin wrote: > > > https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ > > our DNS servers disable those BGP advertisements if they > themselves can not speak to our data centers > > The end result was that our DNS servers became unreachable > even though they were still operational. > > means their DNS servers were serving the zone, even after > they recognize their zone data were too old, that is, expired. > > that's not what this means. I think Mr. Petach previously described this, but: 1) dns server in pop serves some content (ttls aren't important right now) 2) dns server uses some quagga/gated/bird/etc to announce locally: "Hey, foo/32 here!" (imagine this triggers an 'aggregate route' or 'network statement' (pick your vendor solution) to appear in the global table) 3) dns server also 'ping backend server set' 4) when 3 fails for X period of time 'tell quagga/bird/etc to stop announcing the /32' then the local pop no longer sources the aggregate (/24 or /23 or whatever)... so traffic SHOULD (externally) flow toward another copy of the /23 or /24 or whatever... there's not a lot of magic here... and it's not about the zone data really at all.
Re: DNS pulling BGP routes?
William Herrin wrote: If they are not using standard expire mechanism expecting internal data still accessible even after external data has expired, there is difference. I give up. To accept the reality of disastrous facebook failure? I know. Although you have no knowledge whatsoever about how Facebook implemented their DNS https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers The end result was that our DNS servers became unreachable even though they were still operational. means their DNS servers were serving the zone, even after they recognize their zone data were too old, that is, expired. you are obviously correct in all things. If you think so, it's your problem, I'm afraid. Masataka Ohta
Re: DNS pulling BGP routes?
> > In facebook case, it was combined with poor understanding > on short/long expiration period to cause the disaster. > Still, no. The CAUSE of the outage was all of the FB datacenters being completely disconnected from their backbone, and thus the internet. DNS breaking was a direct RESULT of that. Even if FB's DNS was happily still providing answers to IPs that were still unreachable, they were still horked. Could their DNS design possibly have contributed to some delay in the RESTORATION phase? Perhaps. But with the volume of traffic they do, that was certainly going to take a while anyways. On Fri, Oct 8, 2021 at 5:17 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > Sabri Berisha wrote: > > > Let's for a moment contemplate about the sheer magnitude of > > their operation. With almost 3 billion users worldwide, can you imagine > the > > amount of DNS queries they have to process? Their scale is unprecedented. > That's what I predicted about 20 years ago, which is why > I proposed to have anycast name servers analyzing its > implications. > > As such I'm sure anycast route withdrawal ignoring rfc3258 > is poor engineering. > > Scalable solutions can be constructed only with careful > theoretical analysis, against which random hacks, which > may work 99% of the time, are just harmful. > > In facebook case, it was combined with poor understanding > on short/long expiration period to cause the disaster. > > Masataka Ohta >
Re: DNS pulling BGP routes?
On Thu, Oct 7, 2021 at 9:04 PM Masataka Ohta wrote: > William Herrin wrote: > > Facebook withdrawing the BGP > > routes to its anycasted public DNS servers as they expired made no > > difference. > > If they are not using standard expire mechanism expecting > internal data still accessible even after external data > has expired, there is difference. I give up. Although you have no knowledge whatsoever about how Facebook implemented their DNS you are obviously correct in all things. -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
On 2021-10-08, at 07:25, Sabri Berisha wrote: > > Whenever there is an aviation incident, the keyboard warriors at pprune.org > are always the first to start speculating about root causes So we need an NTSB, BFU, ... for the Internet and widely used Internet applications. (And the other national equivalents…) A site like avherald.com would also be useful (minor todo: Find someone as dedicated as Simon Hradecky to run it). Grüße, Carsten
Re: DNS pulling BGP routes?
Sabri Berisha wrote: Let's for a moment contemplate about the sheer magnitude of their operation. With almost 3 billion users worldwide, can you imagine the amount of DNS queries they have to process? Their scale is unprecedented. That's what I predicted about 20 years ago, which is why I proposed to have anycast name servers analyzing its implications. As such I'm sure anycast route withdrawal ignoring rfc3258 is poor engineering. Scalable solutions can be constructed only with careful theoretical analysis, against which random hacks, which may work 99% of the time, are just harmful. In facebook case, it was combined with poor understanding on short/long expiration period to cause the disaster. Masataka Ohta
Re: DNS pulling BGP routes?
On 10/8/21 07:25, Sabri Berisha wrote: Whenever there is an aviation incident, the keyboard warriors at pprune.org are always the first to start speculating about root causes, and complain how the air crew made mistakes. They, the keyboard warriors, of course know how best to fly an aircraft with 20/20 hindsight from their armchairs. Why do I see so many posts that are basically throwing Facebook engineers under the bus? Let's for a moment contemplate about the sheer magnitude of their operation. With almost 3 billion users worldwide, can you imagine the amount of DNS queries they have to process? Their scale is unprecedented. Sure, it's ok to speculate about potential operational or design issues that may have been contributing factors to the outage. But throwing our colleagues in front of the lions like this is something I would not recommend. I'm sure they are aware of these posts, but are unable to reply due to the amount of NDAs signed. Folk love to complain and critique. It's human nature, and unproductive quality that is part of our DNA. The good news is that lots of human beings have the DNA to ignore noise and carry on helping mankind to grow and live better lives. In any event, if you aren't willing to put your neck on the line and fail spectacularly, I don't care what you have to say. Failure is finding out what doesn't work, quickly, and inching closer to what does. I'm all for that. Mark.
Re: DNS pulling BGP routes?
- On Oct 7, 2021, at 9:03 PM, Masataka Ohta mo...@necom830.hpcl.titech.ac.jp wrote: Hi, > It means DNS management of facebook is poor. Whenever there is an aviation incident, the keyboard warriors at pprune.org are always the first to start speculating about root causes, and complain how the air crew made mistakes. They, the keyboard warriors, of course know how best to fly an aircraft with 20/20 hindsight from their armchairs. Why do I see so many posts that are basically throwing Facebook engineers under the bus? Let's for a moment contemplate about the sheer magnitude of their operation. With almost 3 billion users worldwide, can you imagine the amount of DNS queries they have to process? Their scale is unprecedented. Sure, it's ok to speculate about potential operational or design issues that may have been contributing factors to the outage. But throwing our colleagues in front of the lions like this is something I would not recommend. I'm sure they are aware of these posts, but are unable to reply due to the amount of NDAs signed. Thanks, Sabri
Re: DNS pulling BGP routes?
William Herrin wrote: Facebook's _internal_ DNS, while not anycasted, followed a similar logic: if the data center is isolated and their data goes stale, they stop serving potentially wrong answers. As I already wrote, that is a standard mechanism of DNS with SOA expiration period as is documented in rfc1034 Then we agree: Do we? The failure mode was that after the data centers disconnected from each other, all their DNS expired, breaking the tools they'd normally use to recover. It means DNS management of facebook is poor. If they are using standard expire mechanism, they should have used two zones facebook.com for external users with short expire and internal.facebook.com for internal users with long expire. Facebook withdrawing the BGP routes to its anycasted public DNS servers as they expired made no difference. If they are not using standard expire mechanism expecting internal data still accessible even after external data has expired, there is difference. Masataka Ohta
Re: DNS pulling BGP routes?
On Thu, Oct 7, 2021 at 10:23 AM Masataka Ohta wrote: > William Herrin wrote: > > Facebook's _internal_ DNS, while not anycasted, followed a similar > > logic: if the data center is isolated and their data goes stale, they > > stop serving potentially wrong answers. > > As I already wrote, that is a standard mechanism of DNS with SOA > expiration period as is documented in rfc1034 Then we agree: The failure mode was that after the data centers disconnected from each other, all their DNS expired, breaking the tools they'd normally use to recover. Facebook withdrawing the BGP routes to its anycasted public DNS servers as they expired made no difference. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
William Herrin wrote:
Facebook's _internal_ DNS, while not anycasted, followed a similar
logic: if the data center is isolated and their data goes stale, they
stop serving potentially wrong answers.
As I already wrote, that is a standard mechanism of DNS with SOA
expiration period as is documented in rfc1034 as ("an discard"
should be "and discard"):
If the secondary finds it
impossible to perform a serial check for the EXPIRE interval, it must
assume that its copy of the zone is obsolete an discard it.
But, that has nothing to do with anycast or route (BGP or IGP)
withdrawal.
I didn't work for the DNS team when I worked as a production engineer
for Facebook but I worked close enough to understand what happened
from the posted description.
I don't think those who post the description properly understand
what is wrong with their management.
Masataka Ohta
RE: DNS pulling BGP routes?
Well said Bill. I agree with you about having all your tech/adm records + registrar on the same NS... especially for your OOB domain. Probably what killed them. They lost access to their fb-00b-net-mgmt.io cool dns name network. It just went from bad to worst when they realized that they also lost physical access to the building. We all learned a lot and we're still learning. Jean -Original Message- From: Bill Woodcock Sent: October 7, 2021 12:45 PM To: Jean St-Laurent Cc: Masataka Ohta ; Bjørn Mork ; nanog@nanog.org Subject: Re: DNS pulling BGP routes? This was superstition, brought forward from 1992 by the folks who were yelling “damned kids get offa my lawn” at the time. There’s no reason to include a unicast address in an NS set in the 21st century, and plenty of reasons not to (since it’ll be very difficult to load-balance with the rest of the servers). But one should NEVER NEVER depend on a single administrative or technical authority for all your NS records. That’s what shot Facebook in the foot, they were trying to do it all themselves, so when they shot themselves in the foot, they only had the one foot, and nothing left to stand on. Whereas other folks shoot themselves in the foot all the time, and nobody notices, because they paid attention to the spirit of RFC 2182. -Bill
Re: DNS pulling BGP routes?
On Thu, Oct 7, 2021 at 9:52 AM Masataka Ohta wrote: > But, this time, the reality strikes back. Not really. Or at all. Facebook the external service was down hard as soon as the cross-datacenter connections all failed. Whether or not the BGP routes for the external DNS were withdrawn had no impact on the outage. Facebook's _internal_ DNS, while not anycasted, followed a similar logic: if the data center is isolated and their data goes stale, they stop serving potentially wrong answers. Since the routing failure isolated all of the data centers, this left no usable _INTERNAL_ DNS on which more or less everything else depends. I didn't work for the DNS team when I worked as a production engineer for Facebook but I worked close enough to understand what happened from the posted description. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
William Herrin wrote: It wasn't forgotten. Folks gained a lot of experience with anycast DNS between 2002 and 2006. Not withdrawing the routes when the servers are deemed malfunctioning turned out not to be an operationally sound practice. The theory offered in 3258 was wrong. So, from limited experience, you thought it were wrong because: > Simply > turning themselves off, instead of withdrawing the routes, would > result in suboptimal performance. But, this time, the reality strikes back. That you can be safe 99 times out of 100 can mean remaining 1 time is totally disastrous. When servers are deemed malfunctioning, the best practice is to check whether the servers are really malfunctioning or not before blindly shutdown the servers. Masataka Ohta
Re: DNS pulling BGP routes?
> On Oct 7, 2021, at 6:25 PM, Jean St-Laurent via NANOG wrote: > > Nice document. > > In section 2.5 Routing, this is written: > > Distributing Authoritative Name Servers via Shared Unicast Addresses... > > organizations implementing these practices should > always provide at least one authoritative server which is not a > participant in any shared unicast mesh. This was superstition, brought forward from 1992 by the folks who were yelling “damned kids get offa my lawn” at the time. There’s no reason to include a unicast address in an NS set in the 21st century, and plenty of reasons not to (since it’ll be very difficult to load-balance with the rest of the servers). But one should NEVER NEVER depend on a single administrative or technical authority for all your NS records. That’s what shot Facebook in the foot, they were trying to do it all themselves, so when they shot themselves in the foot, they only had the one foot, and nothing left to stand on. Whereas other folks shoot themselves in the foot all the time, and nobody notices, because they paid attention to the spirit of RFC 2182. -Bill signature.asc Description: Message signed with OpenPGP
Re: DNS pulling BGP routes?
On 10/7/21 18:21, William Herrin wrote: It wasn't forgotten. Folks gained a lot of experience with anycast DNS between 2002 and 2006. Not withdrawing the routes when the servers are deemed malfunctioning turned out not to be an operationally sound practice. The theory offered in 3258 was wrong. Especially terrible when you have a DNS daemon that has crashed, but Quagga (or whatever routing suite you use) is still humming. Mark.
RE: DNS pulling BGP routes?
Nice document. In section 2.5 Routing, this is written: Distributing Authoritative Name Servers via Shared Unicast Addresses... organizations implementing these practices should always provide at least one authoritative server which is not a participant in any shared unicast mesh. Could it be that by having the NS a,b in one mesh and c,d in another was a mistake? -Original Message- From: NANOG On Behalf Of Masataka Ohta Sent: October 7, 2021 11:27 AM To: Bjørn Mork Cc: nanog@nanog.org Subject: Re: DNS pulling BGP routes? Bjørn Mork wrote: >>>>> This is quite common to tie an underlying service announcement to >>>>> BGP announcements in an Anycast or similar environment. >>>> >>>> Yes, that is a commonly seen mistake with anycast. >>> You don't know what you're talking about. >> >> I do but you don't. > > https://datatracker.ietf.org/doc/html/rfc4786#section-4.4.1 > > Not a mistake. BCP. My comment on the rfc is that it is simply wrong. See also: https://datatracker.ietf.org/doc/html/rfc3258 While it would be possible to have some process withdraw the route for a specific server instance when it is not available, there is considerable operational complexity involved in ensuring that this occurs reliably. Given the existing DNS failover methods, the marginal improvement in performance will not be sufficient to justify the additional complexity for most uses. which was our consensus at that time in DNSOP. I have no idea why it was forgotten. Masataka Ohta
Re: DNS pulling BGP routes?
On Thu, Oct 7, 2021 at 8:28 AM Masataka Ohta wrote: > My comment on the rfc is that it is simply wrong. > > See also: > > https://datatracker.ietf.org/doc/html/rfc3258 > While it would be > possible to have some process withdraw the route for a specific > server instance when it is not available, there is considerable > operational complexity involved in ensuring that this occurs > reliably. Given the existing DNS failover methods, the marginal > improvement in performance will not be sufficient to justify the > additional complexity for most uses. > > which was our consensus at that time in DNSOP. I have no idea > why it was forgotten. It wasn't forgotten. Folks gained a lot of experience with anycast DNS between 2002 and 2006. Not withdrawing the routes when the servers are deemed malfunctioning turned out not to be an operationally sound practice. The theory offered in 3258 was wrong. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
Bjørn Mork wrote: This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. Yes, that is a commonly seen mistake with anycast. You don't know what you're talking about. I do but you don't. https://datatracker.ietf.org/doc/html/rfc4786#section-4.4.1 Not a mistake. BCP. My comment on the rfc is that it is simply wrong. See also: https://datatracker.ietf.org/doc/html/rfc3258 While it would be possible to have some process withdraw the route for a specific server instance when it is not available, there is considerable operational complexity involved in ensuring that this occurs reliably. Given the existing DNS failover methods, the marginal improvement in performance will not be sufficient to justify the additional complexity for most uses. which was our consensus at that time in DNSOP. I have no idea why it was forgotten. Masataka Ohta
Re: DNS pulling BGP routes?
Masataka Ohta writes: > William Herrin wrote: > This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. >>> >>> Yes, that is a commonly seen mistake with anycast. >> You don't know what you're talking about. > > I do but you don't. https://datatracker.ietf.org/doc/html/rfc4786#section-4.4.1 Not a mistake. BCP. Bjørn
Re: DNS pulling BGP routes?
> > But, the reality is that it is impossible to correctly > recognize server is unavailable or to correctly withdraw > routes only when server is unavailable. > Not true at all. On Thu, Oct 7, 2021 at 9:50 AM Masataka Ohta < mo...@necom830.hpcl.titech.ac.jp> wrote: > William Herrin wrote: > > >>> This is quite common to tie an underlying service announcement to BGP > >>> announcements in an Anycast or similar environment. > >> > >> Yes, that is a commonly seen mistake with anycast. > > > > You don't know what you're talking about. > > I do but you don't. > > > If your anycast node stops > > receiving updated data and you can't reach any of the other nodes to > > check whether they're online, 99 times out of 100 this means a local > > failure of some sort. > > Yes. In case of DNS, if expiration period of a zone is passed > without successful check of the current most zone version, > unicast or anycast name servers stop responding requests for > the zone. > > But, it has nothing specifically to do with anycast. As there > are other name servers with different IP addresses, there is > no reason to withdraw routes. So? > > > You withdraw the node's announcement so that you > > don't serve bad data to the end user. > > That will only introduce new failure modes of mismatches between > server availability and server reachability and is a bad idea. > > > That's what happened here - > > Yes, facebook did wrong thing to actively withdraw routes. > > > Simply > > turning themselves off, instead of withdrawing the routes, would > > result in suboptimal performance. > > This time, facebook is saying that they could not reach their > name servers even though the servers were perfectly working. > > How much performance, do you think, facebook enjoyed? A lot > less than "suboptimal", I'm afraid. > > > And 99 times out of 100, not doing > > one or the other would cause rather than prevent an outage. > > That is a commonly seen misconception wrongly assuming that > server routes were withdrawn if and only if the server is > unavailable. > > But, the reality is that it is impossible to correctly > recognize server is unavailable or to correctly withdraw > routes only when server is unavailable. > > Masataka Ohta >
Re: DNS pulling BGP routes?
On 10/7/21 13:18, Jean St-Laurent wrote: Something public that we know now, is that it's possible to totally shut down facebook and restart it. Can we shutdown the full internet one day and see if it will restart properly without too much hack here and there? I think one thing that I learned from this Facebook outage is that the impact to steady supply of electricity to computing and networking gear under spool-up load is not a small problem to scoff at. We could shutdown the entire Internet, and power companies will probably love us. But they will hate a tad more as we reboot it. Mark.
Re: DNS pulling BGP routes?
William Herrin wrote: This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. Yes, that is a commonly seen mistake with anycast. You don't know what you're talking about. I do but you don't. If your anycast node stops receiving updated data and you can't reach any of the other nodes to check whether they're online, 99 times out of 100 this means a local failure of some sort. Yes. In case of DNS, if expiration period of a zone is passed without successful check of the current most zone version, unicast or anycast name servers stop responding requests for the zone. But, it has nothing specifically to do with anycast. As there are other name servers with different IP addresses, there is no reason to withdraw routes. So? You withdraw the node's announcement so that you don't serve bad data to the end user. That will only introduce new failure modes of mismatches between server availability and server reachability and is a bad idea. That's what happened here - Yes, facebook did wrong thing to actively withdraw routes. Simply turning themselves off, instead of withdrawing the routes, would result in suboptimal performance. This time, facebook is saying that they could not reach their name servers even though the servers were perfectly working. How much performance, do you think, facebook enjoyed? A lot less than "suboptimal", I'm afraid. > And 99 times out of 100, not doing > one or the other would cause rather than prevent an outage. That is a commonly seen misconception wrongly assuming that server routes were withdrawn if and only if the server is unavailable. But, the reality is that it is impossible to correctly recognize server is unavailable or to correctly withdraw routes only when server is unavailable. Masataka Ohta
Re: DNS pulling BGP routes?
On Wed, Oct 6, 2021 at 10:44 PM Masataka Ohta wrote: > Jared Mauch wrote: > > This is quite common to tie an underlying service announcement to BGP > > announcements in an Anycast or similar environment. > > Yes, that is a commonly seen mistake with anycast. You don't know what you're talking about. If your anycast node stops receiving updated data and you can't reach any of the other nodes to check whether they're online, 99 times out of 100 this means a local failure of some sort. You withdraw the node's announcement so that you don't serve bad data to the end user. That's what happened here - because the facebook backbone was down, the DNS servers stopped receiving updates and determined their data to be stale. Simply turning themselves off, instead of withdrawing the routes, would result in suboptimal performance. And 99 times out of 100, not doing one or the other would cause rather than prevent an outage. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
RE: DNS pulling BGP routes?
Something public that we know now, is that it's possible to totally shut down facebook and restart it. Can we shutdown the full internet one day and see if it will restart properly without too much hack here and there? Jean -Original Message- From: NANOG On Behalf Of Mark Tinka Sent: October 7, 2021 2:31 AM To: nanog@nanog.org Subject: Re: DNS pulling BGP routes? On 10/7/21 08:26, Hank Nussbacher wrote: > > Better question is why do we not see any FB netadmins on NANOG? I'm > not talking about October 2021 but rather over the past 3-5 years how > many FB techies have posted here like we see people from Google, > Cloudflare, Akamai, etc.? They are likely here, but BigContent does not really endorse talking about their operations in public fora, typically without PR/Legal OK. For those who talk about stuff, it's either stuff that is already public, publicly-known, or in their own capacity not representing their employer. Mark.
Re: DNS pulling BGP routes?
On 10/7/21 08:26, Hank Nussbacher wrote: Better question is why do we not see any FB netadmins on NANOG? I'm not talking about October 2021 but rather over the past 3-5 years how many FB techies have posted here like we see people from Google, Cloudflare, Akamai, etc.? They are likely here, but BigContent does not really endorse talking about their operations in public fora, typically without PR/Legal OK. For those who talk about stuff, it's either stuff that is already public, publicly-known, or in their own capacity not representing their employer. Mark.
Re: DNS pulling BGP routes?
On 06/10/2021 22:38, Jon Lewis wrote: But I just don't understand why this is a good idea at all. Network topology is not DNS's bailiwick so using it as a trigger to withdraw routes seems Everything I've seen posted about this (whether from Facebook directly, or others) is so vague as to what happened, that I think everyone's just making assumptions based on their own experiences or best guesses as to what really happened. Better question is why do we not see any FB netadmins on NANOG? I'm not talking about October 2021 but rather over the past 3-5 years how many FB techies have posted here like we see people from Google, Cloudflare, Akamai, etc.? -Hank
Re: DNS pulling BGP routes?
Jared Mauch wrote: This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. Yes, that is a commonly seen mistake with anycast. I would say more like Application availability caused the BGP routes to be withdrawn. Considering a failure mode that routes are not withdrawn even if application dies or routes are withdrawn even if application is alive, active withdrawal of routes is unnecessary complication with no improvement of redundancy. DNS (and other protocol's) redundancy is to have multiple unicast/anycast name server addresses. Just rely on it. Masataka Ohta
Re: DNS pulling BGP routes?
On 10/7/21 00:22, Michael Thomas wrote: But it wasn't just their DNS subnets that were pulled, I thought. I'm obviously really confused. Anycast to a DNS server makes sense that they'd pull out if they couldn't contact the backend. But I thought that almost all of their routes to the backend were pulled? That is, the DFZ was emptied of FB routes. During the outage, we kept serving traffic to Facebook in various locations. So it would seem that while a large amount of their NLRI left the DFZ, it wasn't all of them. However, what was left was not sufficient to actually keep typical services up, including their DNS. Mark.
Re: DNS pulling BGP routes?
On 10/7/21 00:37, Michael Thomas wrote: Maybe the problem here is that two things happened and the article conflated the two: the DNS infrastructure pulled its routes from the anycast address and something else pulled all of the other routes but wasn't mentioned in the article. The origin problem was some "automation thingy" that went to check capacity status around the network ahead of some planned maintenance work, and that "automation thingy" decided checking was not enough, let's just turn the whole thing off. Mark.
Re: DNS pulling BGP routes?
On Wed, 6 Oct 2021, Michael Thomas wrote: On 10/6/21 3:33 PM, Jon Lewis wrote: On Wed, 6 Oct 2021, Michael Thomas wrote: People have been anycasting DNS server IPs for years (decades?). So, no. But it wasn't just their DNS subnets that were pulled, I thought. I'm obviously really confused. Anycast to a DNS server makes sense that they'd pull out if they couldn't contact the backend. But I thought that almost all of their routes to the backend were pulled? That is, the DFZ was emptied of FB routes. Well, as someone else said, DNS wasn't the problem...it was just one of the more noticeable casualties. Whatever they did broke the network rather completely, and that took out all of their DNS, which broke lots of other things that depend on DNS. Maybe the problem here is that two things happened and the article conflated the two: the DNS infrastructure pulled its routes from the anycast address and something else pulled all of the other routes but wasn't mentioned in the article. From the engineering.fb.com article: "This was the source of yesterday’s outage. During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally." If you kill the backbone, and every site determines "my connectivity is hosed, suppress anycast propagation.", then you simultaneously have no network, and no anycast (which might otherwise propagate to transit/peers at each or at least some subset of your sites). All of your internal data and communication systems that rely on both network and working DNS suddenly don't work, so internal communications likely degraded to engineers calling or texting each other. From one of the earlier articles, it sounds like they don't have true out of band access to their routers/switches, which makes it kind of hard to fix the network, if it's no longer a network and you have no access to console or management ports. -- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: DNS pulling BGP routes?
On 10/6/21 3:33 PM, Jon Lewis wrote: On Wed, 6 Oct 2021, Michael Thomas wrote: People have been anycasting DNS server IPs for years (decades?). So, no. But it wasn't just their DNS subnets that were pulled, I thought. I'm obviously really confused. Anycast to a DNS server makes sense that they'd pull out if they couldn't contact the backend. But I thought that almost all of their routes to the backend were pulled? That is, the DFZ was emptied of FB routes. Well, as someone else said, DNS wasn't the problem...it was just one of the more noticeable casualties. Whatever they did broke the network rather completely, and that took out all of their DNS, which broke lots of other things that depend on DNS. Maybe the problem here is that two things happened and the article conflated the two: the DNS infrastructure pulled its routes from the anycast address and something else pulled all of the other routes but wasn't mentioned in the article. Mike
Re: DNS pulling BGP routes?
On Wed, 6 Oct 2021, Michael Thomas wrote: People have been anycasting DNS server IPs for years (decades?). So, no. But it wasn't just their DNS subnets that were pulled, I thought. I'm obviously really confused. Anycast to a DNS server makes sense that they'd pull out if they couldn't contact the backend. But I thought that almost all of their routes to the backend were pulled? That is, the DFZ was emptied of FB routes. Well, as someone else said, DNS wasn't the problem...it was just one of the more noticeable casualties. Whatever they did broke the network rather completely, and that took out all of their DNS, which broke lots of other things that depend on DNS. -- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: DNS pulling BGP routes?
On 10/6/21 2:58 PM, Jon Lewis wrote: On Wed, 6 Oct 2021, Michael Thomas wrote: On 10/6/21 2:33 PM, William Herrin wrote: On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas wrote: So if I understand their post correctly, their DNS servers have the ability to withdraw routes if they determine are sub-optimal (fsvo). The servers' IP addresses are anycasted. When one data center determines itself to be malfunctioning, it withdraws the routes so that users will reach a different data center that is, in theory, still functioning. Ah, I was wondering if the anycast part was the relevant bit. But doesn't it seem odd that it would be intertwined with the DNS infrastructure? People have been anycasting DNS server IPs for years (decades?). So, no. But it wasn't just their DNS subnets that were pulled, I thought. I'm obviously really confused. Anycast to a DNS server makes sense that they'd pull out if they couldn't contact the backend. But I thought that almost all of their routes to the backend were pulled? That is, the DFZ was emptied of FB routes. Mike
Re: DNS pulling BGP routes?
On Wed, 6 Oct 2021, Michael Thomas wrote: On 10/6/21 2:33 PM, William Herrin wrote: On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas wrote: So if I understand their post correctly, their DNS servers have the ability to withdraw routes if they determine are sub-optimal (fsvo). The servers' IP addresses are anycasted. When one data center determines itself to be malfunctioning, it withdraws the routes so that users will reach a different data center that is, in theory, still functioning. Ah, I was wondering if the anycast part was the relevant bit. But doesn't it seem odd that it would be intertwined with the DNS infrastructure? People have been anycasting DNS server IPs for years (decades?). So, no. -- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: DNS pulling BGP routes?
On 10/6/21 2:33 PM, William Herrin wrote: On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas wrote: So if I understand their post correctly, their DNS servers have the ability to withdraw routes if they determine are sub-optimal (fsvo). The servers' IP addresses are anycasted. When one data center determines itself to be malfunctioning, it withdraws the routes so that users will reach a different data center that is, in theory, still functioning. Ah, I was wondering if the anycast part was the relevant bit. But doesn't it seem odd that it would be intertwined with the DNS infrastructure? Mike
Re: DNS pulling BGP routes?
On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas wrote: > > So if I understand their post correctly, their DNS servers have the > ability to withdraw routes if they determine are sub-optimal (fsvo). The servers' IP addresses are anycasted. When one data center determines itself to be malfunctioning, it withdraws the routes so that users will reach a different data center that is, in theory, still functioning. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
On Wed, 6 Oct 2021, Michael Thomas wrote: So if I understand their post correctly, their DNS servers have the ability to withdraw routes if they determine are sub-optimal (fsvo). I can certainly understand for the DNS servers to not give answers they think are unreachable but there is always the problem that they may be partitioned and not the routes themselves. At a minimum, I would think they'd need some consensus protocol that says that it's broken across multiple servers. But I just don't understand why this is a good idea at all. Network topology is not DNS's bailiwick so using it as a trigger to withdraw routes seems Everything I've seen posted about this (whether from Facebook directly, or others) is so vague as to what happened, that I think everyone's just making assumptions based on their own experiences or best guesses as to what really happened. In that vein, imagine you have dozens of small sites acting as anycast origins for DNS. Each regularly does some network health tests to determine if its links to the rest of the (region|backbone|world|etc.) are working within defined paramters. If the health test fails, the site needs to be removed from anycast until the network health issue is resolved. You're big, like automating things, and feel the need for speed, so when the health test fails, rather than trigger an alarm which your NOC may or may not act on in a timely manner, the local anycast origin routes are automatically suppressed from propagating beyond the site. Just suppose you pushed out a new network health test that was guaranteed to fail in every POP...and you pushed it out to every POP. All of a sudden, your anycast routes aren't advertised anywhere. Is this what happened? I really have no clue. It sounds like something like this might have happened. Unless someone at Facebook shares an actual detailed account of what they broke, most of us will never know what really happened. -- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: DNS pulling BGP routes?
- On Oct 6, 2021, at 10:42 AM, Michael Thomas m...@mtcc.com wrote: Hi, > My guess is that their post while more clear that most doesn't go into > enough detail, but is it me or does it seem like this is a really weird > thing to do? In large environments, it's not uncommon to have DNS servers announce themselves on an anycast IP. This is also referred to as "host BGP". Basically, the host (or hypervisor) speaks BGP with the TOR. Your spines or superspines will then pick a best route or ECMP across multiple DNS servers. My guess is that Facebook took this concept a step further and anycasted their public DNS servers through their datacenters to the internet. One single config change made the DNS servers think that they were no longer functioning properly which caused them to withdraw the routes. At least, that's what I understand from the post-mortem. Thanks, Sabri
Re: DNS pulling BGP routes?
On Wed, Oct 6, 2021 at 10:45 AM Michael Thomas wrote: > So if I understand their post correctly, their DNS servers have the > ability to withdraw routes if they determine are sub-optimal (fsvo). I > can certainly understand for the DNS servers to not give answers they > think are unreachable but there is always the problem that they may be > partitioned and not the routes themselves. At a minimum, I would think > they'd need some consensus protocol that says that it's broken across > multiple servers. > > But I just don't understand why this is a good idea at all. Network > topology is not DNS's bailiwick so using it as a trigger to withdraw > routes seems really strange and fraught with unintended consequences. > Why is it a good idea to withdraw the route if it doesn't seem reachable > from the DNS server? Give answers that are reachable, sure, but to > actually make a topology decision? Yikes. And what happens to the cached > answers that still point to the supposedly dead route? They're going to > fail until the TTL expires anyway so why is it preferable withdraw the > route too? > > My guess is that their post while more clear that most doesn't go into > enough detail, but is it me or does it seem like this is a really weird > thing to do? > > Mike > Hi Mike, You're kinda thinking about this from the wrong angle. It's not that the route is withdrawn if doesn't seem reachable from the DNS server. It's that your DNS server is geolocating requests to the nearest content delivery cluster, where the CDN cluster is likely fetching content from a core datacenter elsewhere. You don't want that remote/edge CDN node to give back A records for a CDN node that is isolated from the rest of the network and can't reach the datacenter to fetch the necessary content; otherwise, you'll have clients that reach the page, can load the static elements on the page, but all the dynamic elements hang, waiting for a fetch to complete from the origin which won't ever complete. Not a very good end user experience. So, the idea is that if the edge CDN node loses connectivity to the core datacenters, the DNS servers should stop answering queries for A records with the local CDN node's address, and let a different site respond back to the client's DNS request. In particular, you really don't want the client to even send the request to the edge CDN node that's been isolated, you want to allow anycast to find the next-best edge site; so, once the DNS servers fail the "can-I-reach-my-datacenter" health check, they stop announcing the Anycast service address to the local routers; that way, they drop out of the Anycast pool, and normal Internet routing will ensure the client DNS requests are now sent to the next-nearest edge CDN cluster for resolution and retrieving data. This works fine for ensuring that one or two edge sites that get isolated due to fiber cuts don't end up pulling client requests into them, and subsequently leaving the users hanging, waiting for data that will never arrive. However, it fails big-time if *all* sites fail their "can-I-reach-the-datacenter" check simultaneously. When I was involved in the decision making on a design like this, a choice was made to have a set of "really core" sites in the middle of the network always announce the anycast prefixes, as a fallback, so even if the routing wasn't optimal to reach them, the users would still get *some* level of reply back. In this situation, that would have ensured that at least some DNS servers were reachable; but it wouldn't have fixed the "oh crap we pushed 'no router bgp' out to all the routers at the same time" type problem. But that isn't really the core of your question, so we'll just quietly push that aside for now. ^_^; Point being--it's useful and normal for edge sites that may become isolated from the rest of the network to be configured to stop announcing the Anycast service address for DNS out to local peers and transit providers at that site during the period in which they are isolated, to prevent users from being directed to CDN servers which can't fetch content from the origin servers in the datacenter. It's just generally assumed that not every site will become "isolated" at the same time like that. :) I hope this helps clear up the confusion. Thanks! Matt
Re: DNS pulling BGP routes?
Yes, it really is common to announce sink routes via bgp from destination services / proxies and to have those announcements be dynamically based on service viability. On Wed, Oct 6, 2021, 12:56 Jared Mauch wrote: > This is quite common to tie an underlying service announcement to BGP > announcements in an Anycast or similar environment. It doesn’t have to be > externally visible like this event for that to be the case. > > I would say more like Application availability caused the BGP routes to be > withdrawn. > > I know several network operators that run DNS internally (even on > raspberry pi devices) and may have OSPF or BGP announcements internally to > ensure things work well. If the process dies (crash, etc) they want to > route to the next nearest cluster. > > Of course if they all are down there’s negative outcomes. > > - Jared > > > > On Oct 6, 2021, at 1:42 PM, Michael Thomas wrote: > > > > So if I understand their post correctly, their DNS servers have the > ability to withdraw routes if they determine are sub-optimal (fsvo). I can > certainly understand for the DNS servers to not give answers they think are > unreachable but there is always the problem that they may be partitioned > and not the routes themselves. At a minimum, I would think they'd need some > consensus protocol that says that it's broken across multiple servers. > > > > But I just don't understand why this is a good idea at all. Network > topology is not DNS's bailiwick so using it as a trigger to withdraw routes > seems really strange and fraught with unintended consequences. Why is it a > good idea to withdraw the route if it doesn't seem reachable from the DNS > server? Give answers that are reachable, sure, but to actually make a > topology decision? Yikes. And what happens to the cached answers that still > point to the supposedly dead route? They're going to fail until the TTL > expires anyway so why is it preferable withdraw the route too? > > > > My guess is that their post while more clear that most doesn't go into > enough detail, but is it me or does it seem like this is a really weird > thing to do? > > > > Mike > > > > > > On 10/5/21 11:56 PM, Bjørn Mork wrote: > >> Masataka Ohta writes: > >> > >>> As long as name servers with expired zone data won't serve > >>> request from outside of facebook, whether BGP routes to the > >>> name servers are announced or not is unimportant. > >> I am not convinced this is true. You'd normally serve some semi-static > >> content, especially wrt stuff you need yourself to manage your network. > >> Removing all DNS servers at the same time is never a good idea, even in > >> the situation where you believe they are all failing. > >> > >> The problem is of course that you can't let the servers take the > >> decision to withdraw from anycast if you want to prevent this > >> catastrophe. The servers have no knowledge of the rest of the network. > >> They only know that they've lost contact with it. So they all make the > >> same stupid decision. > >> > >> But if the servers can't withdraw, then they will serve stale content if > >> the data center loses backbone access. And with a large enough network > >> then that is probably something which happens on a regular basis. > >> > >> This is a very hard problem to solve. > >> > >> Thanks a lot to facebook for making the detailed explanation available > >> to the public. I'm crossing my fingers hoping they follow up with > >> details about the solutions they come up with. The problem affects any > >> critical anycast DNS service. And it doesn't have to be as big as > >> facebook to be locally critical to an enterprise, ISP or whatever. > >> > >> > >> > >> Bjørn > >
Re: DNS pulling BGP routes?
This is quite common to tie an underlying service announcement to BGP announcements in an Anycast or similar environment. It doesn’t have to be externally visible like this event for that to be the case. I would say more like Application availability caused the BGP routes to be withdrawn. I know several network operators that run DNS internally (even on raspberry pi devices) and may have OSPF or BGP announcements internally to ensure things work well. If the process dies (crash, etc) they want to route to the next nearest cluster. Of course if they all are down there’s negative outcomes. - Jared > On Oct 6, 2021, at 1:42 PM, Michael Thomas wrote: > > So if I understand their post correctly, their DNS servers have the ability > to withdraw routes if they determine are sub-optimal (fsvo). I can certainly > understand for the DNS servers to not give answers they think are unreachable > but there is always the problem that they may be partitioned and not the > routes themselves. At a minimum, I would think they'd need some consensus > protocol that says that it's broken across multiple servers. > > But I just don't understand why this is a good idea at all. Network topology > is not DNS's bailiwick so using it as a trigger to withdraw routes seems > really strange and fraught with unintended consequences. Why is it a good > idea to withdraw the route if it doesn't seem reachable from the DNS server? > Give answers that are reachable, sure, but to actually make a topology > decision? Yikes. And what happens to the cached answers that still point to > the supposedly dead route? They're going to fail until the TTL expires anyway > so why is it preferable withdraw the route too? > > My guess is that their post while more clear that most doesn't go into enough > detail, but is it me or does it seem like this is a really weird thing to do? > > Mike > > > On 10/5/21 11:56 PM, Bjørn Mork wrote: >> Masataka Ohta writes: >> >>> As long as name servers with expired zone data won't serve >>> request from outside of facebook, whether BGP routes to the >>> name servers are announced or not is unimportant. >> I am not convinced this is true. You'd normally serve some semi-static >> content, especially wrt stuff you need yourself to manage your network. >> Removing all DNS servers at the same time is never a good idea, even in >> the situation where you believe they are all failing. >> >> The problem is of course that you can't let the servers take the >> decision to withdraw from anycast if you want to prevent this >> catastrophe. The servers have no knowledge of the rest of the network. >> They only know that they've lost contact with it. So they all make the >> same stupid decision. >> >> But if the servers can't withdraw, then they will serve stale content if >> the data center loses backbone access. And with a large enough network >> then that is probably something which happens on a regular basis. >> >> This is a very hard problem to solve. >> >> Thanks a lot to facebook for making the detailed explanation available >> to the public. I'm crossing my fingers hoping they follow up with >> details about the solutions they come up with. The problem affects any >> critical anycast DNS service. And it doesn't have to be as big as >> facebook to be locally critical to an enterprise, ISP or whatever. >> >> >> >> Bjørn
Re: DNS pulling BGP routes?
They most likely sent an update to the DNS servers for TLV DNSSEC and in oversight forgot they needed to null something's out of the workbook to not touch the BGP instances. I'd hardly believe that would be triggered by the dns server itself. -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume. > On Oct 6, 2021, at 12:45, Michael Thomas wrote: > > So if I understand their post correctly, their DNS servers have the ability > to withdraw routes if they determine are sub-optimal (fsvo). I can certainly > understand for the DNS servers to not give answers they think are unreachable > but there is always the problem that they may be partitioned and not the > routes themselves. At a minimum, I would think they'd need some consensus > protocol that says that it's broken across multiple servers. > > But I just don't understand why this is a good idea at all. Network topology > is not DNS's bailiwick so using it as a trigger to withdraw routes seems > really strange and fraught with unintended consequences. Why is it a good > idea to withdraw the route if it doesn't seem reachable from the DNS server? > Give answers that are reachable, sure, but to actually make a topology > decision? Yikes. And what happens to the cached answers that still point to > the supposedly dead route? They're going to fail until the TTL expires anyway > so why is it preferable withdraw the route too? > > My guess is that their post while more clear that most doesn't go into enough > detail, but is it me or does it seem like this is a really weird thing to do? > > Mike > > >> On 10/5/21 11:56 PM, Bjørn Mork wrote: >> Masataka Ohta writes: >> >>> As long as name servers with expired zone data won't serve >>> request from outside of facebook, whether BGP routes to the >>> name servers are announced or not is unimportant. >> I am not convinced this is true. You'd normally serve some semi-static >> content, especially wrt stuff you need yourself to manage your network. >> Removing all DNS servers at the same time is never a good idea, even in >> the situation where you believe they are all failing. >> >> The problem is of course that you can't let the servers take the >> decision to withdraw from anycast if you want to prevent this >> catastrophe. The servers have no knowledge of the rest of the network. >> They only know that they've lost contact with it. So they all make the >> same stupid decision. >> >> But if the servers can't withdraw, then they will serve stale content if >> the data center loses backbone access. And with a large enough network >> then that is probably something which happens on a regular basis. >> >> This is a very hard problem to solve. >> >> Thanks a lot to facebook for making the detailed explanation available >> to the public. I'm crossing my fingers hoping they follow up with >> details about the solutions they come up with. The problem affects any >> critical anycast DNS service. And it doesn't have to be as big as >> facebook to be locally critical to an enterprise, ISP or whatever. >> >> >> >> Bjørn
DNS pulling BGP routes?
So if I understand their post correctly, their DNS servers have the ability to withdraw routes if they determine are sub-optimal (fsvo). I can certainly understand for the DNS servers to not give answers they think are unreachable but there is always the problem that they may be partitioned and not the routes themselves. At a minimum, I would think they'd need some consensus protocol that says that it's broken across multiple servers. But I just don't understand why this is a good idea at all. Network topology is not DNS's bailiwick so using it as a trigger to withdraw routes seems really strange and fraught with unintended consequences. Why is it a good idea to withdraw the route if it doesn't seem reachable from the DNS server? Give answers that are reachable, sure, but to actually make a topology decision? Yikes. And what happens to the cached answers that still point to the supposedly dead route? They're going to fail until the TTL expires anyway so why is it preferable withdraw the route too? My guess is that their post while more clear that most doesn't go into enough detail, but is it me or does it seem like this is a really weird thing to do? Mike On 10/5/21 11:56 PM, Bjørn Mork wrote: Masataka Ohta writes: As long as name servers with expired zone data won't serve request from outside of facebook, whether BGP routes to the name servers are announced or not is unimportant. I am not convinced this is true. You'd normally serve some semi-static content, especially wrt stuff you need yourself to manage your network. Removing all DNS servers at the same time is never a good idea, even in the situation where you believe they are all failing. The problem is of course that you can't let the servers take the decision to withdraw from anycast if you want to prevent this catastrophe. The servers have no knowledge of the rest of the network. They only know that they've lost contact with it. So they all make the same stupid decision. But if the servers can't withdraw, then they will serve stale content if the data center loses backbone access. And with a large enough network then that is probably something which happens on a regular basis. This is a very hard problem to solve. Thanks a lot to facebook for making the detailed explanation available to the public. I'm crossing my fingers hoping they follow up with details about the solutions they come up with. The problem affects any critical anycast DNS service. And it doesn't have to be as big as facebook to be locally critical to an enterprise, ISP or whatever. Bjørn
