Short version: I think Paul's proposal doesn't help with routing scaling at all, and might only help with the IPv4 address space problem if the Earth's population 30 billion or more.
His proposal is intended to let a NAT box serve more hosts from a single IPv4 address, by way of providing new versions of TCP and UDP (SCTP?) with 32 bit port numbers. Paul states that his proposal would be the basis for a CEE routing scaling solution, but no CEE architecture works with IPv4 - and none work with hosts behind NAT. How can ACLs (Access Control Lists) be implemented with a CEE (Locator/Identifier Separation) architecture? Hi Paul, Thanks for your reply, in which you wrote: >> I believe that once you understood the proposals in detail, the question >> of what is being split or separated would matter to you. > > Ok, I'm not sure I'm completely happy to have my understanding > denigrated like that, when it could perhaps just be that I understand > but don't consider certain things important at this point. :) But hey... I got the impression from your earlier statement: PJ> So I recognise there are 2 different things, call them CES and PJ> CEE if you want, but I don't see the utility in argueing one PJ> that one has a split and the other does not. Anyway, doesn't PJ> matter (to me ;) ). that you weren't making any great claims about your level of understanding - and I believe that if you understood the architectures better then you would understand why I believe the distinctions between CES and CEE architectures are important. The word "separation" or "split" appears in "Core-Edge Separation" and in the "Locator/Identifier Separation" concepts (which is the basis of all "Core-Edge Elimination" architectures - but these are completely different concepts which have often been misunderstood and used too loosely. I have made the same mistakes myself. Two completely different sets of things are being "split" or "separated", so the fact that these two concepts both involve pairs of things being split or separated is of little importance. >> You can do this and so ignore what I wrote - but you haven't argued >> why anyone else should do the same or why I or anyone else should >> respect your choices. > > I didn't ignore what you said - CEE doesnt do tunneling. I'm trying to > connect my terminology with yours and check whether they match. OK - here is the complete exchange: PJ >>>>> We will note that some proposals, in order to be as PJ >>>>> transparent and invisible to end-host transport protocols PJ >>>>> as possible, use a "map and encap" approach – effectively PJ >>>>> tunneling packets over the core of the internet. RW >>>> These are all CES architectures. CEE architectures have no need PJ >>>> for tunneling. PJ >>> I think I'll stick with "map-and-encap" and "rewriting". I have PJ >>> to say, I sometimes wish this WG would try adopt labels that are PJ >>> semi-evocative of their meaning where possible, rather than all PJ >>> these acronyms. Very confusing! :) RW >> You can do this and so ignore what I wrote - but you RW >> haven't argued why anyone else should do the same or why I or RW >> anyone else should respect your choices. PJ > I didn't ignore what you said - CEE doesnt do tunneling. I'm PJ > trying to connect my terminology with yours and check whether PJ > they match. You equated "CES" with "map-and-encap" - but "map-and-encap" (encapsulated tunneling) is not the same thing as tunneling. Ivip has two "Modified Header Forwarding" approaches by with ITRs tunnel traffic packets to ETRs, without encapsulation. One approach is for IPv4 and the other for IPv6. Both require upgrades to all DFZ and some other routers - which in the long-term can be achieved without significant cost. Also, Six-One Router, which I think has more in common with the CES architectures, uses address rewriting. Only some CEE architectures use "rewriting" - GSE and GLI-Split are two which do. ILNP Name Based Sockets and I think quite a few other CEE architectures do not involve address rewriting. I never suggested you equate "CES" with "map-and-encap", or "CEE" with "rewriting" - and you didn't explain why you made these connections. >> No - you can use "Core-Edge Separation" and "Core-Edge Elimination". I >> am trying to be concise. > > I don't really understand those terms tbh. I have had a glance through > the RRG wiki (the "terminology" page particularly), and your > http://www.firstpr.com.au/ip/ page, but I don't find an explanation. The above page of mine is just a pointer to some others. In my message I suggested you read: CES & CEE are completely different (graphs) http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html and the closely related: Today's "IP addr. = ID = Loc" naming model should be retained http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html I had a go at explaining the difference in my message, but it is better to point you to the full, prior, explanations than to repeat things on the mailing list. > The general architecture I'm familiar with that doesn't do tunneling > instead uses address rewriting of some form. So I tend to call it > "rewriting". > > I'll try get accustomed to your terminology, but perhaps till then you > can try tolerate mine. :) OK! >> because there's no way the world will alter all its host stacks or >> (for most CEE architectures) rewrite its applications *completely* > > That is a significant concern alright. The only reason such a suggestion > isn't complete crack is cause: > > - The world basically already did this for IPv6, so we know what it > involves. Only a subset of applications work for IPv6 as far as I know - a pretty small subset, I think. > - The apps that have updated should, by and large, now be using > address family abstracting interafaces, making future changes > slightly less painful. I agree, but this is only for the subset of applications which have been upgraded in this manner, and there are many applications which by the nature of their protocols work much more directly with IPv4 addresses. I think that many protocols can't directly be made to work with the CEE arrangement of "Locator/Identifier Separation", since the protocols themselves assume that the IP address is stable per host, being both an Identifier and a Locator. For instance, how can you do ACLs with the "Locator/Identifier Separation" naming model? Today, a router or an application can allow or disallow access and communication with hosts based on straightforward ranges of IP addresses. With the "Locator/Identifier Separation" naming model, IP addresses can't be used for this purpose. So the routers or software would need to establish that the incoming packet's purported Identifier was in fact valid, by looking up the Locators for this Identifier and checking that the Locator which is in the packet is within the allowed set. Only then, could the packet be allowed or not through a router, or responded to by an application. In the case of CEE architectures such as Name Based Sockets, where the FQDN plays the role of Identifier (or perhaps some tricky combination of FQDN and Locators - I am awaiting guidance from Christian on this) the Identifier is not numeric, so it gets trickier to group ranges of them as one can with IP addresses, for the purposes of creating ACLs. >>>> CEE architectures can only provide substantial benefits to adoptors >>>> and only provide real scaling benefits, when all, or almost all, >>>> networks adopt the new architecture. This means moving everyone to >>>> IPv6 and altering all host stacks to implement the particular CEE >>>> architecture's alteration of IPv6. > >>> I'd disagree with this. If you read the blog entry I'm outlining a >>> scenario where the immediate problem to be solved is pressure on NAT >>> (not multi-homing), leading to hacks being deployed which benefit both >>> customer and ISP together. These hacks then can be later, slowly, >>> extended to things like multi-homing. >> >> I don't see how the above paragraph relates to you disagreeing with the >> paragraph of mine before it. > >> What you are proposing is neither a CEE nor a CES architecture. You are >> proposing to extend the port range of IPv4 TCP and perhaps UDP >> protocols, as a means of enabling NAT boxes to serve more hosts from a >> single IPv4 address. > > Correct, and so provide a basis for a CEE architecture to be built on it. Another distinguishing thing about CEE architectures, as far as I know, is that they don't work for hosts behind NAT. Can you explain how a CEE architecture could work for IPv4 with your extended TCP and UDP etc. port numbering system? > The NAT shortage is the initial pressing problem that gets the basic > option in the field and deployed widely. I don't think it is a pressing problem yet - though one day it may be. Can you show how many IPv4 addresses are currently used for single customer services, such as DSL, and why this is such a large and growing number that ISPs will be so unable to obtain address space at a given price per IPv4 address that it will be more profitable to sell customers a less useful service behind NAT, and so without each customer being able to do their choice of port-forwarding? For mobile hand-held devices, I agree it makes more sense to use NAT to conserve space, since I think people are less likely to want to run port-forwarding applications on these hosts, or to run their own NAT box and private network of hosts. However, I understand that for many home and SOHO users, port forwarding, using generally fixed ports, is an important part of how they use their Internet service today. Take a look at this list of application programs which apparently use port-forwarding: http://www.portforward.com/cports.htm Any such app which requires a fixed port will be difficult or impossible to use if multiple customers are being a single NAT box, as you are suggesting. > Further protocols, offering further facilities, such as > core-separated multi-homing, can then be layered on top. How can a customer behind a NAT box be multihomed? >> This doesn't solve the routing scaling problem in any way. It is an >> attempt to cope with a situation where there is such a shortage of >> IPv4 space that there's not enough to run NAT boxes with whatever >> limitations they have from the current 16 bit TCP/UDP port numbering >> system. > > I think you must have stopped reading that blog post about half way > through, or you skimmed it :). I read the whole thing. >> I don't understand your reasoning - and I can't see where you have >> explained your reasoning. > > I was proposing something more than just NAT. Specifically, I was > describing a path of least-resistance toward a "core/edge separated > internet, using rewriting" (CEE, right?), No!!! I am trying to correct your too-loose use of terminology - in part so other people don't think that this usage is meaningful or useful - and you continue to use it so loosely as to be confusing and/or meaningless. "Core-Edge Separation" (CES) involves separating a subset of the global unicast address space to be "edge", leaving the rest as "core". It is a separation of addresses into two subsets. CES architectures generally do not involve rewriting. Only Six/One Router uses rewriting, and while I think it most resembles a CES architecture, there is an important difference from the other CES architectures - the fact that it requires a split horizon DNS to work with non-upgraded hosts. CES, "rewriting" and CEE are three completely separate concepts, so your sentence above makes no sense. You haven't explained how your proposal to extend the ability of IPv4 NAT boxes to handle more hosts from a single IPv4 address could be used in any way (with CEE, CES or any other approaches) to solve the routing scaling problem. > where the first step on that > iteration of hacks is "make NAT work for higher number of hosts". > Further, this proposal remains backward compatible with the existing > IPv4 network, to a greater degree than other CEE proposals at least. Your proposal involves using IPv4 header options. These are theoretically compatible with IPv4, but they can't be relied upon in across the DFZ. If all routers were updated, then your system could proceed, with upgraded host stacks and applications which were backwards compatible with ordinary IPv4. On their own, I agree, your changes would be more backwards compatible with IPv4 than any CEE architecture. No CEE proposal can be used for scalable routing with IPv4 because each multihomed end-user network requires at least double the address space it actually uses. (This is not true of CES architectures, which use address space 100% efficiently, except Six/One Router which I am thinking is much less like a CES architecture than all the others - a multihomed network with two ISPs needs three times the global unicast space.) > I think perhaps you may not have read the latter parts of that blog > post. (My writing style is dry, dull and sucks generally, so I can > understand you'd have fallen asleep a paragraph or two in! ;) ). I read it and I have just reread it. You haven't explained how a host behind NAT could be multihomed, or how any CES or CEE architecture could be used with your proposal to provide portability, multihoming or inbound TE for hosts behind NAT. >> establishing communications - with more management packets flowing back >> and forth etc. Arguments against doing this are in: >> >> Today's "IP addr. = ID = Loc" naming model should be retained >> http://www.ietf.org/mail-archive/web/rrg/current/msg05864.html > > The signalling overhead of an end-host<->end-host multihoming proposal > is a concern. However, you have the same problem with the map-encap > proposals - the signalling has to be done somewhere. You can do it on x > hosts, or x/y middle-boxes. My message 05864 includes arguments why it is much better to do it on middle-boxes such as ITRs. CEE (Locator/Identifier Separation) involves *all* hosts doing their own routing and addressing management, though some architectures do some of the work in routers. This is particularly a burden for mobile hosts on frequently slow, expensive and less than 100% reliable 3G etc. wireless links. > It seems there's a compromise we have to pick between: > > - carrying information about peripheral^Wedge networks in the core > routing protocol > > - carrying information about peripheral networks in packets *over* > the core routing protocol. > > To my feeble mind, I don't see any way for protocols alone to *reduce* > the amount of signalling required, for a given amount of information > about edge networks. Sure - the idea of CES is to build a streamlined new mechanism for this. CEE offloads the work onto hosts, or sometimes hosts helped by address-rewriting routers. I think CES is superior to CEE for reasons including: Only CES can support IPv4. CES allows the continued use of today's two-layer IP naming model, which some argue is "architecturally impure" - but which I think is better and faster than if we all had to rely on hosts doing their own management of routing and addressing (perhaps with some help from routes) via the CEE arrangement of "Locator/Identifier Separation". CES brings immediate benefits to adopters and progressive routing scaling benefits, while CEE only provides both sets of benefits once all networks and hosts have been upgraded. CES does not require host stack or application changes - while most CEE architectures (GSE and GLI-Split are the exceptions) require stack and application changes to all hosts in order in order for all communications to have the portability, multihoming and inbound TE benefits. > The only way to reduce that information is to organise the edge networks > some way, e.g. through an administratively imposed hierarchy > (aggretation by geography, IX, whatever), or by some clever dynamic > organisation. I see that as an orthogonal matter to the protocol > question, and one best left at the door to this WG. All the current proposals involve some kind of new way of organising routing and addressing for the end-user networks which require portability, multihoming and inbound TE. This is because no current proposal aims to fix the scaling problem by re-engineering BGP or the DFZ in general to cope with ten million or more PI prefixes. >> Please take a look at the graphs here: >> >> CES & CEE are completely different (graphs) >> http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html > > Those graphs are your opinion, rather than based on data :). If you're > right, then CES will see deployment. We don't need to have a speculative > argument about it here. None of us have data about the future. These are my opinions - but I explain the reasons I believe these things so other people can understand why I think this, and so that anyone who disagrees can point out where they think my reasoning is mistaken. You haven't pointed out why you think my "benefit vs. adoption" curves are incorrect. >> If you disagree with this, then please write why. > > If would move access customers to NAT, and use the reclaimed addresses > for services that need it. If I were part of a community of ISPs I would > start looking at cross-organisation reclaiming of ineffectively used > addresses. A free market in IPv4 addresses may well develop to allow > organisations to put a price on the cost of renumbering Vs the cost of > not having public IP space, and so facilitate reclaiming. I agree that there will be a market for IPv4 space. IPv4 space will be increasingly valuable and so there are pressures to use it more efficiently. There is bound to be money changing hands in this process - with the payers encouraging the address suppliers to use less space overall than they did, either by actually using less addresses, and/or by compacting their used addresses into smaller chunks of space, which they retain, so they can sell, return or whatever some of the space they currently advertise. > If I were an ISP, I would do all these things well before deploying CES. Core-Edge Separation is primarily something which end-user networks want to adopt - because it will be a less expensive, and most likely the only, way they can get portability, multihoming and inbound TE. ISPs will probably only want to adopt CES because of their customers. The ISP's own hosts don't need CES, because they are already on the ISP's own prefixes, which are not part of the routing scaling problem. > As I wrote earlier, it is quite plausible that such address space > recyling will serve the needs of the IPv4 internet for quite a while - > at least long enough for the NAT problem to become a pressing one to solve. I think the coming era of IPv4 address compaction will help ISPs to obtain space so they can continue to use it one IPv4 per DSL (etc.) customer. I think CES will enable more and more end-user networks to use IPv4 space more efficiently, since they can get exactly the number of addresses they need (1, 2, 3, 4, 5 etc. with Ivip, or 1, 2, 4, 8 etc. with the other CES architectures) rather than having to obtain space in chunks of 256, 512, 1024 IPv4 addresses as they do now. I guess when there are 1.5 or 2 billion DSL etc. customers each getting their own IPv4 address, like they do today, then things will get very tight and there will be an increased desire to put at least some of these customers behind NAT. Many of them will be OK with this - if they are not using port-forwarding applications. So I can envisage ISPs offering two classes of service - behind NAT or not. Your proposal only helps to the extent that it alters the number of hosts which can be served from a single IPv4 address. There's no fixed number for this, but let's say it is 50 hosts, and the average customer has 5 hosts at their home, office or whatever. (Lets leave aside whether they run their own NAT box in the home or office, and so are doing double NAT, or whether each host gets its own address from the ISP's NAT box.) So with NAT as it currently exists, for a subset of non-mobile customers and probably many or most mobile customers, they can already be served with NAT and so 10 to 50 or whatever customers can use a single IPv4 address. I think your proposal would only help once we ran out of IPv4 addresses even when using many of them to serve 10 to 50 customers per IP address. That is not going to happen as long as we have less then 10 billion people on the planet. So as far as I can see, your proposal wouldn't help unless we had 5 or 10 times the current population, all using IPv4. I think your proposal is about helping to get more use from a limited number of IPv4 addresses. I don't think it helps with the routing scaling problem, since I don't think it provides any new approach to portability, multihoming or inbound TE. > And as I wrote in blog post, solving the NAT problem can provide the > immediate benefit to get an extension widely deployed that can then be > used as the basis for a CEE architecture. There's no CEE architecture which works with IPv4 - or with hosts behind NAT - so I don't see how your proposal could be used as an improved basis for CEE. > Small incremental steps. > >> CES involves adding some things to the network. This is easier than >> changing all the apps. > > Experience with IPv6 suggests otherwise. > > It's 15 odd years now, and the inter-network still does not generally > support IPv6. Despite the lack of IPv6 network service, the end-host > software *did*, to a significant degree, get upgraded. Here is what is required to make IPv6 usable for most people to the extent that they don't need IPv4 any more. Until then - as long as IPv4 is the only way to reach all the hosts they want to reach - few people are going to go to much trouble to use IPv6: 1 - There needs to be an IPv6 network and IPv6 services to customers. 2 - Customer's CPE and host stacks need to cope with IPv6 in a dual-stack or perhaps IPv6-only setting without any fuss. 3 - Their applications need to be upgraded to IPv6. 4 - The hosts they want to communicate with must be on IPv6. 1 and 2 have only partially happened. 3 has occurred partially for some major apps, but the vast majority of applications people use are not ready for IPv6. 4 has hardly occurred at all. Ben Stasiewicz's research last year: http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000708.html indicates that less than 0.1% of major websites have an IPv6 server: The list of IPv6 web servers was derived from the Alexa Top 1 Million Websites list [6]. Of these 1 million domains, only 627 had globally routable IPv6 addresses that could be connected to on port 80 (sad eh?). The problems faced by IPv6 are closely analogous to those faced by CEE proposals, and yours - the benefits do not accrue substantially to any adopters until almost everyone has adopted. Consequently, very few people try to adopt it - and the situation persists indefinitely. CES architectures do not have these problems, as I explained and provided graphs for in (msg05865). > Granted, CES doesn't require the fork-lift upgrade that IPv6 does. But > still, it assumes that ISPs will choose to add relatively complex CES > control plane - which implies administrative overheads, which implies > increased staffing costs - over the tried, tested and simple NAT box + > reclaiming address space. They can already use NAT to sell a second-rate behind-NAT IPv4 service. As long as people are prepared to pay more for the current service and/or their helpdesk costs of explaining to people on the NATed service why they can't do things which work fine for their next-door neighbour and/or the cost of buying more space is less than the cost of buying and running the NAT boxes, while charging less money per customer, then there won't be much uptake of behind-NAT IPv4 services. I think there's no need for your major upgrade to host stacks and applications just to allow a NAT box to serve more hosts than is currently possible. > Anyway, we can't resolve this argument without data. The market will > provide that in time. Our job is to discuss things and choose the best approach to develop - so no-one in the future wastes time and money pursuing something which we can foresee now is not going to work. I have argued why I believe no CEE proposal is suitable. That leaves CES proposals. TIDR doesn't solve the scaling problem and IRON-RANGER has not been clearly enough specified for anyone to estimate how well it would scale. LISP has a bunch of problems: http://www.ietf.org/mail-archive/web/rrg/current/msg05728.html and that leaves my Ivip proposal: http://www.firstpr.com.au/ip/ivip/ The existing critique of Ivip gives it a pretty good rap: http://tools.ietf.org/html/draft-irtf-rrg-recommendation-04#section-4.2 and it would be really good for people to take a greater interest in criticising Ivip if they think it is not the best solution to the routing scaling problem. >> But increasing the number of hosts a NAT box can handle from a single >> IPv4 address has nothing to do with the routing scaling problem, or >> multihoming, portability, inbound TE or mobility. > > I respectfully submit you have considered only the initial part of my > argument :) > > I will of course reread yours to see what of it I have missed. OK - but you haven't explained how your proposal helps on its own, or how it could be used with any other solution. >> Unfortunately it couldn't become even a little popular since it is >> impractical to use it through the DFZ, or probably with most routers. > > I don't buy that. The same thing applied to IPv6 at one stage, indeed it > was worse with IPv6 for the DFZ couldn't forward IPv6 /at all/. Over a long period of time, the DFZ routers could be upgraded with little cost, as more and more of them have entirely firmware-controlled FIB functions, and those which don't are retired. I don't rule this out as a possibility - but if we were going to upgrade all DFZ routers for new types of IPv4 option headers, or new types of headers, then I think there are more important things to do regarding routing scaling than your proposal which just enables a NAT box to handle more hosts per IPv4 address, and then only to the extent that the hosts have their stacks and applications upgraded. > IP options being slow would be resolved within about 1 life-cycle of > routers (at least, their line cards) on the DFZ. I don't have anything > but anecdotal evidence as to what that is, my best guess is circa 5 years. I don't have any better information. To the extent it is true, I would argue for upgrades to support something which I consider more directly useful for routing scaling than your current proposal. >> Its not just slow - it involves such packet loss rates that >> communication would be impractical. Even being slow would be enough >> to ensure no-one wants it. > > They found significant problems with about 15% to 21% of the paths they > tried. So 80% odd of the 210 paths they tried were fine. Plus we need > further studies to see how peculiar these loss rates were to the set of > paths they measured. > > It's a problem, sure, but it doesn't quite seem as bad as you've stated > it. We face similarish connectivity problems trying to recycle addresses > when address ranges have been listed in bogon space, or with low address > ranges that historically have not been used and which have ended up > being used privately in places. E.g. what proportion of paths have PMTU > problems (which would affect CES - I have to read your link still). The techniques Fred and I developed are different, but both can cope with there being PMTU problems between ITRs and ETRs which do not result in PTBs being sent to the ITR. It is complex, and ideally we wouldn't have to go to this trouble, but both approaches could work, I think. Neither can cope with PTBs sent by the ITR being dropped or ignored by the sending host - but that would be a problem caused by the sending host or the sending host's network which needs to be fixed there, since there is no other solution. As long as the ITR is in the sending host's (which is not necessarily the case, at least initially, with LISP or Ivip) network, hopefully this will not much of a problem. > There isn't any perfect solution, it seems. This is about choosing > between problems. OK. >> But until such upgrades are done, there's no way anyone can reliably >> communicate across the IPv4 DFZ with a new kind of protocol, or with >> IPv4 packets with option headers. > > It would take a similar time-scale to start to get IP extensions > deployed in end-hosts. Unfortunately the benefit vs adoption curves for your proposal resemble those of CEE - substantial benefits only when everyone has adopted it - so I can't see how you would ever get to this stage, especially since you need to convince all application programmers to substantially rewrite their software and change their protocols. This is never going to happen, I am sure. >> Once we can contemplate such upgrades to DFZ and other routers, then we >> could do a CES architecture (Ivip) without encapsulation: > >> http://tools.ietf.org/html/draft-whittle-ivip-etr-addr-forw-00 >> >> As you note in the "pjakma said" addition to your page: >> >> http://pjakma.wordpress.com/2010/02/12/making-the-internet-scale-through-nat/ >> >> your proposal would also depend on upgrading applications. I agree >> with your assessment of this aspect of your proposal "This is the most >> obviously crack-full part of it all."! > > Indeed. > > But again, it can be deployed in stages, and each stage can bring some > immediate benefits, while enabling the next stage. No single stage > solves all the problems, but after a number of iterations more and more > problems can be solved. But please consult the curves for CEE, since the curves for your proposal would be at least as uninviting: CES & CEE are completely different (graphs) http://www.ietf.org/mail-archive/web/rrg/current/msg05865.html ^ B | * Core-Edge Elimination (CEE) e | .* n | ..* . Proportion of communications e | ...* with the benefits. f | ....* i | .....* * Actual total benefits generally t | ......* only arise when all, or almost | .......* all, communications have the |........* benefits. 0---------> Effort ~= level of adoption ^ B | * Core-Edge Elimination (CEE) e | * Routing scaling benefits. n | * e | * f | * i | * t | * | * | * 0---------> Effort ~= level of adoption > I'm not saying this is /better/ than map-encap solutions, just saying > that if for some reason map-encap doesn't gain traction either that > there is still a possible engineering path to a rewrite-based n-layer > routing architecture. > > Just saying.. OK - and I am arguing in detail multiple reasons why a good CES architecture will be superior to your approach or to any CEE architecture. Of course, if Ivip is introduced and fails miserably . . . then maybe you will be proven to be right. But you haven't explained why it would do so. - Robin _______________________________________________ rrg mailing list rrg@irtf.org http://www.irtf.org/mailman/listinfo/rrg