Long message ahead:
Discussing Tom's "Liquidity Systems" view of the
Internet's routing and addressing system.
I haven't yet found any insights from it which
help with scalable routing.
Discussion of how with LISP, APT, Ivip or TRRP
more and more address space will be needed by the
"MAB" companies who rent the space out as EID
prefixes micronets or whatever is the name for the
new kind of scalable address space for end-user
networks.
It follows that a greater proportion of space will
be needed by these companies, and a lesser proportion
by ISPs. I give a contrived example of an ISP with a
single /24 being able to serve 10,000 end-user
networks which between them have 100,000 IPv4
addresses of EID / micronet etc. space.
Also, how competition between these MAB companies
will help drive a healthy market for obtaining
space which is less profitably utilised than will
be possible with a good scalable routing solution.
Hi Tom,
Thanks for this:
> Thanks for taking (what must have been considerable) time to reply in
> detail.
>
> The above two modifications do, indeed, adequately reflect the concerns
> I expressed.
It did take a while to work it out. Originally I wasn't going to add
anything, but I am happy your concerns and discussion prompted me to
add these two paragraphs.
>> I want the list of constraints to be a terse description of things we
>> all (ideally) agree are true. The focus is on what is needed for
>> end-user networks and their ISPs to adopt the solution.
>>
>> As a statement of fact, this text does not assume that there is a
>> solution which will meet all these constraints. I think there will
>> be solutions which meet them all in IPv4 - but I agree that during '
>> adoption and in the decades to come, no matter how ubiquitously the
>> scalable routing solution is adopted and how well it works, the
>> address space shortage is likely to limit the IPv4 Internet's
>> capacity to provide multihoming etc. for all end-user networks which
>> want it.
>
> Of course I agree on this point, but this particular goal was never
> explicitly stated, nor implied, nor even contemplated by me.
> The idea that merely "wanting it" -- i.e., full access to all of the
> 'rights, privileges, and responsibilities' that are enjoyed by direct
> participants in the Internet's routing subsystem -- is or should be by
> itself sufficient for any institution to lay claim to a share of that
> system's finite carrying capacity -- fundamentally contradicts the
> findings of my own personal experience and research (maybe common sense
> too, if I presumed to know what that was).
With Ivip at least, and I think also for LISP, APT and TRRP, wanting
/ needing (who is to say which is true?) multihoming, TE and
portability and then getting it via Ivip certainly does not involve:
full access to all of the 'rights, privileges, and
responsibilities' that are enjoyed by direct participants
in the Internet's routing subsystem
If a network either does not exist, or exists on PA space, then it
has a price to pay to obtain some Ivip-mapped "SPI" (Scalable PI)
address space, whether it be a single IPv4 address or thousands of
such addresses.
If the network already exists, it also need to be renumbered to the
new SPI address range.
There will be ongoing rental fees for the SPI space, paid to the MAB
company who "owns" or at least has rights to, the MAB (Mapped Address
Block) which is advertised in BGP and which contains the User Address
Blocks (UAB) of many other end-user networks. How each end-user
network splits up the one or more IP addresses in its UAB into
micronets (1 or more contiguous IP addresses with a single mapping)
is up to the end-user network. They will typically pay per mapping
change - to the MAB company, who will typically pay a Root Update
Authorisation System (RUAS) company to send the mapping change to all
the world's full database Query Servers.
The MAB company will also directly or via some other company run
OITRDs (Open ITRs in the DFZ) which are necessary for handling
packets sent to SPI addresses in its MABs from hosts in networks
without ITRs. Since traffic volumes for OITRDs would vary
enormously, the end-user network will typically need to pay in
proportion to how much traffic these OITRDs handle for their micronets.
So merely "wanting" something won't make it happen. The end-user
network needs to be willing to pay for all this. It won't be very
expensive when compared to the total costs of getting an AS, PI space
etc.
While the end-user network gets its own set of global unicast IP
addresses (SPI addresses), which it can use at any ISP which has an
ETR, this doesn't to me constitute what you wrote and which I quoted
above.
These end-user networks don't need to know anything about BGP etc. or
be responsible for running DFZ routers.
This will be a mass-market service, competitively provided by
multiple MAB companies all round the world. These MAB companies may
be ISPs - but they need not be, because there is no need for physical
infrastructure other than someone to run their OITRDs in various
locations around the Net.
Before Ivip etc. for an end-user network to gain global unicast
addresses it either had to rent some PA space from an ISP or get an
AS, and gain a PI prefix (and then have ISPs advertise this prefix).
Once Ivip or similar is available, they will have a third, highly
attractive, option: rent some SPI address space in a MAB and use it
at any ISP which has an ETR.
Imagine a scenario in which an ISP-M sets up shop purely for the
purpose of serving customers with SPI addresses. ISP-M may or may
not have MABs of its own to rent space in. I will assume it doesn't.
Its customers will already have rented their SPI space from one of
many MAB companies.
In Ivip, the ETR is typically in the ISP network. (Except for with
TTR mobility, where the ETR is the Translating Tunnel Router, ideally
located near the access network.) Let's say each ETR can handle 100
end-user networks. Let's say ISP-M has 10,000 separate end-user
network customers, each with an average of 10 IP4 addresses of SPI space.
The entire ISP-M could be run nicely on a /24. It needs 100 global
unicast ordinary ("RLOC" in LISP terms) IP addresses for its ETRs.
It can use the same addresses for ITRs too - and ITRs will frequently
be in the end-user networks anyway, including in some sending hosts.
ISP-M also needs addresses for query servers etc. So ISP-M could be
serving 100,000 IP addresses of SPI space via 256 "RLOC" addresses.
If all those end-user networks were multihomed to another ISP, then
in theory they could all be served by two similar ISPs ISP-M and
ISP-N, each with only a /24 of space themselves.
LISP as currently conceived is a little differently. The LISP
designers assume that the ETRs will be at the end-user network. So
each such network needs a single "RLOC" address from each of its ISPs
for the ETR which links to each ISP. Therefore, ISP-M would need at
least 10,000 RLOC addresses and so would ISP-N. LISP-ALT, as
currently conceived, needs the ETRs to be at the end-user network for
at least two reasons. Firstly, reachability of the ETRs from any ITR
is taken as meaning reachability of the end-user network. (This is
not the case for Ivip.) Secondly, the ETRs need to act as a unified
system, so they need to be owned and operated by the end-user
network, and robustly linked together no matter what outages etc. occur.
While LISP is less efficient with RLOC space than Ivip, the point
remains that an ISP serving SPI customers does not need to have much
space at all. One or more end-user network with 64K addresses can in
principle work fine from a single ETR.
This significantly changes the address requirements for ISPs. There
still is an address requirement for this SPI space - but it is
handled by MAB companies gaining rights to large slabs of address
space, and then making money by renting that space out.
For an end-user network in any one physical location, they may only
be able to choose between one or a small number of ISPs, since only
those ISPs have physical infrastructure to their site. However, the
end-user network's SPI address can come from any MAB company in the
world.
More on the competition effects of this below.
> More precisely stated, the implications of my recent work suggests that
> the very nature of the constraints that dictate the requirement for
> "incremental deployability" are *the same* structural constraints that
> absolutely dictate that any/every conceivable current and future routing
> system that could ever be adopted and deployed in this manner will --
> inevitably, necessarily -- embody some set of features that ultimately
> make all such systems vulnerable to some version of *the same* systemic
> risks to which the current system is vulnerable, namely:
IOW: a good scalable routing solution can improve scalability within
IPv4, but as far as we know, can't help any one network or all uses,
bust out of IPv4 dependence to use some other system, such as IPv6.
To achieve such an outcome may well be impossible.
> -- "inflation" (c.f., generalized/cumulative/direct impact of individual
> demands on routing system carrying capacity that are borne by all/other
> participants in that routing system)
These financial system analogies strike me as rather loose.
End-user networks adopting SPI space place no burden on the BGP
control plane whatsoever. There is a burden, from each MAB, but that
is not related to how many end-user networks share that MAB, how many
micronets they split the MAB into, or how frequently they change the
mapping for those micronets.
With Ivip, mapping changes are paid for by the end-user network - so
they are not an unreasonable burden on the RUAS. (This is not a
perfect system, but it goes a long way to paying for the cost the
mapping changes impose on all parties which need to carry it.)
Indeed, these charges are a source of revenue and MAB companies would
compete in part by offering lower fees which still enabled them to
run a profitable business.
Hopefully all this will be clear if you read:
http://www.firstpr.com.au/ip/ivip/Ivip-summary.pdf and
Ivip business models: fast push & OITRDs 2008-04-18
http://www.ietf.org/mail-archive/web/rrg/current/msg02100.html
> -- "deflation" (c.f., generalized/cumulative impact of non-availability
> of routing system capacity and/or address resources, which manifests as
> diminished growth potential for existing participants and a progressive
> barrier to the emergence of potential new participants)
IOW: Shortage of address space. This will still happen with Ivip or
whatever, but it will happen later due to the ability to slice and
dice the address space into small increments which can be used
efficiently due to being very useful to many end-user networks and
profitable for MAB companies when rented at a cost end-user networks
are happy to pay.
> -- "stagnation" (c.f., generalized/cumulative impact of non-availability
> of "standard"/"DFZ" routing system capacity and/or "standard"/uniform
> address resources, which manifests as a progressive degradation of and
> loss of confidence in the consistency/predictability of precisely those
> "network effects" that make a shared, inter-domain routing system so
> valuable to its direct participants, as well as to indirect stakeholders).
IOW: When existing and new companies find it harder than is ideal for
them to get address space, due to shortages, hoarding, unfair
distribution systems etc.
> Another way of putting this is that the specific problems that RRG is
> trying to solve are actually just specific manifestations of more
> general risks -- risks that are *not* primarily or exclusively an
> artifact of current routing and addressing technology.
I disagree.
The general problems of IPv4s address space being finite and less
than would be ideal are a totally separate matter from the problem of
unscalable growth in the burden of work on DFZ routers, together with
the inability of those so burdened (DFZ router operators) to recover
costs from those who impose the burden. This growth is primarily due
to the fact that at present, without a scalable routing system, the
only way an end-user network can get multihoming, TE and portability
is to get its own PI prefix (one or more /24s - is a /22 a typical
allocation?) and then to advertise this as one or more prefixes (/24,
/23 etc.) in the global BGP system.
A core-edge separation scheme such as LISP, APT, Ivip or TRRP
(Six/One Router too, but just for IPv6) would change all that, since
there would be this new kind of SPI (my term, not used in LISP etc.)
space which would suit the needs of large and small end-user networks
much better than getting their own AS, PI prefixes etc.
Maybe there is some aloof framework from which you can see the
routing scaling problem as a specific instance of some more
generalised set of circumstances. To me it is a specific set of
circumstances which occurs due to BGP's finite ability to cope with
large numbers of prefixes and frequent updates to those prefixes. It
is the same in principle for IPv4 or IPv6, but there is no problem
yet for IPv6, and the IPv4 scaling problem happens to be occurring at
a time when fresh IPv4 space is running out.
It so happens that a routing scaling solution will, in my view,
enable much better utilisation of IPv4 address space. How much time
that will buy us, I am not sure. It depends a lot on population
growth - which can't be assumed given global warming's effects - on
Internet usage, and on whether cell-phone networks want to use a
global unicast IPv4 address for each device, or whether they will use
IPv4 NAT or IPv6 instead.
> Rather, these
> risks are a natural, organic, and (probably) largely unavoidable feature
> of *all* liquidity mechanisms -- i.e., all "technically mediated
> exchange systems." They are unavoidable because a necessary condition of
> possibility for the existence/emergence of such mediated exchange
> systems is the prior existence of a population of diverse, independent,
> often competing economic agents (individuals, institutions, etc.),
> members of which might often enjoy opportunities to productively
> interact with one another, but for the existence of various
> "transactional frictions" (foregoing details on this for now).
> "Liquidity system" and/or "technical medium of exchange" are the
> interchangeable terms that I've adopted [1] to describe the thing that
> emerges in some (not all) situations, which is functionally
> distinguished by its capacity to reduce if not eliminate those frictions
> for all participants who voluntarily adopt it -- thereby becoming
> participating members in that particular liquidity system. The intrinsic
> friction-reducing benefits of participation (growth, profit, etc.)
> attracts and retains participants within that system, which
> creates/strengthens incentives for system participants to interact with
> each other ever more intensively (producing as a result the devision of
> labor, specialization, etc.). However, those same benefits also have the
> organic effect of intensifying competition between system participants,
> which eventually results in their becoming aware of the liquidity
> mechanism itself, and its possible uses as a strategic,
> (anti)competitive tool.
>
> I'll spare the list the rest of the evolving story. Those interested can
> find more of it is on my personal website (links below).
I think there are some interesting parallels between IP addressing
and currencies, but I am not sure that they have so far provided me
with any important insights which help me solve scalable routing or
the larger problem of transitioning to a system without oppressive
address space limits.
> In any event, I wholeheartedly share RRG's ambition to contribute to the
> alleviation these problems. The story above explains why systems like
> ours need liquidity mechanisms, and what benefits they confer -- and
> risks they impose -- on those systems that are lucky enough to have one.
> However, that "need" does not entail that it will be fulfilled.
I understand this as the purported "need" to solve the IPv4
addressing limitation problem does not mean that this will be achieved.
I am sure humanity will cope with being boxed into IPv4 indefinitely.
It is not ideal, but it is not necessarily disastrous either. So I
don't regard it as a true "need" like our need for air, water, food,
space, the right temperatures, social and political freedom and
structures etc.
> Obviously, another absolute condition of possibility is the existence of
> some candidate mechanism(s) that is capable of initiating and sustaining
> this liquidity process over time. Not all candidate liquidity mechanisms
> are created equal, and I think of RRG's core mandate as the definition
> of the "best possible" successor liquidity mechanism for the Internet
> that we have inherited -- with "best possible" defined as (max.
> [probability of adoption * probability of sustainability, given known
> structural and exogenous risks]).
I am not sure if RRG folks would recognise the group's goals if the
encountered them unawares expressed in those words!
I think the RRG's role is to provide a scalable routing solution
which achieves something like the three goals A, B and C in my footnote:
http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/
I also suggest we aim for D and E (Mobility and open-ended
flexibility), but these are not officially part of the RRG's goals.
> That said, I do not believe that the
> problems that RRG is trying to solve are amenable to any complete or
> final "solution" -- nor even to much symptomatic relief unless the root
> cause(s) of the problems are recognized and addressed directly.
Do you think there is something important we haven't recognised, and
that when we do, we will be able to solve the larger problem of
migrating from IPv4 to something more expansive?
> And that brings me back to the point of departure, Robin, which was the
> contrast between "desire" or (self-asserted) "need" vs. *eligibility*.
> It seems to me that the formula that made it possible for the last
> RRG-relevant domain-wide solution (CIDR) to "work" -- i.e., to be both
> (1) accepted by, at least, the minimum set of parties required to get
> started, and (2) to continue to be regarded as tolerable/acceptable by
> the majority of direct, indirect, and aspiring stakeholders thereafter
> -- involved three key ingredients:
>
> 1. The idea that there is a (i.e., at least one) technically
> justifiable/defensible set of criteria that can be used to determine an
> institution's "eligibility" to consume some share of the common/default
> routing system's finite carrying capacity. To maximize sustainability
> over time, these criteria should establish a link between
> individual-level demand for participation and the potential system-wide
> benefits of allocating a share of finite system carrying capacity to the
> aspiring participant.
>
> 2. To further maximize sustainability over time, eligibility criteria
> should be responsive to changing conditions, e.g., an expansion of the
> system's finite carrying capacity should be reflected in the
> corresponding eligibility criteria. That makes eligibility criteria
> management dependent on the active participation of those who are
> closest to the implementation technologies and practices.
>
> 3. These eligibility criteria should, in both principle and practice,
> form the primary basis for allocating finite routing system carrying
> capacity.
I am wary of using the term "carrying capacity" to refer to the
resource of address space. To me, carrying capacity is to do with
moving packets physically. The question of address space quite a
separate set of limitations on the entire network.
> Achieving this in a sustainable way is only possible if the
> eligibility criteria are administered, and the associated resources
> distributed, through some neutral, transparent mechanism that is
> substantially insulated from the competitive dynamics that govern most
> direct interactions between existing liquidity system participants. In
> other words, the allocation mechanism must be insulated from the very
> dynamics that make the liquidity system itself so valuable, and which
> make the potential risks and consequences of liquidity system capture,
> collapse, or premature obsolescence so severe.
You mean you don't believe that the Market always produces the best
outcomes for all????
This is not Capitalism's finest hour, and I don't have ultimate faith
in markets, but I don't see them as useless or inherently prone to
failure.
I think your previous paragraph asserts that acceptable outcomes can
only arise for all participants, in the case of IPv4 address space
allocation at least, if there is is some overarching, "fair" (however
defined) system which does not at all resemble a "market".
I think there is some truth in this. For instance, if the allocation
system gives 90% of the space to 10% of the applicants, then the
outcome is bound to be unacceptable for most.
But let's consider the power of a market to achieve not so bad
outcomes, even from a lousy starting point.
Let's say Ivip or whatever enables MAB companies to obtain (beg,
borrow, cajole, purchase or rent) space and rent it out profitably as
SPI space.
MAB companies can get address space from anywhere, rent it to anyone
and need no physical infrastructure (they can pay other companies to
run OITRD and to handle the mapping).
So there is going to be a great competitive market developing between
MAB companies and all the end-user networks for whom the costs of SPI
space (fairly low) make it worth getting, for some combination of
multihoming, TE and/or portability between ISPs.
Consider yourself as a MAB company, and me as another one. We are
renting out space at a rapid rate and our ability to do so profitably
depends in large part to our ability to get new space for this purpose.
Companies like yours and mine are able to turn IPv4 space into a cash
cow.
Now consider some ISP ISP-A or AS end-user network NET-B. They have
currently got a bunch of space. That space is worth something to
them, and they have certain costs. If the way they use their space
is less profitable per IPv4 address than what your MAB company or
mine is ready to pay for it, then it is in their interests to move
their usage into a corner of their existing space and rent or sell or
whatever the rest of the space to a MAB company like yours or mine.
This new ability of the MAB companies to use space very effectively
means that market forces alone will cause companies to be motivated
to compact their usage to similar efficiency and then sell whatever
space they no longer need. (This would be dependent on RIR policies
allowing sale or rental of space. I am not assuming that would be
the case - but one way or another, if someone has a bunch of stuff
they don't really need and someone else needs it and is willing to
pay for it, that thing is going to go one way and money the other
before long.)
Your gloomy model seems to assume that all address space is owned by
ISPs, and that an ISP-X relinquishing space to an ISP-Y is
necessarily going to loose out. But ISPs only compete in defined
geographic regions. If an ISP in Melbourne Australia gives some
space to an ISP in Borneo, there is no change in competitive advantage.
The MAB companies do compete globally, so they will be fighting to
get space as long as they can rent it out profitably.
I think the combination of:
Greatly increased ability to use IPv4 space close to 100%
efficiency.
A market for this in the form of non-mobile end-user networks
wanting multihoming, TE and portability.
A further potentially vast and lucrative market for mobile
end-user networks (cellphones / PCs) wanting just a single
IPv4 address they can use anywhere via the TTR Mobility
system.
A market for advertisable prefixes of global unicast IPv4
space developing by hook or by crook.
will enable a gradual but over time thorough saturation of IPv4 usage
with generally stable cost per IPv4 address, due to the global demand
from MAB companies competing with ISPs and the large PI using
end-user networks for available space.
The more end-user networks use SPI space, the less space an ISP needs
to run its network as some proportion of its customers convert from
PA space to to SPI space.
So it is possible to imagine many end-user networks being on MAB
space, the rest being on PA space, and so the ISPs having all the
space not devoted to MABs. That space would be largely devoted to
DSL etc. PI customers and the rest would be used for ETR and ITR
addresses. This is simplistic, but it illustrates how market forces
would encourage networks which were not making much money from their
address space to rent or sell it to those who can.
There would still be iniquities - such as if those who happened to
get a huge amount of space were able to profit by renting or selling
it.
My point is that these market processes, while not ideal in principle
or in practice, would to quite a large extent provide sufficient
space to the ISPs and MAB companies who need it, because they would
be able to make money from this space - and so could afford to buy or
rent it from those who are using it less intensively and presumably
less profitably.
> So, to your point about not being able to "provide multihoming etc. for
> all end-user networks which want it," I respond that that is not a
> realistic goal, given the joint constraints of technology and human
> nature.
I was using "want it" as shorthand for "want it sufficiently to pay
the price, which would be lower than the only current option: an AS
and a PI prefix". I like this better than "need" it, because it is
easy for people outside the network to say that some end-user network
doesn't really "need" multihoming or whatever.
If Ivip or something likes it succeeds, it would have the effect that
SPI space and its benefits be generally available at a price that is
attractive to many more end-user networks than would currently be
able to afford PI space, even if it was available.
> I seriously doubt that it ever will be, because in an uncertain
> world full of real and/or potential competitors, people will always
> "want" more than they currently "need," and nothing short of infinite
> carrying capacity or a change in human nature can remedy that.
Sure, I was trying to be concise by using just "want".
> However,
> I believe with even greater seriousness that we must aspire to preserve
> -- or barring that possibility to recreate with the shortest humanly
> possible interruption -- those three conditions, and the kind of
> "eligibility-based" system admission process that results when those
> conditions are in place.
I know little about the machinations of RIRs. I imagine that already
and increasingly in the years to come, some highly sophisticated and
enlightened policies are being developed and enforced regarding who
gets to keep whatever space was previously allocated them in the
prior days of Plenty, when it was assumed that a much lower rate of
actual utilisation was all that was practical. If Ivip or similar
works as I intend it then this will set a new high standard for close
to 100% utilisation of IP addresses. I expect many end-user network
will be happy with one, a few or a relatively small number of
addresses whereas in the past, they might have been able to justify
getting a /22 or similar.
> I believe this strongly because the history of liquidity systems past
> suggests that if those conditions are not sustained or restored in
> (effectively no) time, whatever endures may not be redeemable, or worth
> redeeming for that matter, and in any event system maintenance by then
> is then likely to have become someone else's problem.
This is pretty bleak. I don't see any great disaster occurring in
IPv4. I just think we will develop means of using the space more and
more intensively. A scalable routing solution such as LISP, APT,
Ivip or TRRP would be a crucial part of this.
> Just as the "need"
> for a liquidity mechanism like TCP/IP does not guarantee that one will
> emerge and be widely adopted, neither does that need guarantee that an
> existing mechanism will endure, or in enduring superficially ("in form
> only") continue to provide those substantive benefits that make
> liquidity systems so valuable, and so vanishingly rare in the wild.
I agree that systems don't automatically run the way most people
would like them, but I don't foresee any great crunch in IPv4, unless
perhaps everyone insists on having a cellphone with its own dedicated
IPv4 address. That is perfectly possible with the TTR Mobility
approach - but there probably won't be enough IPv4 addresses to go
around. That won't be a sudden thing. Humanity will cope with
cellphones not having their own IPv4 address!
> Monetary liquidity systems have come and gone in the past -- but the
> ones that have both "gone" and subsequently "come back" have almost
> universally been backed by sovereign power -- and 100% of those were/are
> exclusively national in scope. What that portends for the future of our
> own, first-of-its kind, extra-territorial, attachment-based liquidity
> mechanism is left as a exercise for the reader.
Hmmm - it is intriguing to think that this uncommonly well designed
networking protocol actually created something like an international
currency. IPv4 address space futures anyone?
> Note: To prevent the weight of this message from giving the Internet a
> hernia, I've split out comments on the footnote next in a separate
> message, to follow shortly.
>
> Thanks again for your responsiveness...
>
> TV
The Internet will be fine. It has an endless supply of recycled
electrons.
The time, interest and attention span of RRG participants is another
matter - it is a precious resource. There should be a refreshment
stall here for people who made the distance.
That unpleasant prickling sensation you feel is one or two RRG folks
sticking pins in your effigy every time you write more than about 4
paragraphs in a list message. Maybe they will give mine a bit of a rest.
- Robin
_______________________________________________
rrg mailing list
[email protected]
http://www.irtf.org/mailman/listinfo/rrg
