Re: [Lightning-dev] General question on routing difficulties

[Saravanan Vijayakumaran]

Hi Christian,

Are there any open source simulators available for trying different routing
strategies? Or even a simulator for the Lightning network as a whole?

Regards
sarva


On Fri, Nov 17, 2017 at 8:00 PM, Christian Decker <
decker.christ...@gmail.com> wrote:

> Oh yeah, my mail tool destroyed that mail quite expertly :-)
>
> The footnotes were
> [1] https://github.com/lightningnetwork/lightning-
> rfc/blob/master/07-routing-gossip.md
> [2] https://medium.com/@rusty_lightning/lightning-routing-
> rough-background-dbac930abbad
> [3] https://www.tik.ee.ethz.ch/file/a20a865ce40d40c8f942cf206a7cba
> 96/Scalable_Funding_Of_Blockchain_Micropayment_Networks%20(1).pdf
>
> We will eventually move away from the hash function based approach in
> favor of something that allows us to decorrelate hops in a route. We have
> indeed started writing down some of the ideas at least for Lightning in the
> project's wiki [4], but they're definitely not fleshed out.
>
> [4] https://github.com/lightningnetwork/lightning-rfc/wiki/Brainstorming
>
>
> On Fri, Nov 17, 2017 at 3:10 PM Benjamin Mord  wrote:
>
>> "I think this is exactly the right venue to discuss these kinds of
>> issue..." - you are probably right! My bad.
>>
>> Christian, thank you for your knowledgable reply. The footnotes did not
>> come through on my end, I am especially interested in [3]. Do you have a
>> link? I am thrilled to hear of a Bitcoin-compatible revive alternative! :)
>>
>> Are we keeping an inventory somewhere of the cryptographic primitives
>> being used in lightning and the specific assumptions being made about them
>> (e.g. preimage resistance vs collision resistance and such)? One project I
>> have not yet found but believe we need across the entire cryptocurrency
>> community, is a (wiki-style?) inventory of unproven mathematical
>> assumptions (e.g. hardness of discrete logarithm) and/or cryptographic
>> primitives, cataloged in terms of the cryptocurrency technologies which
>> require them. Such a resource could help the community respond more
>> quickly, comprehensively, and transparently to the inevitable cryptanalytic
>> surprises that will pop up over time (especially from the quantum
>> cryptanalytic area, but even the classical cryptanalytic community as well).
>>
>> Related, I believe the ideal end state would be to only assume existence
>> of a preimage-resistant hash function, and to code such that one function
>> could be quickly swapped with another and thus update entire system. I'm
>> not sure if that is a realistic goal, but here is my first attempt to move
>> in that direction in case it is of interest to lightning. It is hard to
>> imagine it would be a new idea, although I have not yet found the precedent:
>> http://ben.mord.io/p/delayed-chained-key-revelation-dckr.html
>>
>> Thanks,
>> Ben
>>
>>
>> On Nov 17, 2017 8:04 AM, "Christian Decker" 
>> wrote:
>>
>> On Thu, Nov 16, 2017 at 5:01 PM Benjamin Mord  wrote:
>>
>>> Ivan,
>>>
>>> That is mostly false, but with bits of truth sprinkled in. Contact me at
>>> b...@mord.io for further discussion so we tread lightly on the lists'
>>> email inboxes.
>>>
>>
>> I think this is exactly the right venue to discuss these kinds of issue,
>> so please don't move the conversation somewhere else :-)
>>
>> Routing is still very much in flux, we have a minimally viable routing
>> protocol in the spec [1]. It is minimal in the sense that we just push
>> the entire network's topology to the edges, which can then locally
>> compute routes. This is effectively a source-routed network, which
>> matches the requirements of the onion routing protocol we use for
>> privact as well. But this does not mean that this is protocol is set in
>> stone. We are actively working on finding better solutions to the
>> problem of finding routes across a vast network of millions if not
>> billions of nodes. Distance vector routing such as BGP uses may be one
>> option like Ben suggested.
>>
>> For now the network can easily scale to about 1 million channels [2]
>> even on very limited devices, Upgrading to another protocol at a later
>> point in time is trivial, since none of the routing information is
>> consensus critical. We have all the extension points built in to allow
>> future extensibility.
>>
>>
>>> But briefly: scale-capable routing protocols are possible as
>>> demonstrated by IP and thus by the internet itself. As for centralizing
>>> flow through small number of liquidity providers, yes that does seem
>>> economically probable, at least unless / until off-chain channel
>>> rebalancing mechanism (like the recently proposed "revive" protocol) come
>>> about. Bitcoin script is not currently revive-capable but Ethereum is, so
>>> either Bitcoin revive could be enabled via two-way pegged sidechain
>>> protocol with Ethereum, or even better, by a purpose-built (yet still not
>>> Turing-complete) extension to Bitcoin script itself in the future.

Re: [Lightning-dev] General question on routing difficulties

[Benjamin Mord]

Thanks for the links!

And just to clarify, my other comments relate to cryptographic conservatism
in general, and not specifically to routing. I switched topics, sorry for
the confusion.

On Nov 17, 2017 9:30 AM, "Christian Decker" 
wrote:

> Oh yeah, my mail tool destroyed that mail quite expertly :-)
>
> The footnotes were
> [1] https://github.com/lightningnetwork/lightning-
> rfc/blob/master/07-routing-gossip.md
> [2] https://medium.com/@rusty_lightning/lightning-routing-
> rough-background-dbac930abbad
> [3] https://www.tik.ee.ethz.ch/file/a20a865ce40d40c8f942cf206a7cba
> 96/Scalable_Funding_Of_Blockchain_Micropayment_Networks%20(1).pdf
>
> We will eventually move away from the hash function based approach in
> favor of something that allows us to decorrelate hops in a route. We have
> indeed started writing down some of the ideas at least for Lightning in the
> project's wiki [4], but they're definitely not fleshed out.
>
> [4] https://github.com/lightningnetwork/lightning-rfc/wiki/Brainstorming
>
>
> On Fri, Nov 17, 2017 at 3:10 PM Benjamin Mord  wrote:
>
>> "I think this is exactly the right venue to discuss these kinds of
>> issue..." - you are probably right! My bad.
>>
>> Christian, thank you for your knowledgable reply. The footnotes did not
>> come through on my end, I am especially interested in [3]. Do you have a
>> link? I am thrilled to hear of a Bitcoin-compatible revive alternative! :)
>>
>> Are we keeping an inventory somewhere of the cryptographic primitives
>> being used in lightning and the specific assumptions being made about them
>> (e.g. preimage resistance vs collision resistance and such)? One project I
>> have not yet found but believe we need across the entire cryptocurrency
>> community, is a (wiki-style?) inventory of unproven mathematical
>> assumptions (e.g. hardness of discrete logarithm) and/or cryptographic
>> primitives, cataloged in terms of the cryptocurrency technologies which
>> require them. Such a resource could help the community respond more
>> quickly, comprehensively, and transparently to the inevitable cryptanalytic
>> surprises that will pop up over time (especially from the quantum
>> cryptanalytic area, but even the classical cryptanalytic community as well).
>>
>> Related, I believe the ideal end state would be to only assume existence
>> of a preimage-resistant hash function, and to code such that one function
>> could be quickly swapped with another and thus update entire system. I'm
>> not sure if that is a realistic goal, but here is my first attempt to move
>> in that direction in case it is of interest to lightning. It is hard to
>> imagine it would be a new idea, although I have not yet found the precedent:
>> http://ben.mord.io/p/delayed-chained-key-revelation-dckr.html
>>
>> Thanks,
>> Ben
>>
>>
>> On Nov 17, 2017 8:04 AM, "Christian Decker" 
>> wrote:
>>
>> On Thu, Nov 16, 2017 at 5:01 PM Benjamin Mord  wrote:
>>
>>> Ivan,
>>>
>>> That is mostly false, but with bits of truth sprinkled in. Contact me at
>>> b...@mord.io for further discussion so we tread lightly on the lists'
>>> email inboxes.
>>>
>>
>> I think this is exactly the right venue to discuss these kinds of issue,
>> so please don't move the conversation somewhere else :-)
>>
>> Routing is still very much in flux, we have a minimally viable routing
>> protocol in the spec [1]. It is minimal in the sense that we just push
>> the entire network's topology to the edges, which can then locally
>> compute routes. This is effectively a source-routed network, which
>> matches the requirements of the onion routing protocol we use for
>> privact as well. But this does not mean that this is protocol is set in
>> stone. We are actively working on finding better solutions to the
>> problem of finding routes across a vast network of millions if not
>> billions of nodes. Distance vector routing such as BGP uses may be one
>> option like Ben suggested.
>>
>> For now the network can easily scale to about 1 million channels [2]
>> even on very limited devices, Upgrading to another protocol at a later
>> point in time is trivial, since none of the routing information is
>> consensus critical. We have all the extension points built in to allow
>> future extensibility.
>>
>>
>>> But briefly: scale-capable routing protocols are possible as
>>> demonstrated by IP and thus by the internet itself. As for centralizing
>>> flow through small number of liquidity providers, yes that does seem
>>> economically probable, at least unless / until off-chain channel
>>> rebalancing mechanism (like the recently proposed "revive" protocol) come
>>> about. Bitcoin script is not currently revive-capable but Ethereum is, so
>>> either Bitcoin revive could be enabled via two-way pegged sidechain
>>> protocol with Ethereum, or even better, by a purpose-built (yet still not
>>> Turing-complete) extension to Bitcoin script itself in the 

Re: [Lightning-dev] General question on routing difficulties

[Christian Decker]

Oh yeah, my mail tool destroyed that mail quite expertly :-)

The footnotes were
[1]
https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md
[2]
https://medium.com/@rusty_lightning/lightning-routing-rough-background-dbac930abbad
[3]
https://www.tik.ee.ethz.ch/file/a20a865ce40d40c8f942cf206a7cba96/Scalable_Funding_Of_Blockchain_Micropayment_Networks%20(1).pdf

We will eventually move away from the hash function based approach in favor
of something that allows us to decorrelate hops in a route. We have indeed
started writing down some of the ideas at least for Lightning in the
project's wiki [4], but they're definitely not fleshed out.

[4] https://github.com/lightningnetwork/lightning-rfc/wiki/Brainstorming


On Fri, Nov 17, 2017 at 3:10 PM Benjamin Mord  wrote:

> "I think this is exactly the right venue to discuss these kinds of
> issue..." - you are probably right! My bad.
>
> Christian, thank you for your knowledgable reply. The footnotes did not
> come through on my end, I am especially interested in [3]. Do you have a
> link? I am thrilled to hear of a Bitcoin-compatible revive alternative! :)
>
> Are we keeping an inventory somewhere of the cryptographic primitives
> being used in lightning and the specific assumptions being made about them
> (e.g. preimage resistance vs collision resistance and such)? One project I
> have not yet found but believe we need across the entire cryptocurrency
> community, is a (wiki-style?) inventory of unproven mathematical
> assumptions (e.g. hardness of discrete logarithm) and/or cryptographic
> primitives, cataloged in terms of the cryptocurrency technologies which
> require them. Such a resource could help the community respond more
> quickly, comprehensively, and transparently to the inevitable cryptanalytic
> surprises that will pop up over time (especially from the quantum
> cryptanalytic area, but even the classical cryptanalytic community as well).
>
> Related, I believe the ideal end state would be to only assume existence
> of a preimage-resistant hash function, and to code such that one function
> could be quickly swapped with another and thus update entire system. I'm
> not sure if that is a realistic goal, but here is my first attempt to move
> in that direction in case it is of interest to lightning. It is hard to
> imagine it would be a new idea, although I have not yet found the precedent:
> http://ben.mord.io/p/delayed-chained-key-revelation-dckr.html
>
> Thanks,
> Ben
>
>
> On Nov 17, 2017 8:04 AM, "Christian Decker" 
> wrote:
>
> On Thu, Nov 16, 2017 at 5:01 PM Benjamin Mord  wrote:
>
>> Ivan,
>>
>> That is mostly false, but with bits of truth sprinkled in. Contact me at
>> b...@mord.io for further discussion so we tread lightly on the lists'
>> email inboxes.
>>
>
> I think this is exactly the right venue to discuss these kinds of issue,
> so please don't move the conversation somewhere else :-)
>
> Routing is still very much in flux, we have a minimally viable routing
> protocol in the spec [1]. It is minimal in the sense that we just push
> the entire network's topology to the edges, which can then locally
> compute routes. This is effectively a source-routed network, which
> matches the requirements of the onion routing protocol we use for
> privact as well. But this does not mean that this is protocol is set in
> stone. We are actively working on finding better solutions to the
> problem of finding routes across a vast network of millions if not
> billions of nodes. Distance vector routing such as BGP uses may be one
> option like Ben suggested.
>
> For now the network can easily scale to about 1 million channels [2]
> even on very limited devices, Upgrading to another protocol at a later
> point in time is trivial, since none of the routing information is
> consensus critical. We have all the extension points built in to allow
> future extensibility.
>
>
>> But briefly: scale-capable routing protocols are possible as demonstrated
>> by IP and thus by the internet itself. As for centralizing flow through
>> small number of liquidity providers, yes that does seem economically
>> probable, at least unless / until off-chain channel rebalancing mechanism
>> (like the recently proposed "revive" protocol) come about. Bitcoin script
>> is not currently revive-capable but Ethereum is, so either Bitcoin revive
>> could be enabled via two-way pegged sidechain protocol with Ethereum, or
>> even better, by a purpose-built (yet still not Turing-complete) extension
>> to Bitcoin script itself in the future.
>>
>
> As a matter of fact, Conrad and I just published a similar technique for
> off-chain channel rebalancing and fund re-allocation based solely on
> Bitcoin [3] (major props to Conrad for the excellent writeup!). The
> flexibility in Bitcoin exists.
>
> As for the hubs everybody is assuming will form, I don't think they're
> as likely to form. Creating such a hub is 

Re: [Lightning-dev] General question on routing difficulties

[Benjamin Mord]

"I think this is exactly the right venue to discuss these kinds of
issue..." - you are probably right! My bad.

Christian, thank you for your knowledgable reply. The footnotes did not
come through on my end, I am especially interested in [3]. Do you have a
link? I am thrilled to hear of a Bitcoin-compatible revive alternative! :)

Are we keeping an inventory somewhere of the cryptographic primitives being
used in lightning and the specific assumptions being made about them (e.g.
preimage resistance vs collision resistance and such)? One project I have
not yet found but believe we need across the entire cryptocurrency
community, is a (wiki-style?) inventory of unproven mathematical
assumptions (e.g. hardness of discrete logarithm) and/or cryptographic
primitives, cataloged in terms of the cryptocurrency technologies which
require them. Such a resource could help the community respond more
quickly, comprehensively, and transparently to the inevitable cryptanalytic
surprises that will pop up over time (especially from the quantum
cryptanalytic area, but even the classical cryptanalytic community as well).

Related, I believe the ideal end state would be to only assume existence of
a preimage-resistant hash function, and to code such that one function
could be quickly swapped with another and thus update entire system. I'm
not sure if that is a realistic goal, but here is my first attempt to move
in that direction in case it is of interest to lightning. It is hard to
imagine it would be a new idea, although I have not yet found the precedent:
http://ben.mord.io/p/delayed-chained-key-revelation-dckr.html

Thanks,
Ben

On Nov 17, 2017 8:04 AM, "Christian Decker" 
wrote:

On Thu, Nov 16, 2017 at 5:01 PM Benjamin Mord  wrote:

> Ivan,
>
> That is mostly false, but with bits of truth sprinkled in. Contact me at
> b...@mord.io for further discussion so we tread lightly on the lists'
> email inboxes.
>

I think this is exactly the right venue to discuss these kinds of issue,
so please don't move the conversation somewhere else :-)

Routing is still very much in flux, we have a minimally viable routing
protocol in the spec [1]. It is minimal in the sense that we just push
the entire network's topology to the edges, which can then locally
compute routes. This is effectively a source-routed network, which
matches the requirements of the onion routing protocol we use for
privact as well. But this does not mean that this is protocol is set in
stone. We are actively working on finding better solutions to the
problem of finding routes across a vast network of millions if not
billions of nodes. Distance vector routing such as BGP uses may be one
option like Ben suggested.

For now the network can easily scale to about 1 million channels [2]
even on very limited devices, Upgrading to another protocol at a later
point in time is trivial, since none of the routing information is
consensus critical. We have all the extension points built in to allow
future extensibility.


> But briefly: scale-capable routing protocols are possible as demonstrated
> by IP and thus by the internet itself. As for centralizing flow through
> small number of liquidity providers, yes that does seem economically
> probable, at least unless / until off-chain channel rebalancing mechanism
> (like the recently proposed "revive" protocol) come about. Bitcoin script
> is not currently revive-capable but Ethereum is, so either Bitcoin revive
> could be enabled via two-way pegged sidechain protocol with Ethereum, or
> even better, by a purpose-built (yet still not Turing-complete) extension
> to Bitcoin script itself in the future.
>

As a matter of fact, Conrad and I just published a similar technique for
off-chain channel rebalancing and fund re-allocation based solely on
Bitcoin [3] (major props to Conrad for the excellent writeup!). The
flexibility in Bitcoin exists.

As for the hubs everybody is assuming will form, I don't think they're
as likely to form. Creating such a hub is extremely costly since it'll
have to allocate sufficient funds to cover the maximum imbalance of all
of its channels ahead of time. Then the fees must cover the opportunity
cost of allocating all of those funds to channels instead of investing
them somewhere else. On top of that the funds will not be moved alot
since they serve only a small number of endpoints connected through
those channels, this compounds the problem of having high fees. The high
fees make the hub channels a really bad choice for your payments, after
all you were looking for small fees for your payments, right? It opens
up an opportunity for nodes to open bypasses that grab some of the
traffic and associated fees from the expensive hub.

All of that being said, we should be careful about our predictions on
how the topology will look, I added some counter arguments to a
hub-and-spoke network forming, but nobody can really be sure about
what'll happen.


> In either