Re: [Lightning-dev] Resizing Lightning Channels Off-Chain With Hierarchical Channels
On Tue, Apr 18, 2023 at 07:17:34PM +, jlspc wrote: > > One thing that confuses me about the paper is how to think about routing > > to a "channel" rather than a node -- ie the payment from "E->FG->A" where > > "FG" isn't "F" or "G", but "both of them". > Yes, I found it very difficult to think about, and I kept getting confused > between concepts like "user", "node", "channel", and "factory". > The thing that works best for me is to create a clear definition of each of > these terms, along with the term "party". Okay, that makes sense. I think it might work better to treat "node" as synonymous with "user" rather than "party" though -- that way you can say "you create a lightning node by running lightning node software such as lnd/cln/eclair/etc". That means not all vertices in the payment routing network are nodes; but all vertices in the *gossip* network are nodes, so that seems okay. Just saying "channel" (instead of "logical channel") and "utxo/off-chain utxo" (instead of "physical channel") might also work okay. > I also think it's best to imagine a world in which there are hierarchical > channels, but there are no "factories" per se. Personally, I use "channel factory" to mean "anything that lets a single utxo contain multiple channels between different users, that can be reorganised without going on-chain", so I think once you've got hierarchial channels, you've implicitly got (a variation of) channel factories. (I'm not sure "channel factories" is really the most evocative way of describing them -- at least when I think of a factory, I think the product should be accessible to everyone; but for channel factories you have to be involved in the factory's original mutlisig to be able to use one of its channels. Maybe better to call them "channel coops", where you're creating a little commune of friends/allies to work together with each other. Could be pronounced like "workers' co-op" or like "chicken coop", either way :) > * Logical Channel: a layer 2 construct that consists of all of the physical > channels owned by a specific pair of parties > - the size (capacity) of a logical channel is the sum of the sizes of their > physical channels > - (Footnote: It's possible, with a significant amount of software > development work that I in no way discount, to route a payment through a > logical channel where the payment traverses multiple physical channels at the > same time. This is done by using separate HTLCs, all sharing the same secret, > in each of the physical channels that the payment traverses. I can write more > about this if that would be helpful.) I think it might already be interesting to write a BOLT/BLIP for that? Having a single channel backed by multiple on-chain utxos is probably interesting for splicing (adding or spending a utxo while keeping the channel open on the other utxos might be able to be done more simply than splicing in general), and having multiple utxos might let you increase some of your channel limits, eg `max_accepted_htlcs` might be able to be increased to 483*U where U is the number of UTXOs backing the channel. > > It feels like there's a whole > > mass of complications hidden in there from a routing perspective; like how > > do you link "FG" back with "F" and "G", how do you decide fees, how do > > you communicate fees/max_htlc/etc. > Regarding the specific issues you raised: > Q: How do you link "FG" back with "F" and "G"? > A: In terms of gossiping and the channel graph, you don't Yeah, I think that simplifies things substantially. I think the main thing that misled me here was the "CD->E->FG" payment chain -- it doesn't make sense to me why E would want to devote funds that can only be used for rebalancing channels, but not normal payments. Having that be CD->DE->FG seems like it would make much more sense in that model. (Though, obviously, no one except D and E could necessarily tell the difference between those two scenarios in practice, and just because something doesn't make sense, doesn't mean nobody will actually do it) The other thing was that going from N nodes to C*N channels, then re-considering each of the C*N channels (A->B, etc) as also potentially being nodes and adding an additional K*C*N channels (AB->CD, etc) seemed like it might be quadratic to me. But it's probably not -- C (channels per node) and K (utxos per channel) are probably constant or logarithmic in N, so it's probably okay? On the other hand, I could see the rebalancing channels not actually being very useful for routing payments (they require 3+ signatures, and may not even be publicly connected to any of the level-1 nodes), so it could make sense to just treat them as two different networks, where regular people doing payments only see the base channels, but high-availability nodes also find out about the rebalancing channels. If so, then the extra nodes/channels in the rebalancing graph only affect people who can afford to dedicate the resources to
Re: [Lightning-dev] [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning
Hi Kostas > Could you please point me to a resource that describes the default policy > changes (that are happening for lightning)? I have seen discussions here and > there but it would help if they are aggregated somewhere for reviewing. It is hard to follow because most of the discussions aren't on public channels, a number of the devs working on these proposed default policy changes aren't taking the BIP process seriously and no one is willing to discuss the criteria for merging default policy changes (and consensus changes for that matter) into bitcoin-inquisition (default signet). In addition the work (to the extent that it is public) is scattered all over the place. So yeah it seems like a mess to me from the perspective of someone is seeking to follow, review and monitor it. This Bitcoin StackExchange post [0] is my first attempt to do what you're looking for and these Bitcoin Core PR review clubs [1], [2] were really good from Gloria. But yeah the BIP process (as I've said a thousand times and been ignored) is the place to hammer out specifications for complex things like default policy proposals and when people don't care about formalizing specifications it becomes very hard for people like you and me to follow. > Separation of concerns always makes sense to manage complexity. One would > need to have really strong incentives to counter the complexity argument. >> I might be missing some context here but what would the actual benefit of >> integrating them be? Not having to install lightning node separately and >> maybe a more intuitive UX? As I say in the original email having two separate P2P networks and two separate P2P protocols (perhaps) doesn't make much sense if all (or most of the nodes) are both full nodes and Lightning nodes. A testing framework that integrates both base layer and Lightning tests could potentially be easier to track edge cases which fall in the grey area between base layer and Lightning but are critically important for Lightning. A Core wallet that doesn't support Lightning doesn't make much sense in a world where transaction fees are really high and you have to use Lightning unless you are transferring huge amounts. I agree with you that these arguments have to be strong to counter the separation of concerns angle and maybe it is too early to consider it. But if moving in the direction of more widely used consensus compatible forks of Core then having Lightning integrated could make it an attractive option. > PS. Besides, the amount of effort would be significant. Wouldn't that effort > be better spent on, say, separating the consensus logic (i.e. a second > libbitcoinkernel attempt)? libbitcoinkernel can achieve smaller (and still important) goals but I'm skeptical that the more ambitious goal of having lots of different implementations in different languages with libbitcoinkernel at its core and not having to worry about consensus bugs will be reached in the medium term. As we saw with the recent btcd/lnd bugs [3] consensus bugs can crop up in places you might not expect. In that case it was a wire parsing library that you wouldn't expect to be part of a libbitcoinkernel. So if you're still going to encounter consensus bugs outside of libbitcoinkernel there isn't much point (in my view) in using it for alternative implementations. Thanks Michael [0]: https://bitcoin.stackexchange.com/questions/117315/what-and-where-are-the-current-status-of-the-bip-125-replacement-the-v3-policy [1]: https://bitcoincore.reviews/25038 [2]: https://bitcoincore.reviews/25038-2 [3]: https://bitcoin.stackexchange.com/questions/115527/what-is-the-october-2022-bug-in-lnd-what-caused-it-and-what-would-prevent-a-sim -- Michael Folkson Email: michaelfolkson at [protonmail.com](http://protonmail.com/) Keybase: michaelfolkson PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3 --- Original Message --- On Wednesday, April 19th, 2023 at 10:05, Kostas Karasavvas wrote: > Hi Michael, > > On Wed, Apr 19, 2023 at 2:40 AM Michael Folkson via bitcoin-dev > wrote: > >> Any thoughts on this from the Core Lightning contributors? The way I see it >> with upcoming proposed changes to default policy (primarily though not >> exclusively for Lightning) and a soft fork activation attempt of APO/APOAS >> (primarily though not > > Could you please point me to a resource that describes the default policy > changes (that are happening for lightning)? I have seen discussions here and > there but it would help if they are aggregated somewhere for reviewing. > >> exclusively for Lightning) that a tighter coupling between the full node and >> the Lightning node could eventually make sense. In a world where transaction >> fees were much higher you'd think almost every full node would also want to >> be a Lightning node and so the separation of concerns would make less sense. > > Separation of concerns always makes sense to manage complexity. One
Re: [Lightning-dev] [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning
Hi Michael, On Wed, Apr 19, 2023 at 2:40 AM Michael Folkson via bitcoin-dev < bitcoin-...@lists.linuxfoundation.org> wrote: > Any thoughts on this from the Core Lightning contributors? The way I see > it with upcoming proposed changes to default policy (primarily though not > exclusively for Lightning) and a soft fork activation attempt of APO/APOAS > (primarily though not > Could you please point me to a resource that describes the default policy changes (that are happening for lightning)? I have seen discussions here and there but it would help if they are aggregated somewhere for reviewing. > exclusively for Lightning) that a tighter coupling between the full node > and the Lightning node could eventually make sense. In a world where > transaction fees were much higher you'd think almost every full node would > also want to be a Lightning node and so the separation of concerns would > make less sense. > Separation of concerns always makes sense to manage complexity. One would need to have really strong incentives to counter the complexity argument. I might be missing some context here but what would the actual benefit of integrating them be? Not having to install lightning node separately and maybe a more intuitive UX? Having two separate P2P networks and two separate P2P protocols also > wouldn't make much sense in this scenario. You could obviously still opt > out of Lightning P2P messages if you weren't interested in Lightning. > > The alternative would be just to focus on Knots style consensus compatible > forks of Core with limited additional functionality. But I think we've > reached the point of no return on Core dominance and not having widely used > "distros". As the ecosystem scales systems and processes should be > constantly evolving and improving and to me if anything Core's seem to be > going backwards. > > Personally, I have great difficulty seeing lightning as something other than an L2 build on top of Bitcoin. There will be other L2s. Regards, Kostas PS. Besides, the amount of effort would be significant. Wouldn't that effort be better spent on, say, separating the consensus logic (i.e. a second libbitcoinkernel attempt)? > Thanks > Michael > > -- > Michael Folkson > Email: michaelfolkson at protonmail.com > Keybase: michaelfolkson > PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3 > > --- Original Message --- > On Saturday, January 14th, 2023 at 20:26, Michael Folkson < > michaelfolk...@protonmail.com> wrote: > > I tweeted this [0] back in November 2022. > > "With the btcd bugs and the analysis paralysis on a RBF policy option in > Core increasingly thinking @BitcoinKnots and consensus compatible forks of > Core are the future. Gonna chalk that one up to another thing @LukeDashjr > was right about all along." > > A new bare bones Knots style Bitcoin implementation (in C++/C) integrated > with Core Lightning was a long term idea I had (and presumably many others > have had) but the dysfunction on the Bitcoin Core project this week (if > anything it has been getting worse over time, not better) has made me start > to take the idea more seriously. It is clear to me that the current way the > Bitcoin Core project is being managed is not how I would like an open > source project to be managed. Very little discussion is public anymore and > decisions seem to be increasingly made behind closed doors or in private > IRC channels (to the extent that decisions are made at all). Core Lightning > seems to have the opposite problem. It is managed effectively in the open > (admittedly with fewer contributors) but doesn't have the eyeballs or the > usage that Bitcoin Core does. Regardless, selfishly I at some point would > like a bare bones Bitcoin and Lightning implementation integrated in one > codebase. The Bitcoin Core codebase has collected a lot of cruft over time > and the ultra conservatism that is needed when treating (potential) > consensus code seems to permeate into parts of the codebase that no one is > using, definitely isn't consensus code and should probably just be removed. > > The libbitcoinkernel project was (is?) an attempt to extract the consensus > engine out of Core but it seems like it won't achieve that as consensus is > just too slippery a concept and Knots style consensus compatible codebase > forks of Bitcoin Core seem to still the model. To what extent you can > safely chop off this cruft and effectively maintain this less crufty fork > of Bitcoin Core also isn't clear to me yet. > > Then there is the question of whether it makes sense to mix C and C++ code > that people have different views on. C++ is obviously a superset of C but > assuming this merging of Bitcoin Core and Core Lightning is/was the optimal > final destination it surely would have been better if Core Lightning was > written in the same language (i.e. with classes) as Bitcoin Core. > > I'm just floating the idea to (hopefully) hear from people who are much > more familiar with
Re: [Lightning-dev] Resizing Lightning Channels Off-Chain With Hierarchical Channels
> One thing that confuses me about the paper is how to think about routing > to a "channel" rather than a node -- ie the payment from "E->FG->A" where > "FG" isn't "F" or "G", but "both of them". Yes, I found it very difficult to think about, and I kept getting confused between concepts like "user", "node", "channel", and "factory". The thing that works best for me is to create a clear definition of each of these terms, along with the term "party". I also think it's best to imagine a world in which there are hierarchical channels, but there are no "factories" per se. The distinction is meaningful in the sense that a hierarchical channel requires that there are exactly *two* *fixed* "parties" to which the hierarchical channel pays out, and all changes to the balances paid to those parties are made via HTLCs. In contrast, a "factory" can pay out to an arbitrary number of users or groups of users, and changes to the balances paid to those users or groups of users are made via updating the state of the factory (typically without an HTLC being required). So, in a world with hierarchical channels, but without factories, we have the following terms: * User: a person * Party: a fixed group consisting of one or more users * Node: a party - so "party" and "node" are synonyms, with "party" emphasizing the human side of things and "node" emphasizing the graph side of things, such as when discussing the Lightning routing graph * Physical channel: a layer 2 construct that is funded by an on-chain or an off-chain UTXO, is owned by 2 parties, pays a balance to each of the parties, and updates the parties' balances through HTLCs and a channel protocol - a physical channel can be non-hierarchical (where each of the 2 parties is a single user) or hierarchical (where at least one party consists of more than 1 user) - (Footnote: The concept of an "off-chain UTXO" is itself a bit tricky, as it often refers to one of multiple possible transaction outputs, where the transactions or their ancestors conflict, so only one of them can actually be instantiated. For example, a Lightning channel (using the current penalty protocol) can be said to create an off-chain UTXO that pays to one of the users, even though it's not clear which of the users' Commitment transactions will actually be instantiated. In general, an off-chain UTXO is a guaranteed way of getting a certain portion of a specific on-chain UTXO's funds. Let's skip going down this rabbit hole any further and just assume we understand these issues.) * Logical Channel: a layer 2 construct that consists of all of the physical channels owned by a specific pair of parties - the size (capacity) of a logical channel is the sum of the sizes of their physical channels - (Footnote: It's possible, with a significant amount of software development work that I in no way discount, to route a payment through a logical channel where the payment traverses multiple physical channels at the same time. This is done by using separate HTLCs, all sharing the same secret, in each of the physical channels that the payment traverses. I can write more about this if that would be helpful.) * Lightning Network (LN): a layer 2 network consisting of parties and logical channels between parties - the LN can be represented as a directed graph, where each party is a node and each logical channel is pair of arcs (one in each direction) between two nodes - the LN can be used to make payments from one node to another node via a path through the graph - the purpose of the payment could be to send bitcoin from one user to another (in this case, the payment's source and destination nodes are both 1-user parties) - the purpose of the payment could be to increase or decrease the size of a logical channel (in this case, the payment's source and/or destination node is a multi-user party) - (Footnote: Here's another great opportunity for confusion, as the source and/or destination of the payment is a multi-user party, which is defined as more than two users, and yet the purpose of the payment is to resize a logical channel. Obviously, a group of users is different from a logical channel (as shown in the definitions above), so how does making a payment to or from a group of users change the size of a logical channel? The answer is that the logical channel being resized is the one owned by the group of users.) * Factory: a term that doesn't exist in this world view Sorry if all of this seems overly pedantic. It's the only way I've been able to keep these concepts straight in my mind. I've also found that drawings help clarify these issues, which is why the paper has 20 drawings, many of which are examples of payments and channel graphs. > It feels like there's a whole > mass of complications hidden in there from a routing perspective; like how > do you link "FG" back with "F" and "G", how do you decide fees, how do > you communicate
Re: [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning
Any thoughts on this from the Core Lightning contributors? The way I see it with upcoming proposed changes to default policy (primarily though not exclusively for Lightning) and a soft fork activation attempt of APO/APOAS (primarily though not exclusively for Lightning) that a tighter coupling between the full node and the Lightning node could eventually make sense. In a world where transaction fees were much higher you'd think almost every full node would also want to be a Lightning node and so the separation of concerns would make less sense. Having two separate P2P networks and two separate P2P protocols also wouldn't make much sense in this scenario. You could obviously still opt out of Lightning P2P messages if you weren't interested in Lightning. The alternative would be just to focus on Knots style consensus compatible forks of Core with limited additional functionality. But I think we've reached the point of no return on Core dominance and not having widely used "distros". As the ecosystem scales systems and processes should be constantly evolving and improving and to me if anything Core's seem to be going backwards. Thanks Michael -- Michael Folkson Email: michaelfolkson at [protonmail.com](http://protonmail.com/) Keybase: michaelfolkson PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3 --- Original Message --- On Saturday, January 14th, 2023 at 20:26, Michael Folkson wrote: > I tweeted this [0] back in November 2022. > > "With the btcd bugs and the analysis paralysis on a RBF policy option in Core > increasingly thinking @BitcoinKnots and consensus compatible forks of Core > are the future. Gonna chalk that one up to another thing @LukeDashjr was > right about all along." > > A new bare bones Knots style Bitcoin implementation (in C++/C) integrated > with Core Lightning was a long term idea I had (and presumably many others > have had) but the dysfunction on the Bitcoin Core project this week (if > anything it has been getting worse over time, not better) has made me start > to take the idea more seriously. It is clear to me that the current way the > Bitcoin Core project is being managed is not how I would like an open source > project to be managed. Very little discussion is public anymore and decisions > seem to be increasingly made behind closed doors or in private IRC channels > (to the extent that decisions are made at all). Core Lightning seems to have > the opposite problem. It is managed effectively in the open (admittedly with > fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin > Core does. Regardless, selfishly I at some point would like a bare bones > Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin > Core codebase has collected a lot of cruft over time and the ultra > conservatism that is needed when treating (potential) consensus code seems to > permeate into parts of the codebase that no one is using, definitely isn't > consensus code and should probably just be removed. > > The libbitcoinkernel project was (is?) an attempt to extract the consensus > engine out of Core but it seems like it won't achieve that as consensus is > just too slippery a concept and Knots style consensus compatible codebase > forks of Bitcoin Core seem to still the model. To what extent you can safely > chop off this cruft and effectively maintain this less crufty fork of Bitcoin > Core also isn't clear to me yet. > > Then there is the question of whether it makes sense to mix C and C++ code > that people have different views on. C++ is obviously a superset of C but > assuming this merging of Bitcoin Core and Core Lightning is/was the optimal > final destination it surely would have been better if Core Lightning was > written in the same language (i.e. with classes) as Bitcoin Core. > > I'm just floating the idea to (hopefully) hear from people who are much more > familiar with the entirety of the Bitcoin Core and Core Lightning codebases. > It would be an ambitious long term project but it would be nice to focus on > some ambitious project(s) (even if just conceptually) for a while given > (thankfully) there seems to be a lull in soft fork activation chaos. > > Thanks > Michael > > [0]: > https://twitter.com/michaelfolkson/status/1589220155006910464?s=20=GbPm7w5BqS7rS3kiVFTNcw > > -- > Michael Folkson > Email: michaelfolkson at [protonmail.com](http://protonmail.com/) > Keybase: michaelfolkson > PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] Resizing Lightning Channels Off-Chain With Hierarchical Channels
On Sat, Apr 08, 2023 at 10:26:45PM +, jlspc via Lightning-dev wrote: > From my perspective, this paper makes two contributions (which, to be fair, > may only be "interesting" :) One thing that confuses me about the paper is how to think about routing to a "channel" rather than a node -- ie the payment from "E->FG->A" where "FG" isn't "F" or "G", but "both of them". It feels like there's a whole mass of complications hidden in there from a routing perspective; like how do you link "FG" back with "F" and "G", how do you decide fees, how do you communicate fees/max_htlc/etc. I think it also implies that channel capacity is no longer really something you can gossip very sensibly -- if you have a factory ((A,B),C,D,E) then every payment through AB to C or D or E will decrease AB's channel capacity. You could still gossip the capacity of the overall factory, and sum that to get an overall lightning network capacity, of course. But a lot of the ways of simplifying it also make it harder to do all the nice rebalancing. Anyway, I've tried a few times now to put some thoughts together on that and come up with nothing that I'm happy with, so figured it might be at least worth posing explicitly as a problem... Cheers, aj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
