Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
On Fri, Jul 02, 2021 at 02:20:51PM -0400, Antoine Riard wrote: > More personally, I feel it would be better if such a new specification > process doesn't completely share the same communication infrastructure as > the BOLTs, like [avoiding] having them in the same repository. In addition to Antoine's perception-based concern, I think an additional problem with keeping both BOLTs and BLIPs in the same repository is that there's no easy way for contributors to subscribe to only a subset of issues and PRs. E.g., if Alice is only interested in BOLTs and she clicks the GitHub Watch Repository button, she'll receive notifications for issues and PRs about BLIPs that she's not interested in; vice-versa for Bob who's only interested in BLIPs. If you still think it's desirable to keep BOLTs and BLIPs in the same source tree, you could maybe consider the monotree approach that originated with the Linux kernel project (AFAIK) and which the Bitcoin Core project began experimenting with about a year ago[1] (to moderate success AFAICT). -Dave [1] https://bitcoinops.org/en/newsletters/2020/06/24/#bitcoin-core-19071 signature.asc Description: PGP signature ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Hi all, Thanks so much for the great feedback over the last week. Seems like general agreement that adding a simple home for descriptive design documents focusing on new LN features would be a good thing, and augment the prescriptive BOLTs (which have done a great job getting us this far!). If there is a point of contention, it seems to be about how not only this interacts with the existing BIP system, but also how the BOLTs interact with the BIP system. The only problem I have with BOLTs and bLIPs as BIPs is that it introduces large scope creep over what was originally a pretty simple proposal. I don't really care where these design documents exist, only that there is a standard format and that LN developers and users feel empowered to create them and share them with the broader ecosystem. If we proceed with creating bLIPs in the lightning-rfc repo today and later decide to recreate the BOLTs as BIPs, it will be no trouble at all to recreate bLIPs as BIPs as well. The BIP Process Wishlist sounds great and can be addressed independently. If recruits for merging the BOLTs can be found, we can tackle the mechanics of a merge then (alongside maybe some of the other bitcoin-related *IP repos that exist outside the BIPs? [1] [2]). Best, Ryan [1] https://github.com/satoshilabs/slips [2] https://github.com/rsksmart/RSKIPs On Fri, Jul 2, 2021 at 1:21 PM Antoine Riard wrote: > Hi Ryan, > > Thanks for starting this discussion, I agree it's a good time for the > Lightning development community to start this self-introspection on its own > specification process :) > > First and foremost, maybe we could take a minute off to celebrate the > success of the BOLT process and the road traveled so far ? What was a fuzzy > heap of ideas on a whiteboard a few years ago has bloomed up to a living > and pulsating distributed ecosystem of thousands of nodes all around the > world. If the bet was to deliver on fast, instant, cheap, reasonably > scalable, reasonably confidential Bitcoin payments, it's a won one and > that's really cool. > > Retrospectively, it was a foolhardy bet for a wide diversity of factors. > One could think about opinionated, early design choices deeply affecting > protocol safety and efficiency of which the ultimate validity was still a > function of fluky base layer evolutions [0]. Another could consider the > communication challenges of softly aligning development teams on the common > effort of designing and deploying from scratch a cryptographic protocol as > sophisticated as Lightning. Not an easy task when you're mindful about the > timezones spread, the diversity of software engineering backgrounds and the > differing schedules of priorities. > > So kudos to everyone who has played a part in the Lightning dev process. > The OGs who started the tale, the rookies who jumped on the wagon on the > way and today newcomers showing up with new seeds to nurture the ecosystem > :) > > Now, I would say we more-or-less all agree that the current BOLT process > has reached its limits. Both from private conservations across the teams > but also frustrations expressed during the irc meetings in the past months. > Or as a simple data point, the only meaningful spec object we did merge on > the last 18 months is anchor output, it did consumes a lot of review and > engineering bandwidth from contributors, took few refinement to finalize > (`option_anchors_zero_fee_htlc_tx`) and I believe every implementations are > still scratching their heads on a robust, default fee-bumping strategy. > > So if we agree about the BOLT process limitations, the next question to > raise is how to improve it. Though there, as expressed in other replies, > I'm more we're not going to be able to do that much, as ultimately we're > upper bounded by a fast-pacing, always-growing, permissionless ecosystem of > applications and experiments moving forward in baazar-style and > lower-bounded by a decentralized process across teams allocating their > engineering resources with different priorities or even exploring Lightning > massive evolution stages in heterogenous, synergic directions. > > Breeding another specification process on top of Lightning sounds a good > way forward. Though I believe it might be better to take time to operate > the disentanglement nicely. If we take the list of ideas which could be > part of such a process, one of them, dynamic commitments could make a lot > of sense to be well-designed and well-supported by every implementation. In > case of emergency fixes to deploy safer channel types, if you have to close > all your channels with other implementations, on a holistic scale, it might > cloak the mempools and spike the feerate, strickening safety of every other > channel on the network. Yes we might have safety interdepencies between > implementations :/ > > And it's also good to have thoughtful, well-defined specification bounds > when you're working on coordinated security disclosures to know who has >
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Hi Ryan, Thanks for starting this discussion, I agree it's a good time for the Lightning development community to start this self-introspection on its own specification process :) First and foremost, maybe we could take a minute off to celebrate the success of the BOLT process and the road traveled so far ? What was a fuzzy heap of ideas on a whiteboard a few years ago has bloomed up to a living and pulsating distributed ecosystem of thousands of nodes all around the world. If the bet was to deliver on fast, instant, cheap, reasonably scalable, reasonably confidential Bitcoin payments, it's a won one and that's really cool. Retrospectively, it was a foolhardy bet for a wide diversity of factors. One could think about opinionated, early design choices deeply affecting protocol safety and efficiency of which the ultimate validity was still a function of fluky base layer evolutions [0]. Another could consider the communication challenges of softly aligning development teams on the common effort of designing and deploying from scratch a cryptographic protocol as sophisticated as Lightning. Not an easy task when you're mindful about the timezones spread, the diversity of software engineering backgrounds and the differing schedules of priorities. So kudos to everyone who has played a part in the Lightning dev process. The OGs who started the tale, the rookies who jumped on the wagon on the way and today newcomers showing up with new seeds to nurture the ecosystem :) Now, I would say we more-or-less all agree that the current BOLT process has reached its limits. Both from private conservations across the teams but also frustrations expressed during the irc meetings in the past months. Or as a simple data point, the only meaningful spec object we did merge on the last 18 months is anchor output, it did consumes a lot of review and engineering bandwidth from contributors, took few refinement to finalize (`option_anchors_zero_fee_htlc_tx`) and I believe every implementations are still scratching their heads on a robust, default fee-bumping strategy. So if we agree about the BOLT process limitations, the next question to raise is how to improve it. Though there, as expressed in other replies, I'm more we're not going to be able to do that much, as ultimately we're upper bounded by a fast-pacing, always-growing, permissionless ecosystem of applications and experiments moving forward in baazar-style and lower-bounded by a decentralized process across teams allocating their engineering resources with different priorities or even exploring Lightning massive evolution stages in heterogenous, synergic directions. Breeding another specification process on top of Lightning sounds a good way forward. Though I believe it might be better to take time to operate the disentanglement nicely. If we take the list of ideas which could be part of such a process, one of them, dynamic commitments could make a lot of sense to be well-designed and well-supported by every implementation. In case of emergency fixes to deploy safer channel types, if you have to close all your channels with other implementations, on a holistic scale, it might cloak the mempools and spike the feerate, strickening safety of every other channel on the network. Yes we might have safety interdepencies between implementations :/ And it's also good to have thoughtful, well-defined specification bounds when you're working on coordinated security disclosures to know who has implemented what and whom you should reach out when something is broken. Another orthogonal point to consider is the existence of already higher-layer protocol specifications such as the dlcspecs. Even if the ecosystem is still in the bootstrap phase for now, we already have a discussion to split between a "consensus" track and more optional features. I believe some features discussed there such as negotiation layer about premium fee to compensate unilateral fee-bumping responsibility risk could belong to such a new bLIPs process ? So here my thinking, as a BOLT contributor, what the common subset of problems we want to keep tackling down together in the coming years, what is the remaining subset we're happy to be engage by a higher layer development community and how to draw both communication and software interfaces in-between ? Personally, I would be glad if we not extend the scope of the current BOLT coverage and focus more on fixing the known-issues, simplifying state machines, fixing oddities of channel policies announcements [1], writing down best practices on fee-bumping strategies, agreeing on channel types upgrades raw mechanisms, features discovery and if we want to innovate focus on taproot well-done integration which should keep us busy for few years, among others PTLC support, funding output taproot support, composable taptree for revokeable outputs, ... IHMO, if the BOLT process is officialized it will enter in a more boring phase, focused on safety/reliability/privacy
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> The other thing bLIPs do is do away with the whole "human picks the number of
> documents", and "don't assign your own number, you must wait".
So TL;DR BIPs and BOLTs sometimes require waiting for things (like
review and consensus) and there should be a new acronym and process
("bLIPs") to avoid us having to wait for things. I just think "bLIPs"
adds confusion e.g. should something be a bLIP or a BOLT? Does a bLIP
eventually become a BOLT when it is mature enough? This tendency to
fragment and introduce new acronyms and new processes should be
resisted imo. If a new process is introduced every time there is a
disagreement or perceived friction it just erodes the value of
existing processes and means they all get bypassed. Strengthen and
improve existing processes and only introduce a new one as an absolute
last resort.
Other than the minor frictions described above I don't see why "bLIPs"
can't just be draft BOLTs.
> Adding a third BIP editor more involved with Lightning sounds like a good
> idea.
Or alternatively if BOLTs were subsumed into BIPs I think Bastien
would be a great additional BIP editor to cover Lightning related BIPs
:) I think BOLTs being subsumed into BIPs would be nice but I'm
pessimistic it will happen. Like legislation and regulation in the
legacy financial system alphabet soups only expand they never get
simplified. Let's at least resist alphabet soup expansion here.
On Fri, Jul 2, 2021 at 9:01 AM Bastien TEINTURIER wrote:
>>
>> Will it actually add any more fragmentation that already exists? Due to all
>> the extensibility we've added in the protocol, it's already possible for any
>> implementation to start to work on their own sub-protocols. This just gives
>> them a new venue to at least _describe_ what they're using.
>
>
> It's only my 2 cents, but I'm afraid it will indeed add more fragmentation,
> because
> the fact that there exists a bLIP for feature XXX will likely act as a green
> light to
> deploy it faster instead of spending more time talking about it with the
> community
> and thinking about potential issues, forward-compatibility, etc.
>
> But I agree with you that it also gives more freedom to experiment in the
> real world,
> which helps find issues and correct them, paving the way for better features
> for
> end users.
>
>> It's also likely the case that already implementations, or typically forks
>> of implementations are already using "undocumented" TLVs or feature bits in
>> the wild today.
>
>
> But today we're usually very careful when we do that, and use numbers in high
> ranges
> for these use-cases. In our case for example we use message type 35007 for our
> swap-in and we expect that to change once standardized, so we did extra work
> to
> ensure we wouldn't paint ourselves into a corner when switching to a standard
> version.
>
> I think that if we have a centralized bLIP repo, we can take this opportunity
> to safely
> assign "final" values for types and feature bits that are used by each bLIP,
> and stronger
> guarantees that they will not conflict with another bLIP or BOLT. Of course
> that doesn't
> stop anyone from deploying a conflict, but their use of the same bits won't
> be documented
> so it shouldn't be widely deployed, and browsing the BOLTs and bLIPs should
> let anyone
> see what the "correct" meaning of those bits should be.
>
> Cheers,
> Bastien
>
>
> Le jeu. 1 juil. 2021 à 22:43, Olaoluwa Osuntokun a écrit :
>>
>> > But they don't address the first point at all, they instead work around
>> > it. To be fair, I don't think we can completely address that first point:
>> > properly reviewing spec proposals takes a lot of effort and accepting
>> > complex changes to the BOLTs shouldn't be done lightly.
>>
>> I think this is a fair characterization that I agree with. I also agree that
>> there isn't really a way to fundamentally address it. The issue of scarce
>> review resources is something just about any large open source project needs
>> to deal with: everyone wants to make a PR, but no one wants to review the
>> PRs of others, unless it scratches some tangential itch they may have. IMO
>> it's also the case that the problem/solution space of LN is so large, that
>> it's hard to expect every developer to review each new proposal that comes
>> in, as they themselves have their own set of priorities (product,
>> businesses, protocol, personal, etc).
>>
>> In the end though, I think when there've been critical items that affect all
>> implementations and/or the existence of the protocol itself, developers
>> typically band together to commit resources to help a proposal move forward.
>> One upcoming example of this will be the "base" taproot channel type (the
>> design space is pretty large in that it even permits a new type of symmetric
>> state revocation-based channel).
>>
>> > it will add fragmentation to the network, it will add maintenance costs
>> > and backwards-compatibility issues
>>
>> Will it actually add
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> > Will it actually add any more fragmentation that already exists? Due to all > the extensibility we've added in the protocol, it's already possible for > any > implementation to start to work on their own sub-protocols. This just gives > them a new venue to at least _describe_ what they're using. > It's only my 2 cents, but I'm afraid it will indeed add more fragmentation, because the fact that there exists a bLIP for feature XXX will likely act as a green light to deploy it faster instead of spending more time talking about it with the community and thinking about potential issues, forward-compatibility, etc. But I agree with you that it also gives more freedom to experiment in the real world, which helps find issues and correct them, paving the way for better features for end users. It's also likely the case that already implementations, or typically forks > of implementations are already using "undocumented" TLVs or feature bits in > the wild today. > But today we're usually very careful when we do that, and use numbers in high ranges for these use-cases. In our case for example we use message type 35007 for our swap-in and we expect that to change once standardized, so we did extra work to ensure we wouldn't paint ourselves into a corner when switching to a standard version. I think that if we have a centralized bLIP repo, we can take this opportunity to safely assign "final" values for types and feature bits that are used by each bLIP, and stronger guarantees that they will not conflict with another bLIP or BOLT. Of course that doesn't stop anyone from deploying a conflict, but their use of the same bits won't be documented so it shouldn't be widely deployed, and browsing the BOLTs and bLIPs should let anyone see what the "correct" meaning of those bits should be. Cheers, Bastien Le jeu. 1 juil. 2021 à 22:43, Olaoluwa Osuntokun a écrit : > > But they don't address the first point at all, they instead work around > > it. To be fair, I don't think we can completely address that first > point: > > properly reviewing spec proposals takes a lot of effort and accepting > > complex changes to the BOLTs shouldn't be done lightly. > > I think this is a fair characterization that I agree with. I also agree > that > there isn't really a way to fundamentally address it. The issue of scarce > review resources is something just about any large open source project > needs > to deal with: everyone wants to make a PR, but no one wants to review the > PRs of others, unless it scratches some tangential itch they may have. IMO > it's also the case that the problem/solution space of LN is so large, that > it's hard to expect every developer to review each new proposal that comes > in, as they themselves have their own set of priorities (product, > businesses, protocol, personal, etc). > > In the end though, I think when there've been critical items that affect > all > implementations and/or the existence of the protocol itself, developers > typically band together to commit resources to help a proposal move > forward. > One upcoming example of this will be the "base" taproot channel type (the > design space is pretty large in that it even permits a new type of > symmetric > state revocation-based channel). > > > it will add fragmentation to the network, it will add maintenance costs > > and backwards-compatibility issues > > Will it actually add any more fragmentation that already exists? Due to all > the extensibility we've added in the protocol, it's already possible for > any > implementation to start to work on their own sub-protocols. This just gives > them a new venue to at least _describe_ what they're using. As usual, it's > up to other implementations if they want to adopt it or not, or advise > against its use. > > > many bLIPs will be sub-optimal solutions to the problem they try to > solve > > and some bLIPs will be simply insecure and may put users' funds at risk > > (L2 protocols are hard and have subtle issues that can be easily missed) > > This may be the case, but I guess at times it's hard to know if something > is > objectively sub-optimal without further exploration of the design space, > which usually means either more people involved, or more time examining the > problem. Ultimately, different wallets/implementations may also be willing > to make different usability/security trade-offs. One example here is zero > conf channels: they assume a greater degree of trust with the party you're > _accepting_ the channel from, as if you receive funds over the channel, > they > can be double spent away. However it's undeniable that they improve the UX > by reducing the amount of time a user needs to wait around before they can > actually jump in and use LN. > > In the end though, there's no grand global committee that prevents people > from deploying software they think is interesting or useful. In the long > run, I guess one simply needs to hope that bad ideas die out, or speak out >
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> bLIPs have a slightly different process than BIPs, as well as a different set > of editors/maintainers (more widely distributed). As we saw with the Speedy > Trial saga (fingers crossed), the sole (?) maintainer of the BIP process was > able to effectively steelman the progression of an author document, with no > sound technical objection (they had a competing proposal that could've been > a distinct document). bLIPs sidestep shenanigans like this by having the > primary maintainer/editors be more widely distributed and closer to the > target domain (LN). That's fair. The BIP repository as a whole lost credibility when the sole maintainer at the time steelmanned progress for no good reason. It was done in bad faith and no steps were taken to remove even the *perception* of impropriety. I hope the maintainers of the bLIP repository can be more impartial. The automatic assignment of proposal numbers is an excellent improvement. It's an idea that the BIP repo should adopt since it removes responsibility and power from the maintainers. > > The other thing bLIPs do is do away with the whole "human picks the number > of documents", and "don't assign your own number, you must wait". Borrowing > from EIPs, the number of a document is simply the number of the PR that > proposes the document. This reduces friction, and eliminates a possible > bikeshedding vector. > > -- Laolu > > > On Wed, Jun 30, 2021 at 5:31 PM Ariel Luaces wrote: >> >> BIPs are already the Bazaar style of evolution that simultaneously >> allows flexibility and coordination/interoperability (since anyone can >> create a BIP and they create an environment of discussion). >> >> BOLTs are essentially one big BIP in the sense that they started as a >> place for discussion but are now more rigid. BOLTs must be followed >> strictly to ensure a node is interoperable with the network. And BOLTs >> should be rigid, as rigid as any widely used BIP like 32 for example. >> Even though BOLTs were flexible when being drafted their purpose has >> changed from descriptive to prescriptive. >> Any alternatives, or optional features should be extracted out of >> BOLTs, written as BIPs. The BIP should then reference the BOLT and the >> required flags set, messages sent, or alterations made to signal that >> the BIP's feature is enabled. >> >> A BOLT may at some point organically change to reference a BIP. For >> example if a BIP was drafted as an optional feature but then becomes >> more widespread and then turns out to be crucial for the proper >> operation of the network then a BOLT can be changed to just reference >> the BIP as mandatory. There isn't anything wrong with this. >> >> All of the above would work exactly the same if there was a bLIP >> repository instead. I don't see the value in having both bLIPs and >> BIPs since AFAICT they seem to be functionally equivalent and BIPs are >> not restricted to exclude lightning, and never have been. >> >> I believe the reason this move to BIPs hasn't happened organically is >> because many still perceive the BOLTs available for editing, so >> changes continue to be made. If instead BOLTs were perceived as more >> "consensus critical", not subject to change, and more people were >> strongly encouraged to write specs for new lightning features >> elsewhere (like the BIP repo) then you would see this issue of growing >> BOLTs resolved. >> >> Cheers >> Ariel Lorenzo-Luaces >> >> On Wed, Jun 30, 2021 at 1:16 PM Olaoluwa Osuntokun wrote: >> > >> > > That being said I think all the points that are addressed in Ryan's mail >> > > could very well be formalized into BOLTs but maybe we just need to >> > > rethink >> > > the current process of the BOLTs to make it more accessible for new ideas >> > > to find their way into the BOLTs? >> > >> > I think part of what bLIPs are trying to solve here is promoting more >> > loosely >> > coupled evolution of the network. I think the BOLTs do a good job >> > currently of >> > specifying what _base_ functionality is required for a routing node in a >> > prescriptive manner (you must forward an HTLC like this, etc). However >> > there's >> > a rather large gap in describing functionality that has emerged over time >> > due >> > to progressive evolution, and aren't absolutely necessary, but enhance >> > node/wallet operation. >> > >> > Examples of include things like: path finding heuristics (BOLTs just say >> > you >> > should get from Alice to Bob, but provides no recommendations w.r.t _how_ >> > to do >> > so), fee bumping heuristics, breach retribution handling, channel >> > management, >> > rebalancing, custom records usage (like the podcast index meta-data, >> > messaging, >> > etc), JIT channel opening, hosted channels, randomized channel IDs, fee >> > optimization, initial channel boostrapping, etc. >> > >> > All these examples are effectively optional as they aren't required for >> > base >> > node operation, but they've organically evolved over time as node >> >
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
BOLTs should be BIPs too. Ideally, the authors should be the ones to migrate them, but mirroring is fine - just nobody's taken the time to do it yet. Please stop promoting lies about the BIP repo. I did not "steelman" anything. Adding a third BIP editor more involved with Lightning sounds like a good idea. On Thursday 01 July 2021 20:25:23 Olaoluwa Osuntokun wrote: > > BIPs are already the Bazaar style of evolution that simultaneously > > allows flexibility and coordination/interoperability (since anyone can > > create a > > > BIP and they create an environment of discussion). > > The answer to why not BIPs here applies to BOLTs as well, as bLIPs are > intended to effectively be nested under the BOLT umbrella (same repo, etc). > It's also the case that any document can be mirrored as a BIP, this has > been suggested before, but the BIP editors have decided not to do so. > > bLIPs have a slightly different process than BIPs, as well as a different > set > of editors/maintainers (more widely distributed). As we saw with the Speedy > Trial saga (fingers crossed), the sole (?) maintainer of the BIP process > was able to effectively steelman the progression of an author document, > with no sound technical objection (they had a competing proposal that > could've been a distinct document). bLIPs sidestep shenanigans like this by > having the primary maintainer/editors be more widely distributed and closer > to the target domain (LN). > > The other thing bLIPs do is do away with the whole "human picks the number > of documents", and "don't assign your own number, you must wait". Borrowing > from EIPs, the number of a document is simply the number of the PR that > proposes the document. This reduces friction, and eliminates a possible > bikeshedding vector. > > -- Laolu > > On Wed, Jun 30, 2021 at 5:31 PM Ariel Luaces wrote: > > BIPs are already the Bazaar style of evolution that simultaneously > > allows flexibility and coordination/interoperability (since anyone can > > create a BIP and they create an environment of discussion). > > > > BOLTs are essentially one big BIP in the sense that they started as a > > place for discussion but are now more rigid. BOLTs must be followed > > strictly to ensure a node is interoperable with the network. And BOLTs > > should be rigid, as rigid as any widely used BIP like 32 for example. > > Even though BOLTs were flexible when being drafted their purpose has > > changed from descriptive to prescriptive. > > Any alternatives, or optional features should be extracted out of > > BOLTs, written as BIPs. The BIP should then reference the BOLT and the > > required flags set, messages sent, or alterations made to signal that > > the BIP's feature is enabled. > > > > A BOLT may at some point organically change to reference a BIP. For > > example if a BIP was drafted as an optional feature but then becomes > > more widespread and then turns out to be crucial for the proper > > operation of the network then a BOLT can be changed to just reference > > the BIP as mandatory. There isn't anything wrong with this. > > > > All of the above would work exactly the same if there was a bLIP > > repository instead. I don't see the value in having both bLIPs and > > BIPs since AFAICT they seem to be functionally equivalent and BIPs are > > not restricted to exclude lightning, and never have been. > > > > I believe the reason this move to BIPs hasn't happened organically is > > because many still perceive the BOLTs available for editing, so > > changes continue to be made. If instead BOLTs were perceived as more > > "consensus critical", not subject to change, and more people were > > strongly encouraged to write specs for new lightning features > > elsewhere (like the BIP repo) then you would see this issue of growing > > BOLTs resolved. > > > > Cheers > > Ariel Lorenzo-Luaces > > > > On Wed, Jun 30, 2021 at 1:16 PM Olaoluwa Osuntokun > > > > wrote: > > > > That being said I think all the points that are addressed in Ryan's > > > > mail > > > > > > could very well be formalized into BOLTs but maybe we just need to > > > > rethink > > > > > > the current process of the BOLTs to make it more accessible for new > > > > ideas > > > > > > to find their way into the BOLTs? > > > > > > I think part of what bLIPs are trying to solve here is promoting more > > > > loosely > > > > > coupled evolution of the network. I think the BOLTs do a good job > > > > currently of > > > > > specifying what _base_ functionality is required for a routing node in > > > a prescriptive manner (you must forward an HTLC like this, etc). > > > However > > > > there's > > > > > a rather large gap in describing functionality that has emerged over > > > > time due > > > > > to progressive evolution, and aren't absolutely necessary, but enhance > > > node/wallet operation. > > > > > > Examples of include things like: path finding heuristics (BOLTs just > > > > say you > > > > > should get from Alice to Bob, but provides no
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> But they don't address the first point at all, they instead work around > it. To be fair, I don't think we can completely address that first point: > properly reviewing spec proposals takes a lot of effort and accepting > complex changes to the BOLTs shouldn't be done lightly. I think this is a fair characterization that I agree with. I also agree that there isn't really a way to fundamentally address it. The issue of scarce review resources is something just about any large open source project needs to deal with: everyone wants to make a PR, but no one wants to review the PRs of others, unless it scratches some tangential itch they may have. IMO it's also the case that the problem/solution space of LN is so large, that it's hard to expect every developer to review each new proposal that comes in, as they themselves have their own set of priorities (product, businesses, protocol, personal, etc). In the end though, I think when there've been critical items that affect all implementations and/or the existence of the protocol itself, developers typically band together to commit resources to help a proposal move forward. One upcoming example of this will be the "base" taproot channel type (the design space is pretty large in that it even permits a new type of symmetric state revocation-based channel). > it will add fragmentation to the network, it will add maintenance costs > and backwards-compatibility issues Will it actually add any more fragmentation that already exists? Due to all the extensibility we've added in the protocol, it's already possible for any implementation to start to work on their own sub-protocols. This just gives them a new venue to at least _describe_ what they're using. As usual, it's up to other implementations if they want to adopt it or not, or advise against its use. > many bLIPs will be sub-optimal solutions to the problem they try to solve > and some bLIPs will be simply insecure and may put users' funds at risk > (L2 protocols are hard and have subtle issues that can be easily missed) This may be the case, but I guess at times it's hard to know if something is objectively sub-optimal without further exploration of the design space, which usually means either more people involved, or more time examining the problem. Ultimately, different wallets/implementations may also be willing to make different usability/security trade-offs. One example here is zero conf channels: they assume a greater degree of trust with the party you're _accepting_ the channel from, as if you receive funds over the channel, they can be double spent away. However it's undeniable that they improve the UX by reducing the amount of time a user needs to wait around before they can actually jump in and use LN. In the end though, there's no grand global committee that prevents people from deploying software they think is interesting or useful. In the long run, I guess one simply needs to hope that bad ideas die out, or speak out against them to the public. As LN sits a layer above the base protocol, widespread global consensus isn't really required to make certain classes of changes, and you can't stop people from experimenting on their own. > We can't have collisions on any of these three things. Yeah, collisions are def possible. IMO, this is where the interplay with BOLTs comes in: BOLTs are the global feature bit/tlv/message namespace. A bLIP might come with the amendment of BOLT 9 to define feature bits they used. Of course, this should be done on a best effort basis, as even if you assign a bit for your idea, someone can just go ahead and deploy something else w/ that same bit, and they may never really intersect depending on the nature or how widespread the new feature is. It's also likely the case that already implementations, or typically forks of implementations are already using "undocumented" TLVs or feature bits in the wild today. I don't know exactly which TLV type things like applications that tunnel messages over the network use, but afaik so far there haven't been any disastrous collisions in the wild. -- Laolu On Thu, Jul 1, 2021 at 2:19 AM Bastien TEINTURIER wrote: > Thanks for starting that discussion. > > In my opinion, what we're really trying to address here are the two > following > points (at least from the point of view of someone who works on the spec > and > an implementation): > > - Implementers get frustrated when they've worked on something that they > think > is useful and they can't get it into the BOLTs (the spec PR isn't reviewed, > it progresses too slowly or there isn't enough agreement to merge it) > - Implementers expect other implementers to specify the optional features > they > ship: we don't want to have to reverse-engineer a sub-protocol when users > want our implementation to provide support for feature XXX > > Note that these are two very different concerns. > > bLIPs/SPARKS/BIPs clearly address the second point, which is good. > But they don't
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> BIPs are already the Bazaar style of evolution that simultaneously > allows flexibility and coordination/interoperability (since anyone can create a > BIP and they create an environment of discussion). The answer to why not BIPs here applies to BOLTs as well, as bLIPs are intended to effectively be nested under the BOLT umbrella (same repo, etc). It's also the case that any document can be mirrored as a BIP, this has been suggested before, but the BIP editors have decided not to do so. bLIPs have a slightly different process than BIPs, as well as a different set of editors/maintainers (more widely distributed). As we saw with the Speedy Trial saga (fingers crossed), the sole (?) maintainer of the BIP process was able to effectively steelman the progression of an author document, with no sound technical objection (they had a competing proposal that could've been a distinct document). bLIPs sidestep shenanigans like this by having the primary maintainer/editors be more widely distributed and closer to the target domain (LN). The other thing bLIPs do is do away with the whole "human picks the number of documents", and "don't assign your own number, you must wait". Borrowing from EIPs, the number of a document is simply the number of the PR that proposes the document. This reduces friction, and eliminates a possible bikeshedding vector. -- Laolu On Wed, Jun 30, 2021 at 5:31 PM Ariel Luaces wrote: > BIPs are already the Bazaar style of evolution that simultaneously > allows flexibility and coordination/interoperability (since anyone can > create a BIP and they create an environment of discussion). > > BOLTs are essentially one big BIP in the sense that they started as a > place for discussion but are now more rigid. BOLTs must be followed > strictly to ensure a node is interoperable with the network. And BOLTs > should be rigid, as rigid as any widely used BIP like 32 for example. > Even though BOLTs were flexible when being drafted their purpose has > changed from descriptive to prescriptive. > Any alternatives, or optional features should be extracted out of > BOLTs, written as BIPs. The BIP should then reference the BOLT and the > required flags set, messages sent, or alterations made to signal that > the BIP's feature is enabled. > > A BOLT may at some point organically change to reference a BIP. For > example if a BIP was drafted as an optional feature but then becomes > more widespread and then turns out to be crucial for the proper > operation of the network then a BOLT can be changed to just reference > the BIP as mandatory. There isn't anything wrong with this. > > All of the above would work exactly the same if there was a bLIP > repository instead. I don't see the value in having both bLIPs and > BIPs since AFAICT they seem to be functionally equivalent and BIPs are > not restricted to exclude lightning, and never have been. > > I believe the reason this move to BIPs hasn't happened organically is > because many still perceive the BOLTs available for editing, so > changes continue to be made. If instead BOLTs were perceived as more > "consensus critical", not subject to change, and more people were > strongly encouraged to write specs for new lightning features > elsewhere (like the BIP repo) then you would see this issue of growing > BOLTs resolved. > > Cheers > Ariel Lorenzo-Luaces > > On Wed, Jun 30, 2021 at 1:16 PM Olaoluwa Osuntokun > wrote: > > > > > That being said I think all the points that are addressed in Ryan's > mail > > > could very well be formalized into BOLTs but maybe we just need to > rethink > > > the current process of the BOLTs to make it more accessible for new > ideas > > > to find their way into the BOLTs? > > > > I think part of what bLIPs are trying to solve here is promoting more > loosely > > coupled evolution of the network. I think the BOLTs do a good job > currently of > > specifying what _base_ functionality is required for a routing node in a > > prescriptive manner (you must forward an HTLC like this, etc). However > there's > > a rather large gap in describing functionality that has emerged over > time due > > to progressive evolution, and aren't absolutely necessary, but enhance > > node/wallet operation. > > > > Examples of include things like: path finding heuristics (BOLTs just > say you > > should get from Alice to Bob, but provides no recommendations w.r.t > _how_ to do > > so), fee bumping heuristics, breach retribution handling, channel > management, > > rebalancing, custom records usage (like the podcast index meta-data, > messaging, > > etc), JIT channel opening, hosted channels, randomized channel IDs, fee > > optimization, initial channel boostrapping, etc. > > > > All these examples are effectively optional as they aren't required for > base > > node operation, but they've organically evolved over time as node > > implementations and wallet seek to solve UX and operational problems for > > their users. bLIPs can be a _descriptive_ (this is
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Here's another feature which just appeared and would benefit from a bLIP for compatibility: https://twitter.com/SimpleBtcWallet/status/1410506889545359365 Atomic splitting of bills. A very small thing, but also very cool. I can't imagine it fitting in the BOLTs at all. 2021-06-30 09:10 (GMT-05:00), Ryan Gentry via Lightning-dev said: > Hi all, > The recent thread around zero-conf channels [1] provides an opportunity to > discuss how the BOLT process handles features and best practices that arise in > the wild vs. originating within the process itself. Zero-conf channels are one > of many LN innovations on the app layer that have struggled to make their way > into the spec. John Carvalho and Bitrefill launched Turbo channels in April > 2019 [2], Breez posted their solution to the mailing list for feedback in > August 2020 [3], and we know at least ACINQ and Muun (amongst others) have > their own implementations. In an ideal world there would be a descriptive > design document that the app layer implementers had collaborated on over the > years that the spec group could then pick up and merge into the BOLTs now that > the feature is deemed spec-worthy. > Over the last couple of months, we have discussed the idea of adding a > BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various > members > of the community, and have received positive feedback from both app layer and > protocol devs. This would not affect the existing BOLT process at all, but > simply add a place for app layer best practices to be succinctly described and > organized, especially those that require coordination. These features are > being > built outside of the BOLT process today anyways, so ideally a bLIP process > would bring them into the fold instead of leaving them buried in old ML posts > or not documented at all. > Some potential bLIP ideas that people have mentioned include: each lnurl > variant, on-the-fly channel opens, AMP, dynamic commitments, podcast payment > metadata, p2p messaging formats, new pathfinding heuristics, remote node > connection standards, etc. > If the community is interested in moving forward, we've started a branch [5] > describing such a process. It's based on BIP-0002, so not trying to reinvent > any wheels. It would be great to have developers from various implementations > and from the broader app layer ecosystem volunteer to be listed as editors > (basically the same role as in the BIPs). > Looking forward to hearing your thoughts! > Best, > Ryan > [1] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-June/003074.html > [2] > https://www.coindesk.com/bitrefills-thor-turbo-lets-you-get-started-with-bitcoins-lightning-faster > [3] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-August/002780.html > [4] bLIP = Bitcoin Lightning Improvement Proposal and SPARK = Standardization > of Protocols at the Request of the Kommunity (h/t fiatjaf) > [5] > https://github.com/ryanthegentry/lightning-rfc/blob/blip-0001/blips/blip-0001.mediawiki___ > Lightning-dev mailing list > Lightning-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Thanks for starting that discussion. In my opinion, what we're really trying to address here are the two following points (at least from the point of view of someone who works on the spec and an implementation): - Implementers get frustrated when they've worked on something that they think is useful and they can't get it into the BOLTs (the spec PR isn't reviewed, it progresses too slowly or there isn't enough agreement to merge it) - Implementers expect other implementers to specify the optional features they ship: we don't want to have to reverse-engineer a sub-protocol when users want our implementation to provide support for feature XXX Note that these are two very different concerns. bLIPs/SPARKS/BIPs clearly address the second point, which is good. But they don't address the first point at all, they instead work around it. To be fair, I don't think we can completely address that first point: properly reviewing spec proposals takes a lot of effort and accepting complex changes to the BOLTs shouldn't be done lightly. I am mostly in favor of this solution, but I want to highlight that it isn't only rainbows and unicorns: it will add fragmentation to the network, it will add maintenance costs and backwards-compatibility issues, many bLIPs will be sub-optimal solutions to the problem they try to solve and some bLIPs will be simply insecure and may put users' funds at risk (L2 protocols are hard and have subtle issues that can be easily missed). On the other hand, it allows for real world experimentation and iteration, and it's easier to amend a bLIP than the BOLTs. On the nuts-and-bolts (see the pun?) side, bLIPs cannot embrace a fully bazaar style of evolution. Most of them will need: - to assign feature bit(s) - to insert new tlv fields in existing messages - to create new messages We can't have collisions on any of these three things. bLIP XXX cannot use the same tlv types as bLIP YYY otherwise we're creating network incompatibilities. So they really need to be centralized, and we need a process to assign these and ensure they don't collide. It's not a hard problem, but we need to be clear about the process around those. Regarding the details of where they live, I don't have a strong opinion, but I think they must be easy to find and browse, and I think it's easier for readers if they're inside the spec repository. We already have PRs that use a dedicated "proposals" folder (e.g. [1], [2]). Cheers, Bastien [1] https://github.com/lightningnetwork/lightning-rfc/pull/829 [2] https://github.com/lightningnetwork/lightning-rfc/pull/854 Le jeu. 1 juil. 2021 à 02:31, Ariel Luaces a écrit : > BIPs are already the Bazaar style of evolution that simultaneously > allows flexibility and coordination/interoperability (since anyone can > create a BIP and they create an environment of discussion). > > BOLTs are essentially one big BIP in the sense that they started as a > place for discussion but are now more rigid. BOLTs must be followed > strictly to ensure a node is interoperable with the network. And BOLTs > should be rigid, as rigid as any widely used BIP like 32 for example. > Even though BOLTs were flexible when being drafted their purpose has > changed from descriptive to prescriptive. > Any alternatives, or optional features should be extracted out of > BOLTs, written as BIPs. The BIP should then reference the BOLT and the > required flags set, messages sent, or alterations made to signal that > the BIP's feature is enabled. > > A BOLT may at some point organically change to reference a BIP. For > example if a BIP was drafted as an optional feature but then becomes > more widespread and then turns out to be crucial for the proper > operation of the network then a BOLT can be changed to just reference > the BIP as mandatory. There isn't anything wrong with this. > > All of the above would work exactly the same if there was a bLIP > repository instead. I don't see the value in having both bLIPs and > BIPs since AFAICT they seem to be functionally equivalent and BIPs are > not restricted to exclude lightning, and never have been. > > I believe the reason this move to BIPs hasn't happened organically is > because many still perceive the BOLTs available for editing, so > changes continue to be made. If instead BOLTs were perceived as more > "consensus critical", not subject to change, and more people were > strongly encouraged to write specs for new lightning features > elsewhere (like the BIP repo) then you would see this issue of growing > BOLTs resolved. > > Cheers > Ariel Lorenzo-Luaces > > On Wed, Jun 30, 2021 at 1:16 PM Olaoluwa Osuntokun > wrote: > > > > > That being said I think all the points that are addressed in Ryan's > mail > > > could very well be formalized into BOLTs but maybe we just need to > rethink > > > the current process of the BOLTs to make it more accessible for new > ideas > > > to find their way into the BOLTs? > > > > I think part of what bLIPs are trying to solve here
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
BIPs are already the Bazaar style of evolution that simultaneously allows flexibility and coordination/interoperability (since anyone can create a BIP and they create an environment of discussion). BOLTs are essentially one big BIP in the sense that they started as a place for discussion but are now more rigid. BOLTs must be followed strictly to ensure a node is interoperable with the network. And BOLTs should be rigid, as rigid as any widely used BIP like 32 for example. Even though BOLTs were flexible when being drafted their purpose has changed from descriptive to prescriptive. Any alternatives, or optional features should be extracted out of BOLTs, written as BIPs. The BIP should then reference the BOLT and the required flags set, messages sent, or alterations made to signal that the BIP's feature is enabled. A BOLT may at some point organically change to reference a BIP. For example if a BIP was drafted as an optional feature but then becomes more widespread and then turns out to be crucial for the proper operation of the network then a BOLT can be changed to just reference the BIP as mandatory. There isn't anything wrong with this. All of the above would work exactly the same if there was a bLIP repository instead. I don't see the value in having both bLIPs and BIPs since AFAICT they seem to be functionally equivalent and BIPs are not restricted to exclude lightning, and never have been. I believe the reason this move to BIPs hasn't happened organically is because many still perceive the BOLTs available for editing, so changes continue to be made. If instead BOLTs were perceived as more "consensus critical", not subject to change, and more people were strongly encouraged to write specs for new lightning features elsewhere (like the BIP repo) then you would see this issue of growing BOLTs resolved. Cheers Ariel Lorenzo-Luaces On Wed, Jun 30, 2021 at 1:16 PM Olaoluwa Osuntokun wrote: > > > That being said I think all the points that are addressed in Ryan's mail > > could very well be formalized into BOLTs but maybe we just need to rethink > > the current process of the BOLTs to make it more accessible for new ideas > > to find their way into the BOLTs? > > I think part of what bLIPs are trying to solve here is promoting more loosely > coupled evolution of the network. I think the BOLTs do a good job currently of > specifying what _base_ functionality is required for a routing node in a > prescriptive manner (you must forward an HTLC like this, etc). However there's > a rather large gap in describing functionality that has emerged over time due > to progressive evolution, and aren't absolutely necessary, but enhance > node/wallet operation. > > Examples of include things like: path finding heuristics (BOLTs just say you > should get from Alice to Bob, but provides no recommendations w.r.t _how_ to > do > so), fee bumping heuristics, breach retribution handling, channel management, > rebalancing, custom records usage (like the podcast index meta-data, > messaging, > etc), JIT channel opening, hosted channels, randomized channel IDs, fee > optimization, initial channel boostrapping, etc. > > All these examples are effectively optional as they aren't required for base > node operation, but they've organically evolved over time as node > implementations and wallet seek to solve UX and operational problems for > their users. bLIPs can be a _descriptive_ (this is how things can be done) > home for these types of standards, while BOLTs can be reserved for > _prescriptive_ measures (an HTLC looks like this, etc). > > The protocol as implemented today has a number of extensions (TLVs, message > types, feature bits, etc) that allow implementations to spin out their own > sub-protocols, many of which won't be considered absolutely necessary for node > operation. IMO we should embrace more of a "bazaar" style of evolution, and > acknowledge that loosely coupled evolution allows participants to more broadly > explore the design space, without the constraints of "it isn't a thing until N > of us start to do it". > > Historically, BOLTs have also had a rather monolithic structure. We've used > the same 11 or so documents for the past few years with the size of the > documents swelling over time with new exceptions, features, requirements, > etc. If you were hired to work on a new codebase and saw that everything is > defined in 11 "functions" that have been growing linearly over time, you'd > probably declare the codebase as being unmaintainable. By having distinct > documents for proposals/standards, bLIPs (author documents really), each new > standard/proposal is able to be more effectively explained, motivated, > versionsed, > etc. > > -- Laolu > > > On Wed, Jun 30, 2021 at 7:35 AM René Pickhardt via Lightning-dev > wrote: >> >> Hey everyone, >> >> just for reference when I was new here (and did not understand the processes >> well enough) I proposed a similar idea (called LIP) in 2018 c.f.: >>
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
> That being said I think all the points that are addressed in Ryan's mail > could very well be formalized into BOLTs but maybe we just need to rethink > the current process of the BOLTs to make it more accessible for new ideas > to find their way into the BOLTs? I think part of what bLIPs are trying to solve here is promoting more loosely coupled evolution of the network. I think the BOLTs do a good job currently of specifying what _base_ functionality is required for a routing node in a prescriptive manner (you must forward an HTLC like this, etc). However there's a rather large gap in describing functionality that has emerged over time due to progressive evolution, and aren't absolutely necessary, but enhance node/wallet operation. Examples of include things like: path finding heuristics (BOLTs just say you should get from Alice to Bob, but provides no recommendations w.r.t _how_ to do so), fee bumping heuristics, breach retribution handling, channel management, rebalancing, custom records usage (like the podcast index meta-data, messaging, etc), JIT channel opening, hosted channels, randomized channel IDs, fee optimization, initial channel boostrapping, etc. All these examples are effectively optional as they aren't required for base node operation, but they've organically evolved over time as node implementations and wallet seek to solve UX and operational problems for their users. bLIPs can be a _descriptive_ (this is how things can be done) home for these types of standards, while BOLTs can be reserved for _prescriptive_ measures (an HTLC looks like this, etc). The protocol as implemented today has a number of extensions (TLVs, message types, feature bits, etc) that allow implementations to spin out their own sub-protocols, many of which won't be considered absolutely necessary for node operation. IMO we should embrace more of a "bazaar" style of evolution, and acknowledge that loosely coupled evolution allows participants to more broadly explore the design space, without the constraints of "it isn't a thing until N of us start to do it". Historically, BOLTs have also had a rather monolithic structure. We've used the same 11 or so documents for the past few years with the size of the documents swelling over time with new exceptions, features, requirements, etc. If you were hired to work on a new codebase and saw that everything is defined in 11 "functions" that have been growing linearly over time, you'd probably declare the codebase as being unmaintainable. By having distinct documents for proposals/standards, bLIPs (author documents really), each new standard/proposal is able to be more effectively explained, motivated, versionsed, etc. -- Laolu On Wed, Jun 30, 2021 at 7:35 AM René Pickhardt via Lightning-dev < lightning-dev@lists.linuxfoundation.org> wrote: > Hey everyone, > > just for reference when I was new here (and did not understand the > processes well enough) I proposed a similar idea (called LIP) in 2018 c.f.: > https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-July/001367.html > > > I wonder what exactly has changed in the reasoning by roasbeef which I > will repeat here: > > *> We already have the equiv of improvement proposals: BOLTs. Historically* > > >* new standardization documents are proposed initially as issues or PR's > >when * > > >* ultimately accepted. Why do we need another repo? * > > > As far as I can tell there was always some form of (invisible?) barrier to > participate in the BOLTs but there are also new BOLTs being offered: > * BOLT 12: https://github.com/lightningnetwork/lightning-rfc/pull/798 > * BOLT 14: https://github.com/lightningnetwork/lightning-rfc/pull/780 > and topics to be included like: > * dual funding > * splicing > * the examples given by Ryan > > I don't see how a new repo would reduce that barrier - Actually I think it > would even create more confusion as I for example would not know where > something belongs. That being said I think all the points that are > addressed in Ryan's mail could very well be formalized into BOLTs but maybe > we just need to rethink the current process of the BOLTs to make it more > accessible for new ideas to find their way into the BOLTs? One thing that I > can say from answering lightning-network questions on stackexchange is that > it would certainly help if the BOLTs where referenced on lightning.network > web page and in the whitepaper as the place to be if one wants to learn > about the Lightning Network > > with kind regards Rene > > On Wed, Jun 30, 2021 at 4:10 PM Ryan Gentry via Lightning-dev < > lightning-dev@lists.linuxfoundation.org> wrote: > >> Hi all, >> >> The recent thread around zero-conf channels [1] provides an opportunity >> to discuss how the BOLT process handles features and best practices that >> arise in the wild vs. originating within the process itself. Zero-conf >> channels are one of many LN innovations on the app layer that have >> struggled to make their way into the
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
hello René, I think the idea of having separate standards is good because we can keep the core spec mandatory and other things optional. Since the core spec, defined by the BOLTs, is mandatory, it's better if it's as small as possible, basically barely enough to allow peers to talk to each other and open a channel, then define what an HTLC is and the basic payment flow. All the rest is optional. The BOLTs themselves encourage experimentation by having TLVs, rules for optional and experimental message type numbers and so on. And then it doesn't make sense to put optional things in the BOLTs otherwise no one will be spec-compliant anymore and it will cause confusion. Some things, like splicing and dual-funded channels could be created as blips and after everybody had implemented them moved to the BOLTs, other things, like the podcast tipping protocol, cannot. Still, it is better to have a spec for the podcast tipping protocol than to not have, or to have it hidden somewhere. It makes it more open and easier for everyone. Ultimately I think dual-funded channels, trampoline routing and other lower level things should still be kept out of the BOLTs as long as they are optional. While things like splicing and blinded paths seem to be more like things that should enforced. This is my opinion, but I think it's good to have this clear distinction. Finally, a list of other things that deserve a spec so they are made standard and interoperable across wallets and services: 1. keysend 2. AMP 3. hosted channels 4. trampoline routing v1 5. trampoline routing v2 6. turbo channels 7. podcast tipping protocol 8. dual-funding 9. on-demand channels 10. sphinx chat messaging thing 11. private routing as done by immortan 12. alternative graph for unannounced channels as done by immortan 13. lnurl-withdraw 14. lnurl-pay 15. lnurl-channel 16. bitcoin-liquid lightning bridge 17. I thought I had more but apparently I forgot So we have to hunt these people down and make them submit specs. --- fiatjaf 2021-06-30 16:35 (GMT+02:00), "René Pickhardt via Lightning-dev" said: > Hey everyone, > just for reference when I was new here (and did not understand the processes > well enough) I proposed a similar idea (called LIP) in 2018 c.f.: > https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-July/001367.html > I wonder what exactly has changed in the reasoning by roasbeef which I will > repeat here: > We already have the equiv of improvement proposals: BOLTs. Historically new > standardization documents are proposed initially as issues or PR's when > ultimately accepted. Why do we need another repo? > As far as I can tell there was always some form of (invisible?) barrier to > participate in the BOLTs but there are also new BOLTs being offered: > * BOLT 12: https://github.com/lightningnetwork/lightning-rfc/pull/798 > * BOLT 14: https://github.com/lightningnetwork/lightning-rfc/pull/780 > and topics to be included like: > * dual funding > * splicing > * the examples given by Ryan > I don't see how a new repo would reduce that barrier - Actually I think it > would even create more confusion as I for example would not know where > something belongs. That being said I think all the points that are addressed > in > Ryan's mail could very well be formalized into BOLTs but maybe we just need to > rethink the current process of the BOLTs to make it more accessible for new > ideas to find their way into the BOLTs? One thing that I can say from > answering > lightning-network questions on stackexchange is that it would certainly help > if > the BOLTs where referenced on lightning.network web page and in the whitepaper > as the place to be if one wants to learn about the Lightning Network > with kind regards Rene > On Wed, Jun 30, 2021 at 4:10 PM Ryan Gentry via Lightning-dev wrote: > > > > Hi all, > > > > > The recent thread around zero-conf channels [1] provides an opportunity to > discuss how the BOLT process handles features and best practices that > arise in the wild vs. originating within the process itself. Zero-conf > channels are one of many LN innovations on the app layer that have > struggled to make their way into the spec. John Carvalho and Bitrefill > launched Turbo channels in April 2019 [2], Breez posted their solution to > the mailing list for feedback in August 2020 [3], and we know at least > ACINQ and Muun (amongst others) have their own implementations. In an > ideal world there would be a descriptive design document that the app > layer implementers had collaborated on over the years that the spec group > could then pick up and merge into the BOLTs now that the feature is deemed > spec-worthy. > > > > > Over the last couple of months, we have discussed the idea of adding a > BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various > members of the community, and have received positive feedback from both > app layer and protocol devs. This would not affect the existing BOLT > process at
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Hi Rene, Thank you for the feedback! Very interesting to look back at the same proposal from 2018, we clearly could have done a better job researching past attempts. I have two main comments: 1) not trying to introduce a new repo, the linked lightning-rfc branch [1] simply adds a new bLIPs folder in the existing repo (like you suggested as an option in 2018) 2) major difference between 2018 and now is one of scale (which is a great problem to have!). In 2018 the LN dev ecosystem was mostly ACINQ, Blockstream, and Lightning Labs and the minimalist BOLTs process worked well. At this point the broader ecosystem is significantly bigger than those three teams combined, and it seems the process should adjust to reflect the new environment. The main goal of the suggested change is simply to provide a home for emerging "best practices", especially those that require coordination amongst multiple groups. I think LNURL provides a good example of a "best practice" that has been spec'd out [2], is completely extra protocol so probably doesn't belong as a BOLT, but carries tension with it for new developers since it's been widely adopted yet not "officially supported". What do you think about that? Best, Ryan [1] https://github.com/ryanthegentry/lightning-rfc/blob/blip-0001/blips/blip-0001.mediawiki [2] https://github.com/fiatjaf/lnurl-rfc On Wed, Jun 30, 2021 at 9:35 AM René Pickhardt wrote: > Hey everyone, > > just for reference when I was new here (and did not understand the > processes well enough) I proposed a similar idea (called LIP) in 2018 c.f.: > https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-July/001367.html > > > I wonder what exactly has changed in the reasoning by roasbeef which I > will repeat here: > > *> We already have the equiv of improvement proposals: BOLTs. Historically* > > >* new standardization documents are proposed initially as issues or PR's > >when * > > >* ultimately accepted. Why do we need another repo? * > > > As far as I can tell there was always some form of (invisible?) barrier to > participate in the BOLTs but there are also new BOLTs being offered: > * BOLT 12: https://github.com/lightningnetwork/lightning-rfc/pull/798 > * BOLT 14: https://github.com/lightningnetwork/lightning-rfc/pull/780 > and topics to be included like: > * dual funding > * splicing > * the examples given by Ryan > > I don't see how a new repo would reduce that barrier - Actually I think it > would even create more confusion as I for example would not know where > something belongs. That being said I think all the points that are > addressed in Ryan's mail could very well be formalized into BOLTs but maybe > we just need to rethink the current process of the BOLTs to make it more > accessible for new ideas to find their way into the BOLTs? One thing that I > can say from answering lightning-network questions on stackexchange is that > it would certainly help if the BOLTs where referenced on lightning.network > web page and in the whitepaper as the place to be if one wants to learn > about the Lightning Network > > with kind regards Rene > > On Wed, Jun 30, 2021 at 4:10 PM Ryan Gentry via Lightning-dev < > lightning-dev@lists.linuxfoundation.org> wrote: > >> Hi all, >> >> The recent thread around zero-conf channels [1] provides an opportunity >> to discuss how the BOLT process handles features and best practices that >> arise in the wild vs. originating within the process itself. Zero-conf >> channels are one of many LN innovations on the app layer that have >> struggled to make their way into the spec. John Carvalho and Bitrefill >> launched Turbo channels in April 2019 [2], Breez posted their solution to >> the mailing list for feedback in August 2020 [3], and we know at least >> ACINQ and Muun (amongst others) have their own implementations. In an ideal >> world there would be a descriptive design document that the app layer >> implementers had collaborated on over the years that the spec group could >> then pick up and merge into the BOLTs now that the feature is deemed >> spec-worthy. >> >> Over the last couple of months, we have discussed the idea of adding a >> BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various >> members of the community, and have received positive feedback from both app >> layer and protocol devs. This would not affect the existing BOLT process at >> all, but simply add a place for app layer best practices to be succinctly >> described and organized, especially those that require coordination. These >> features are being built outside of the BOLT process today anyways, so >> ideally a bLIP process would bring them into the fold instead of leaving >> them buried in old ML posts or not documented at all. >> >> Some potential bLIP ideas that people have mentioned include: each lnurl >> variant, on-the-fly channel opens, AMP, dynamic commitments, podcast >> payment metadata, p2p messaging formats, new pathfinding heuristics, remote >> node
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Hey everyone, just for reference when I was new here (and did not understand the processes well enough) I proposed a similar idea (called LIP) in 2018 c.f.: https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-July/001367.html I wonder what exactly has changed in the reasoning by roasbeef which I will repeat here: *> We already have the equiv of improvement proposals: BOLTs. Historically* >* new standardization documents are proposed initially as issues or PR's when * >* ultimately accepted. Why do we need another repo? * As far as I can tell there was always some form of (invisible?) barrier to participate in the BOLTs but there are also new BOLTs being offered: * BOLT 12: https://github.com/lightningnetwork/lightning-rfc/pull/798 * BOLT 14: https://github.com/lightningnetwork/lightning-rfc/pull/780 and topics to be included like: * dual funding * splicing * the examples given by Ryan I don't see how a new repo would reduce that barrier - Actually I think it would even create more confusion as I for example would not know where something belongs. That being said I think all the points that are addressed in Ryan's mail could very well be formalized into BOLTs but maybe we just need to rethink the current process of the BOLTs to make it more accessible for new ideas to find their way into the BOLTs? One thing that I can say from answering lightning-network questions on stackexchange is that it would certainly help if the BOLTs where referenced on lightning.network web page and in the whitepaper as the place to be if one wants to learn about the Lightning Network with kind regards Rene On Wed, Jun 30, 2021 at 4:10 PM Ryan Gentry via Lightning-dev < lightning-dev@lists.linuxfoundation.org> wrote: > Hi all, > > The recent thread around zero-conf channels [1] provides an opportunity to > discuss how the BOLT process handles features and best practices that arise > in the wild vs. originating within the process itself. Zero-conf channels > are one of many LN innovations on the app layer that have struggled to make > their way into the spec. John Carvalho and Bitrefill launched Turbo > channels in April 2019 [2], Breez posted their solution to the mailing list > for feedback in August 2020 [3], and we know at least ACINQ and Muun > (amongst others) have their own implementations. In an ideal world there > would be a descriptive design document that the app layer implementers had > collaborated on over the years that the spec group could then pick up and > merge into the BOLTs now that the feature is deemed spec-worthy. > > Over the last couple of months, we have discussed the idea of adding a > BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various > members of the community, and have received positive feedback from both app > layer and protocol devs. This would not affect the existing BOLT process at > all, but simply add a place for app layer best practices to be succinctly > described and organized, especially those that require coordination. These > features are being built outside of the BOLT process today anyways, so > ideally a bLIP process would bring them into the fold instead of leaving > them buried in old ML posts or not documented at all. > > Some potential bLIP ideas that people have mentioned include: each lnurl > variant, on-the-fly channel opens, AMP, dynamic commitments, podcast > payment metadata, p2p messaging formats, new pathfinding heuristics, remote > node connection standards, etc. > > If the community is interested in moving forward, we've started a branch > [5] describing such a process. It's based on BIP-0002, so not trying to > reinvent any wheels. It would be great to have developers from various > implementations and from the broader app layer ecosystem volunteer to be > listed as editors (basically the same role as in the BIPs). > > Looking forward to hearing your thoughts! > > Best, > Ryan > > [1] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-June/003074.html > > [2] > https://www.coindesk.com/bitrefills-thor-turbo-lets-you-get-started-with-bitcoins-lightning-faster > > [3] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-August/002780.html > > [4] bLIP = Bitcoin Lightning Improvement Proposal and SPARK = > Standardization of Protocols at the Request of the Kommunity (h/t fiatjaf) > > [5] > https://github.com/ryanthegentry/lightning-rfc/blob/blip-0001/blips/blip-0001.mediawiki > ___ > Lightning-dev mailing list > Lightning-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev > -- https://www.rene-pickhardt.de ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Or just use BIPs instead of further fracturing...? On Jun 30, 2021 10:10 AM, Ryan Gentry via Lightning-dev wrote: > > Hi all, > > > The recent thread around zero-conf channels [1] provides an opportunity to > discuss how the BOLT process handles features and best practices that arise > in the wild vs. originating within the process itself. Zero-conf channels are > one of many LN innovations on the app layer that have struggled to make their > way into the spec. John Carvalho and Bitrefill launched Turbo channels in > April 2019 [2], Breez posted their solution to the mailing list for feedback > in August 2020 [3], and we know at least ACINQ and Muun (amongst others) have > their own implementations. In an ideal world there would be a descriptive > design document that the app layer implementers had collaborated on over the > years that the spec group could then pick up and merge into the BOLTs now > that the feature is deemed spec-worthy. > > > Over the last couple of months, we have discussed the idea of adding a > BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various > members of the community, and have received positive feedback from both app > layer and protocol devs. This would not affect the existing BOLT process at > all, but simply add a place for app layer best practices to be succinctly > described and organized, especially those that require coordination. These > features are being built outside of the BOLT process today anyways, so > ideally a bLIP process would bring them into the fold instead of leaving them > buried in old ML posts or not documented at all. > > > Some potential bLIP ideas that people have mentioned include: each lnurl > variant, on-the-fly channel opens, AMP, dynamic commitments, podcast payment > metadata, p2p messaging formats, new pathfinding heuristics, remote node > connection standards, etc. > > > If the community is interested in moving forward, we've started a branch [5] > describing such a process. It's based on BIP-0002, so not trying to reinvent > any wheels. It would be great to have developers from various implementations > and from the broader app layer ecosystem volunteer to be listed as editors > (basically the same role as in the BIPs). > > > Looking forward to hearing your thoughts! > > > Best, > Ryan > > > [1] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-June/003074.html > > [2] > https://www.coindesk.com/bitrefills-thor-turbo-lets-you-get-started-with-bitcoins-lightning-faster > > [3] > https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-August/002780.html > > [4] bLIP = Bitcoin Lightning Improvement Proposal and SPARK = Standardization > of Protocols at the Request of the Kommunity (h/t fiatjaf) > > [5] > https://github.com/ryanthegentry/lightning-rfc/blob/blip-0001/blips/blip-0001.mediawiki ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
[Lightning-dev] bLIPs: A proposal for community-driven app layer and protocol extension standardization
Hi all, The recent thread around zero-conf channels [1] provides an opportunity to discuss how the BOLT process handles features and best practices that arise in the wild vs. originating within the process itself. Zero-conf channels are one of many LN innovations on the app layer that have struggled to make their way into the spec. John Carvalho and Bitrefill launched Turbo channels in April 2019 [2], Breez posted their solution to the mailing list for feedback in August 2020 [3], and we know at least ACINQ and Muun (amongst others) have their own implementations. In an ideal world there would be a descriptive design document that the app layer implementers had collaborated on over the years that the spec group could then pick up and merge into the BOLTs now that the feature is deemed spec-worthy. Over the last couple of months, we have discussed the idea of adding a BIP-style process (bLIPs? SPARKs? [4]) on top of the BOLTs with various members of the community, and have received positive feedback from both app layer and protocol devs. This would not affect the existing BOLT process at all, but simply add a place for app layer best practices to be succinctly described and organized, especially those that require coordination. These features are being built outside of the BOLT process today anyways, so ideally a bLIP process would bring them into the fold instead of leaving them buried in old ML posts or not documented at all. Some potential bLIP ideas that people have mentioned include: each lnurl variant, on-the-fly channel opens, AMP, dynamic commitments, podcast payment metadata, p2p messaging formats, new pathfinding heuristics, remote node connection standards, etc. If the community is interested in moving forward, we've started a branch [5] describing such a process. It's based on BIP-0002, so not trying to reinvent any wheels. It would be great to have developers from various implementations and from the broader app layer ecosystem volunteer to be listed as editors (basically the same role as in the BIPs). Looking forward to hearing your thoughts! Best, Ryan [1] https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-June/003074.html [2] https://www.coindesk.com/bitrefills-thor-turbo-lets-you-get-started-with-bitcoins-lightning-faster [3] https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-August/002780.html [4] bLIP = Bitcoin Lightning Improvement Proposal and SPARK = Standardization of Protocols at the Request of the Kommunity (h/t fiatjaf) [5] https://github.com/ryanthegentry/lightning-rfc/blob/blip-0001/blips/blip-0001.mediawiki ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
