Re: [bitcoin-dev] BIP process friction
Hello, First off, apologies about my lack of participation. I am working on mostly unrelated things and I'm afraid I have failed the community in terms of what I can do on my end to keep the BIP process functional. As such I am hereby resigning as BIP editor effective immediately. Please remove my access privileges to the repository when possible. -Kalle. On Fri, 19 Jan 2024 at 09:46, Anthony Towns via bitcoin-dev wrote: > > On Thu, Jan 18, 2024 at 05:41:14AM -1000, David A. Harding via bitcoin-dev > wrote: > > Question: is there a recommended way to produce a shorter identifier for > > inline use in reading material? For example, for proposal > > BIN-2024-0001-000, I'm thinking: > > > > - BIN24-1 (references whatever the current version of the proposal is) > > - BIN24-1.0 (references revision 0) > > > > I think that doesn't look too bad even if there are over 100 proposals a > > year, with some of them getting into over a hundred revisions: > > > > - BIN24-123 > > - BIN24-123.123 > > Having lived through y2k, two-digit years give me the ick, but otherwise > sure. > > Cheers, > aj, that's how the kids who didn't live through y2k say it, right? > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP process friction
On Thu, Jan 18, 2024 at 05:41:14AM -1000, David A. Harding via bitcoin-dev wrote: > Question: is there a recommended way to produce a shorter identifier for > inline use in reading material? For example, for proposal > BIN-2024-0001-000, I'm thinking: > > - BIN24-1 (references whatever the current version of the proposal is) > - BIN24-1.0 (references revision 0) > > I think that doesn't look too bad even if there are over 100 proposals a > year, with some of them getting into over a hundred revisions: > > - BIN24-123 > - BIN24-123.123 Having lived through y2k, two-digit years give me the ick, but otherwise sure. Cheers, aj, that's how the kids who didn't live through y2k say it, right? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] One-Shot Replace-By-Fee-Rate
Reposting this blog post here for discussion:
https://petertodd.org/2024/one-shot-replace-by-fee-rate
---
layout: post
title: "One-Shot Replace-by-Fee-Rate"
date: 2024-01-18
tags:
- bitcoin
- rbf
- pinning
---
Currently Bitcoin Core implements a Replace-by-Fee (RBF) policy, where
transactions are not replaced unless the new transaction pays at least a higher
total fee than the replaced transaction, regardless of fee-rate.
When RBF was first implemented over 8 years ago this was a reasonable,
conservative, default. However, since then we've found that strictly requiring
a higher absolute fee creates the potential for [transaction
pinning](https://bitcoinops.org/en/topics/transaction-pinning/) attacks in
contracting protocols such as Lightning; replacing transactions based on
*fee-rate* would make it possible[^bip125-rule-5-pinning] to eliminate these
attacks by eliminating BIP-125 Rule #3 pinning.
[^bip125-rule-5-pinning]: The other notable pinning attack against RBF is to
cause BIP-125 Rule #5 to be exceeded. But that is easily solved by just
rejecting transactions that would make transactions already in the mempool to
be irreplaceable.
Here we will propose an incentive compatible solution, One-Shot
Replace-By-Fee-Rate, that mitigates prior concerns over replace-by-fee-rate
policies by allowing the replacement to only happen when it would immediately
bring a transaction close enough to the top of the mempool to be mined in the
next block or so. Finally, we will show that both one-shot and pure
replace-by-fee-rate policies sufficiently resist bandwidth exhaustion attacks
to be implementable.
*Thanks goes to [Fulgur Ventures](https://fulgur.ventures/) for sponsoring this
research. They had no editorial control over the contents of this post and did
not review it prior to publication.*
# Contents
{:.no_toc}
0. TOC
{:toc}
## Background
### The Expected Return of an Unconfirmed Transaction
Suppose there exists a transaction that pays a fee of $$F$$ at a fee-rate $$r$$.
What is the expected return $$E$$ of that transaction to miners as a whole? If
the transaction pays a fee-rate high enough to definitely get mined in the next
block, the answer seems obvious: $$E = F$$. The transaction will be mined, and
miners as a whole will earn the entire $$F$$ fee.
But what if that transaction pays a lower fee-rate? For example, as I write
this, [mempool.space](https://mempool.space) reports that their mempool
contains $$535\mathrm{MvB}$$ of transactions, enough that a typical Bitcoin
Core node with its typical $$300\mathrm{MB}$$ mempool size limit would reject
transactions paying less than $$22.9\frac{\mathrm{sat}}{\mathrm{vB}}$$.
If a transaction has a fee-rate of $$23\frac{\mathrm{sat}}{\mathrm{vB}}$$, just
barely enough to get into a default mempool, what is that transaction worth to
miners? How do we even answer this question?
Intuitively it seems obvious that the low fee-rate transaction should be worth
less than the high fee-rate transaction, because
the low fee-rate transaction probably won't be mined for days, or even weeks,
if ever.
Certainly, in a shorter time frame, a transaction at the bottom of the mempool
does not directly represent income to the miner.
We can think about this a bit more rigorously by observing that because block
finding is a Poisson Process, **even if** we ignore the supply of new
transactions, the probability that a transaction $$n$$ blocks deep is mined in
a time interval $$t$$ is the probability that $$N \ge n$$ blocks are found in
the time interval $$t$$. That probability rapidly diminishes as $$n$$
increases, because it's less and less likely for so many blocks to be found in
a short period of time.
### Unconfirmed Transactions are Honest Signals
Do low fee-rate transactions have a value? Yes!
Assuming your node is well connected, unconfirmed transactions in your mempool
are *honest signals*: because unconfirmed transactions *could* be mined,
they're clear evidence that if you wish your transaction to be mined sooner, you
need to offer an even higher fee-rate. There's a constant supply of people with
high time preference who want their transactions mined in a short period of
time. So low fee-rate transactions indirectly increase the revenue of miners in
the short term, because they force higher time preference transactors to outbid
them.
Note how I said a higher fee-rate, not fee: because it maximizes revenue to mine
transactions in fee-rate order, fee-rate is what matters in terms of priority.
### Expected Return vs Fee-Rate
Suppose we now have *two* different conflicting transactions, $$a$$ and $$b$$.
Suppose that the total size of $$a$$ is $$10\mathrm{vB}$$, and pays
$$23\frac{\mathrm{sat}}{\mathrm{vB}}$$, for a total fee of
$$230\mathrm{sat}$$. Meanwhile $$b$$ has a size of just $$150\mathrm{vB}$$,
and pays $$15000\frac{\mathrm{sat}}{\mathrm{vB}}$$, for a total fee of
$$225\mathrm{sat}$$.
It seems intuitive that transaction $$b$$ is the
Re: [bitcoin-dev] BIP process friction
On Wed, Jan 17, 2024 at 05:29:48PM +, Michael Folkson via bitcoin-dev wrote: > Hey Luke > > I'd be happy to pick up working on BIP 3 again ([0], [1]) in light of this > issue and others that are repeatedly cropping up (e.g. confusion on the > recommended flow for working on proposed consensus changes, when to open a > pull request to bitcoin-inquisition, when to open a pull request to Core, > when to include/exclude activation params etc). > > I don't think there is much I personally disagree with you on with regards to > BIPs but one issue where I do think there is disagreement is on whether > proposed default policy changes can/should be BIPed. > > I previously drafted this on proposed default policy changes [2]: > > "To address problems such as pinning attacks on Lightning and multiparty > protocols (e.g. vaults) there are and will continue to be draft proposals on > changing the default policy rules in Bitcoin Core and/or alternative > implementations. As these proposals are for default policy rules and **not** > consensus rules there is absolutely no obligation for Bitcoin Core and/or > alternative implementations to change their default policy rules nor users to > run any particular policy rules (default or otherwise). The authors of these > draft proposals are clearly recommending what they think the default policy > rules should be and what policy rules users should run but it is merely a > recommendation. There are a lot of moving parts, subtleties and complexities > involved in designing default policy rules so any recommendation(s) to > significantly upgrade default policy rules would benefit from being subject > to a spec process. This would also aid the review of any policy related pull > requests in Bitcoin Core and/or alternative implementations. An interesting > recent case study washttps://github.com/bitcoin/bitcoin/pull/22665and Bitcoin > Core not implementing the exact wording of BIP 125 RBF. In this case (for > various reasons) as a response Bitcoin Core just removed references to BIP > 125 and started documenting the replacement to BIP 125 rules in the Bitcoin > Core repo instead. However, it is my view that recommendations for default > policy rules should be recommendations to all implementations, reviewed by > Lightning and multiparty protocol developers (not just Bitcoin Core) and > hence they would benefit from being standardized and being subject to a > specification process. I will reiterate once again though that policy rules > are **not** consensus rules. Consensus rules are much closer to an obligation > as divergences from consensus rules risk the user being forked off the > blockchain and could ultimately end up in network splits. This does not apply > to policy rules." > > Are you open to adjusting your stance on proposed policy changes being BIPed? > I do think it really stunts work on proposed default policy changes and > people's ability to follow work on these proposals when the specifications > for them are littered all over the place. I've certainly struggled to follow > the latest state of V3 policy proposals without clear reference points for > the latest state of these proposals e.g. [3]. In addition some proposed > consensus change BIPs are starting to want to include sections on policy > (e.g. BIP345, OP_VAULT [4]) where it would be much better if they could just > refer to a separate policy BIP rather than including proposals for both > policy and consensus in the same BIP. BIP-125 is I think an interesting case study. It is a much more fundamental standard than Ordinals or Taproot Assets, in the sense that transaction replacement is expected to be used by essentially all wallets as all wallets have to deal with fee-rate fluctuations; I do not think that Ordinals or Taproot assets are appropriate BIP material due to their niche use-case. The V3 proposal, and ephemeral anchors, would be expected to be used by a wide range of contracting protocols, most notably lightning. This isn't quite as broad usage as BIP-124 RBF. But it is close. And yes, Core making changes to what is essentially BIP125 has an impact: just the other day I ran into someone that didn't know RBF Rule #6 existed, which Core added on top of BIP-125, and had made a mistake in their safety analysis of a protocol due to that. Meanwhile, engineering documentation on V3 is extremely lacking, with basics like worked through use-case examples not being available. We don't even have solid agreement let alone documentation on how Lightning channels are meant to use V3 transactions and what the trade-offs are. And that has lead to mistaken claims, such as overstating(1) the benefit V3 transactions in their current form have for transaction pinning. I don't think V3 necessarily needs a formal BIP. But it would benefit from a proper engineering process where use-cases are actually worked out and analyzed properly. Finally, I think the lesson to be
Re: [bitcoin-dev] BIP process friction
On Thu, Jan 18, 2024 at 04:47:33PM +, alicexbt via bitcoin-dev wrote: > Hi AJ, > > I like the idea and agree with everything you shared in the email except one > thing: > > > So I'm switching inquisition over to having a dedicated "IANA"-ish > > thing that's independent of BIP process nonsense. It's at: > > > > * https://github.com/bitcoin-inquisition/binana > > I think "authority" is a strong word especially in bitcoin and this process > could even work with BINN (Bitcoin Inquisition Numbers And Names). IANA (a > function of ICANN) is different thing altogether which was founded by US > government. Bitcoin Inquisition is based on signet, which is a centralized blockchain for testing run by a central authority whose consensus is based on signatures from that authority. Using the term "authority" in this context is fine. -- https://petertodd.org 'peter'[:-1]@petertodd.org signature.asc Description: PGP signature ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP process friction
Hi AJ, I like the idea and agree with everything you shared in the email except one thing: > So I'm switching inquisition over to having a dedicated "IANA"-ish > thing that's independent of BIP process nonsense. It's at: > > * https://github.com/bitcoin-inquisition/binana I think "authority" is a strong word especially in bitcoin and this process could even work with BINN (Bitcoin Inquisition Numbers And Names). IANA (a function of ICANN) is different thing altogether which was founded by US government. /dev/fd0 floppy disk guy Sent with Proton Mail secure email. On Wednesday, January 17th, 2024 at 2:42 AM, Anthony Towns via bitcoin-dev wrote: > Hi all, > > Just under three years ago there was some discussion about the BIPs repo, > with the result that Kalle became a BIPs editor in addition to Luke, eg: > > * https://gnusha.org/bitcoin-core-dev/2021-04-22.log > * > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-April/018859.html > > It remains, however, quite hard to get BIPs merged into the repo, eg > the following PRs have been open for quite some time: > > * #1408: Ordinal Numbers; opened 2023-01-21, editors comments: > Kalle: > https://github.com/bitcoin/bips/pull/1408#issuecomment-1421641390 > https://github.com/bitcoin/bips/pull/1408#issuecomment-1435389476 > > Luke: > https://github.com/bitcoin/bips/pull/1408#issuecomment-1429146796 > https://github.com/bitcoin/bips/pull/1408#issuecomment-1438831607 > https://github.com/bitcoin/bips/pull/1408#issuecomment-1465016571 > > * #1489: Taproot Assets Protocol; opened 2023-09-07, editors comments: > Kalle: https://github.com/bitcoin/bips/pull/1489#issuecomment-1855079626 > Luke: https://github.com/bitcoin/bips/pull/1489#issuecomment-1869721535j > > * #1500: OP_TXHASH; opened 2023-09-30, editors comments: > Luke: > https://github.com/bitcoin/bips/pull/1500#pullrequestreview-1796550166 > https://twitter.com/LukeDashjr/status/1735701932520382839 > > The range of acceptable BIPs seems to also be becoming more limited, > such that mempool/relay policy is out of scope: > > * https://github.com/bitcoin/bips/pull/1524#issuecomment-1869734387 > > Despite having two editors, only Luke seems to be able to assign new > numbers to BIPs, eg: > > * https://github.com/bitcoin/bips/pull/1458#issuecomment-1597917780 > > There's also been some not very productive delays due to the editors > wanting backwards compatibility sections even if authors don't think > that's necessary, eg: > > * https://github.com/bitcoin/bips/pull/1372#issuecomment-1439132867 > > Even working out whether to go back to allowing markdown as a text format > is a multi-month slog due to process confusion: > > * https://github.com/bitcoin/bips/pull/1504 > > Anyway, while it's not totally dysfunctional, it's very high friction. > > There are a variety of recent proposals that have PRs open against > inquisition; up until now I've been suggesting people write a BIP, and > have been keying off the BIP number to signal activation. But that just > seems to be introducing friction, when all I need is a way of linking > an arbitrary number to a spec. > > So I'm switching inquisition over to having a dedicated "IANA"-ish > thing that's independent of BIP process nonsense. It's at: > > * https://github.com/bitcoin-inquisition/binana > > If people want to use it for bitcoin-related proposals that don't have > anything to do with inquisition, that's fine; I'm intending to apply the > policies I think the BIPs repo should be using, so feel free to open a PR, > even if you already know I think your idea is BS on its merits. If someone > wants to write an automatic-merge-bot for me, that'd also be great. > > If someone wants to reform the BIPs repo itself so it works better, > that'd be even better, but I'm not volunteering for that fight. > > Cheers, > aj > > (It's called "numbers and names" primarily because that way the acronym > amuses me, but also in case inquisition eventually needs an authoritative > dictionary for what "cat" or "txhash" or similar terms refer to) > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP process friction
On 2024-01-16 16:42, Anthony Towns via bitcoin-dev wrote: I'm switching inquisition over to having a dedicated "IANA"-ish thing that's independent of BIP process nonsense. It's at: * https://github.com/bitcoin-inquisition/binana If people want to use it for bitcoin-related proposals that don't have anything to do with inquisition, that's fine Thank you for doing this! Question: is there a recommended way to produce a shorter identifier for inline use in reading material? For example, for proposal BIN-2024-0001-000, I'm thinking: - BIN24-1 (references whatever the current version of the proposal is) - BIN24-1.0 (references revision 0) I think that doesn't look too bad even if there are over 100 proposals a year, with some of them getting into over a hundred revisions: - BIN24-123 - BIN24-123.123 Rationale: - Using "BIN" for both full-length and shortened versions makes it explicit which document set we're talking about - Eliminating the first dash losslessly saves space and reduces visual clutter - Shortening a four-digit year to two digits works for the next 75 years. Adding more digits as necessary after that won't produce any ambiguity - Although I'd like to eliminate the second dash, and it wouldn't introduce any ambiguity in machine parsing for the next 175 years, I think it would lead to people interpreting numbers incorrectly. E.g., "BIN241" would be read "BIN two-hundred fourty-one" instead of a more desirable "BIN twenty-four dash one" - Eliminating prefix zeroes in the proposal and revision numbers losslessly saves space and reduces visual clutter - A decimal point between the proposal number and revision number creates less visual clutter than the third dash and still conveys the intended meaning - Overall, for the typical case I'd expect---BIN proposals numbered 1-99 with no mention of revision---this produces strings only one or two or characters longer than a typical modern BIP number in shortened format, e.g. BIN24-12 versus BIP123. Thoughts? -Dave ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP process friction
Hey Luke I'd be happy to pick up working on BIP 3 again ([0], [1]) in light of this issue and others that are repeatedly cropping up (e.g. confusion on the recommended flow for working on proposed consensus changes, when to open a pull request to bitcoin-inquisition, when to open a pull request to Core, when to include/exclude activation params etc). I don't think there is much I personally disagree with you on with regards to BIPs but one issue where I do think there is disagreement is on whether proposed default policy changes can/should be BIPed. I previously drafted this on proposed default policy changes [2]: "To address problems such as pinning attacks on Lightning and multiparty protocols (e.g. vaults) there are and will continue to be draft proposals on changing the default policy rules in Bitcoin Core and/or alternative implementations. As these proposals are for default policy rules and **not** consensus rules there is absolutely no obligation for Bitcoin Core and/or alternative implementations to change their default policy rules nor users to run any particular policy rules (default or otherwise). The authors of these draft proposals are clearly recommending what they think the default policy rules should be and what policy rules users should run but it is merely a recommendation. There are a lot of moving parts, subtleties and complexities involved in designing default policy rules so any recommendation(s) to significantly upgrade default policy rules would benefit from being subject to a spec process. This would also aid the review of any policy related pull requests in Bitcoin Core and/or alternative implementations. An interesting recent case study washttps://github.com/bitcoin/bitcoin/pull/22665and Bitcoin Core not implementing the exact wording of BIP 125 RBF. In this case (for various reasons) as a response Bitcoin Core just removed references to BIP 125 and started documenting the replacement to BIP 125 rules in the Bitcoin Core repo instead. However, it is my view that recommendations for default policy rules should be recommendations to all implementations, reviewed by Lightning and multiparty protocol developers (not just Bitcoin Core) and hence they would benefit from being standardized and being subject to a specification process. I will reiterate once again though that policy rules are **not** consensus rules. Consensus rules are much closer to an obligation as divergences from consensus rules risk the user being forked off the blockchain and could ultimately end up in network splits. This does not apply to policy rules." Are you open to adjusting your stance on proposed policy changes being BIPed? I do think it really stunts work on proposed default policy changes and people's ability to follow work on these proposals when the specifications for them are littered all over the place. I've certainly struggled to follow the latest state of V3 policy proposals without clear reference points for the latest state of these proposals e.g. [3]. In addition some proposed consensus change BIPs are starting to want to include sections on policy (e.g. BIP345, OP_VAULT [4]) where it would be much better if they could just refer to a separate policy BIP rather than including proposals for both policy and consensus in the same BIP. Thanks for your long and continued work on the BIP process. It is thankless work and we don't see the alternative futures where all sorts of garbage was merged and given BIP numbers because the BIP editors just merged everything without looking at it and not caring about quality standards. Thanks Michael [0]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019412.html [1]: https://github.com/bitcoin/bips/wiki/BIP-Process-wishlist [2]: https://github.com/bitcoin/bips/wiki/BIP-Process-wishlist#default-policy-changes-eg-v3-a-recommendation-but-not-an-obligation-for-bitcoin-implementations [3]: https://bitcoin.stackexchange.com/questions/117315/what-and-where-are-the-current-status-of-the-bip-125-replacement-the-v3-policy [4]: https://github.com/bitcoin/bips/pull/1421 -- Michael Folkson Email: michaelfolkson at [protonmail.com](http://protonmail.com/) GPG: A2CF5D71603C92010659818D2A75D601B23FEE0F Learn about Bitcoin: https://www.youtube.com/@portofbitcoin On Wednesday, 17 January 2024 at 16:45, Luke Dashjr via bitcoin-dev wrote: > Perhaps a BIP 3 is in order, but most of the real issue is simply a matter of > volunteer time. > > AJ's attempt to conflate that with his own personal disagreements with how > BIPs have always worked, is unrelated. > > Luke > > On 1/17/24 01:55, Christopher Allen via bitcoin-dev wrote: > >> On Tue, Jan 16, 2024 at 6:43 PM Anthony Towns via bitcoin-dev >> wrote: >> >>> If people want to use it for bitcoin-related proposals that don't have >>> anything to do with inquisition, that's fine; I'm intending to apply the >>> policies I think the BIPs repo should be
Re: [bitcoin-dev] Compressed Bitcoin Transactions
One point to add here is that, while V1 non-encrypted p2p traffic could be compressed on a different OSI layer in theory, v2 encrypted traffic – due to its pseudorandom nature – will likely have no size savings and thus need to be compressed on the application layer with a proposal like this. Would be nice to see size comparison of this compression proposal vs LZO/gzip compression of legacy transaction encoding. A possible advantage of this proposal is that it could save more space with less CPU impact, which might be important for block propagation. Previous discussion about compressing blocks before sending them: https://github.com/bitcoin/bitcoin/pull/6973 /jonas > Am 16.01.2024 um 18:08 schrieb Tom Briar via bitcoin-dev > : > > Hi, > > In addition to the use cases listed in the schema, such as steganography, > satellite, and radio broadcast, an application can be made for Peer-to-peer > communication between Bitcoin nodes. Except when compressing the Txid/Vout, > which is optional, Transactions can gain up to 30% size savings while still > being completely reversible. Furthermore, in a BIP-324 world, these savings > are nontrivial. > > BIP-324: https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki > Compressed Transaction Schema: compressed_transactions.md > > Thanks- > Tom. > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
