Re: [bitcoin-dev] Package Relay Proposal
Hi Tom, Yesterday a PR was opened to do just that, with caveats: https://github.com/bitcoin/bitcoin/pull/27609 For higher level tracking of the project: https://github.com/bitcoin/bitcoin/issues/27463 Cheers, Greg On Wed, May 10, 2023 at 11:39 AM Tom Trevethan via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > The submitpackage RPC is available on regtest in the current core > release. Is there any plan or timeline for deploying this on mainnet in the > next release? Can't find any recent discussion. It would be very helpful > given current (and likely future) issues with mempool purge. > > Tom > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Package Relay Proposal
Hi everyone, I've made some significant changes to my package relay proposal based on observations while implementing, feedback on this thread, and offline discussions [1]. The new proposal is called Ancestor Package Relay, BIP331, and PR'd at https://github.com/bitcoin/bips/pull/1382 The major changes to the proposal are: 1. Scope reduction to receiver-initiated only 2. Scope reduction to ancestor packages only 3. Removal of block hash from package information 1. Scope reduction to receiver-initiated only Receiver-intiated package relay enables a node to ask for more information when they suspect they are missing something (i.e. in this case to resolve a missing parent tx). Sender-initiated package relay should, theoretically, save a round trip by notifying the receiver ahead of time that "hey, this is going to be a package, so make sure you download and submit these transactions together." As with any proactive communication, there is a chance that the node already knows this information, so this network bandwidth was wasted. The logic used to decide _when_ to announce a package proactively determines whether it is a net increase or decrease for overall bandwidth usage. However, it's difficult to design anything to save bandwidth without any idea of what its bandwidth usage actually looks like in practice. We'll want to design the sender-initiated protocol carefully, and inform the design decisions using data collected from the mainnet p2p network. However, there is no historical transaction data to use because the goal is to enable currently-rejected transactions to propagate. In order to get this right, I propose we hold off on sender-initiated for now, deploy receiver-initiated package relay, observe its usage and figure out where we can save a round trip, and then produce a well-researched sender-initiated package relay proposal. 2. Scope reduction to ancestor packages only The proposal now only includes ancestor packages (previously called tx-with-unconfirmed-ancestors or "v2" packages). The child-with-unconfirmed-parents (previously called "v1") package has been removed since it is a subset of ancestor packages and sender-initiated relay has been removed. It may be relevant again in the future with sender-initiated packages. If you were reviewing the previous proposal, "pkginfo2" message has been renamed to "ancpkginfo" and "MSG_PKGINFO2" inv type to "MSG_ANCPKGINFO". 3. Removal of block hash from package information Most of the rationale is already on this thread. The block hash was an attempt to enforce topology when chainstates differ, but isn't worth it. It does not make much sense to drop or delay transaction data requests due to mismatched chainstates, and the chainstate may change again between package information and transaction data rounds. Instead, differences in chainstate should be handled internally at the mempool validation level. The node should de-duplicate recently-confirmed transactions and make a best effort to validate the transactions it has already downloaded. Thanks, Gloria [1] https://diyhpl.us/wiki/transcripts/bitcoin-core-dev-tech/2022-10-11-package-relay/ On Fri, Jun 17, 2022 at 9:08 PM Antoine Riard wrote: > Hi Gloria, > > Thanks for working on that, > > > Always overestimating fees may sidestep this issue temporarily (while > mempool > > traffic is low and predictable), but this solution is not foolproof > > and wastes users' money. The feerate market can change due to sudden > > spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or > > sustained, high volume of Bitcoin payments (e.g. April 2021 and > > December 2017). > > Even if the LN implementations started to overestimate fees based on the > historical worst-case of block inclusion feerates, there is still room for > exploitation due to bip125 rule#3. Indeed, as long as the adversary is able > to stick in the mempool a higher fee package while the feerate is not > compelling enough to get it mined, your "honest" LN package should be > bounced off. > > Considering Core's `MAX_STANDARD_TX_WEIGHT` of 40 WU, I think it's > practical for an attacker to succeed with this pinning tactic in periods of > traffic spikes. Of course, LN implementation could overestimate fees with a > target like `MAX_STANDARD_WEIGHT` * `worst_case_block_inclusion_feerate` to > mitigate. However, assuming a value of 20sat for the latter, it would > require from any LN user a minimal channel value of 200 satoshis to be > theoretically secure against this type of pinning. > > So package relay is required to mitigate efficiently and realistically > against pinning attacks, while conserving the same level of "economic" > openness for Lightning. Beyond, it should be also noted that package relay > is only building block of the full set of mitigations, and there should be > a yet to-find-consensus-as-of-today other policy change such as > user-elected package limits or replace-by-feerate. > > Anyway, I think it would
Re: [bitcoin-dev] Package Relay Proposal
Hi Gloria,
Thanks for working on that,
> Always overestimating fees may sidestep this issue temporarily (while
mempool
> traffic is low and predictable), but this solution is not foolproof
> and wastes users' money. The feerate market can change due to sudden
> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
> sustained, high volume of Bitcoin payments (e.g. April 2021 and
> December 2017).
Even if the LN implementations started to overestimate fees based on the
historical worst-case of block inclusion feerates, there is still room for
exploitation due to bip125 rule#3. Indeed, as long as the adversary is able
to stick in the mempool a higher fee package while the feerate is not
compelling enough to get it mined, your "honest" LN package should be
bounced off.
Considering Core's `MAX_STANDARD_TX_WEIGHT` of 40 WU, I think it's
practical for an attacker to succeed with this pinning tactic in periods of
traffic spikes. Of course, LN implementation could overestimate fees with a
target like `MAX_STANDARD_WEIGHT` * `worst_case_block_inclusion_feerate` to
mitigate. However, assuming a value of 20sat for the latter, it would
require from any LN user a minimal channel value of 200 satoshis to be
theoretically secure against this type of pinning.
So package relay is required to mitigate efficiently and realistically
against pinning attacks, while conserving the same level of "economic"
openness for Lightning. Beyond, it should be also noted that package relay
is only building block of the full set of mitigations, and there should be
a yet to-find-consensus-as-of-today other policy change such as
user-elected package limits or replace-by-feerate.
Anyway, I think it would be beneficial to document the design trade-offs of
pinning mitigations in the `Rationale` subsection, at the attention of
future L2s devs and users ?
> {|
> | Field Name || Type || Size || Purpose
> |-
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |-
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |-
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.
> |-
> |}
It's unclear to me what's the purpose of `max_count` and `max_weight` in
the overall package relay flow, if they are intended to be exposed as
configurable settings to node operators. If those fields are present to
allow DoS protection increase of low-performance host, I believe it would
be better to restrain the number of consumed UTXOs or executed sigops per
package, as DoS vectors are more likely to be CPU-based, rather than
memory-based as package size already bounded at acceptance by
`MAX_PACKAGE_COUNT`.
Thinking more we might introduce a `MAX_SIGOPS_PER_PACKAGR` limit, as
otherwise if we naively grant one package announcement as equal to one
transaction announcement in our tx-request logic, we might increase our DoS
surface, node ressources staying equivalent ?
> {|
> | Field Name || Type || Size || Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> requested.
I'm not sure if we'll ever allow 3-bytes of package size, that would be
~32k of transactions.
> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
in
> the package.
> |}
I think there is a bandwidth consumption trade-off to be aware of in the
function of the package-relay usage. Let's consider a single issuer
broadcasting the package to spend a shared-utxo, after the first shot the
parent component should be spread across the network mempools. At each
fee-bump, only the bumped CPFP will propagate on the network, the parent
wtxid is reannounced in `pckginfo1` though there is no need to fetch it
redundantly and waste bandwidth.
However, I think the bandwidth saving does not hold in case of competing
transaction issuers to spend a shared-utxo. In that case, the parent might
differ at each broadcast and the list of wtxid is dissemblable at every
claim of the shared-utxo. We could save the 32 bytes * number of packages
elements by announcing a package_id, computed from the list of wtxids.
I don't know about the occurrence of competing broadcasts among LN
non-cooperative closes, where bandwidth could be potentially saved. I would
say it's likely low because IIRC there is nothing in the LN protocol where
the counterparties signal to each other they're going on-chain to introduce
a competing broadcast synchronizing event. That said, it might increase in
the future in a post-eltoo, multi-party contracting protocol world.
So it might be interesting to document this design trade-off, if we seek
bandwidth optimizations in function of a changing landscape in the type of
transaction issuers in the future.
> 3. The sender provides package information using "pckginfo1",
>including the blockhash of the sender's best block, the wtxids of
> the
Re: [bitcoin-dev] Package Relay Proposal
uarantee, >> not an intended use case for package relay, and not worsened by this. >> >> Thanks for your feedback! >> >> Best, >> Gloria >> >> [1]: >> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#cmpctblock >> [2]: >> https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.cpp#L49 >> [3]: >> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#short-transaction-id-calculation >> >> On Thu, May 26, 2022 at 3:59 AM wrote: >> >>> Given that packages have no header, the package requires identity in a >>> BIP152 scheme. For example 'header' and 'blockhash' fields can be >>> replaced >>> with a Merkle root (e.g. "identity" field) for the package, uniquely >>> identifying the partially-ordered set of txs. And use of 'getdata' (to >>> obtain a package by hash) can be eliminated (not a use case). >>> >>> e >>> >>> > -Original Message- >>> > From: e...@voskuil.org >>> > Sent: Wednesday, May 25, 2022 1:52 PM >>> > To: 'Anthony Towns' ; 'Bitcoin Protocol Discussion' >>> > ; 'Gloria Zhao' >>> > >>> > Subject: RE: [bitcoin-dev] Package Relay Proposal >>> > >>> > > From: bitcoin-dev On >>> > Behalf >>> > > Of Anthony Towns via bitcoin-dev >>> > > Sent: Wednesday, May 25, 2022 11:56 AM >>> > >>> > > So the other thing is what happens if the peer announcing packages >>> to us >>> > is >>> > > dishonest? >>> > > >>> > > They announce pkg X, say X has parents A B C and the fee rate is >>> garbage. >>> > But >>> > > actually X has parent D and the fee rate is excellent. Do we request >>> the >>> > > package from another peer, or every peer, to double check? Otherwise >>> > we're >>> > > allowing the first peer we ask about a package to censor that tx from >>> us? >>> > > >>> > > I think the fix for that is just to provide the fee and weight when >>> > announcing >>> > > the package rather than only being asked for its info? Then if one >>> peer >>> > makes >>> > > it sound like a good deal you ask for the parent txids from them, >>> dedupe, >>> > > request, and verify they were honest about the parents. >>> > >>> > Single tx broadcasts do not carry an advertised fee rate, however the' >>> > feefilter' message (BIP133) provides this distinction. This should be >>> > interpreted as applicable to packages. Given this message there is no >>> reason >>> > to send a (potentially bogus) fee rate with every package. It can only >>> be >>> > validated by obtaining the full set of txs, and the only recourse is >>> > dropping (etc.) the peer, as is the case with single txs. Relying on >>> the >>> > existing message is simpler, more consistent, and more efficient. >>> > >>> > > >> Is it plausible to add the graph in? >>> > > >>> > > Likewise, I think you'd have to have the graph info from many nodes >>> if >>> > you're >>> > > going to make decisions based on it and don't want hostile peers to >>> be >>> > able to >>> > > trick you into ignoring txs. >>> > > >>> > > Other idea: what if you encode the parent txs as a short hash of the >>> wtxid >>> > > (something like bip152 short ids? perhaps seeded per peer so >>> collisions >>> > will >>> > > be different per peer?) and include that in the inv announcement? >>> Would >>> > > that work to avoid a round trip almost all of the time, while still >>> giving >>> > you >>> > > enough info to save bw by deduping parents? >>> > >>> > As I suggested earlier, a package is fundamentally a compact block (or >>> > block) announcement without the header. Compact block (BIP152) >>> > announcement >>> > is already well-defined and widely implemented. A node should never be >>> > required to retain an orphan, and BIP152 ensures this is not required. >>> > >>> > Once a validated set of txs within the package has been obtained with >>> > sufficient fee, a fee-optimal node would accept the largest subgraph of >>
Re: [bitcoin-dev] Package Relay Proposal
s://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#cmpctblock > [2]: > https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.cpp#L49 > [3]: > https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#short-transaction-id-calculation > > On Thu, May 26, 2022 at 3:59 AM wrote: > >> Given that packages have no header, the package requires identity in a >> BIP152 scheme. For example 'header' and 'blockhash' fields can be replaced >> with a Merkle root (e.g. "identity" field) for the package, uniquely >> identifying the partially-ordered set of txs. And use of 'getdata' (to >> obtain a package by hash) can be eliminated (not a use case). >> >> e >> >> > -Original Message- >> > From: e...@voskuil.org >> > Sent: Wednesday, May 25, 2022 1:52 PM >> > To: 'Anthony Towns' ; 'Bitcoin Protocol Discussion' >> > ; 'Gloria Zhao' >> > >> > Subject: RE: [bitcoin-dev] Package Relay Proposal >> > >> > > From: bitcoin-dev On >> > Behalf >> > > Of Anthony Towns via bitcoin-dev >> > > Sent: Wednesday, May 25, 2022 11:56 AM >> > >> > > So the other thing is what happens if the peer announcing packages to >> us >> > is >> > > dishonest? >> > > >> > > They announce pkg X, say X has parents A B C and the fee rate is >> garbage. >> > But >> > > actually X has parent D and the fee rate is excellent. Do we request >> the >> > > package from another peer, or every peer, to double check? Otherwise >> > we're >> > > allowing the first peer we ask about a package to censor that tx from >> us? >> > > >> > > I think the fix for that is just to provide the fee and weight when >> > announcing >> > > the package rather than only being asked for its info? Then if one >> peer >> > makes >> > > it sound like a good deal you ask for the parent txids from them, >> dedupe, >> > > request, and verify they were honest about the parents. >> > >> > Single tx broadcasts do not carry an advertised fee rate, however the' >> > feefilter' message (BIP133) provides this distinction. This should be >> > interpreted as applicable to packages. Given this message there is no >> reason >> > to send a (potentially bogus) fee rate with every package. It can only >> be >> > validated by obtaining the full set of txs, and the only recourse is >> > dropping (etc.) the peer, as is the case with single txs. Relying on the >> > existing message is simpler, more consistent, and more efficient. >> > >> > > >> Is it plausible to add the graph in? >> > > >> > > Likewise, I think you'd have to have the graph info from many nodes if >> > you're >> > > going to make decisions based on it and don't want hostile peers to be >> > able to >> > > trick you into ignoring txs. >> > > >> > > Other idea: what if you encode the parent txs as a short hash of the >> wtxid >> > > (something like bip152 short ids? perhaps seeded per peer so >> collisions >> > will >> > > be different per peer?) and include that in the inv announcement? >> Would >> > > that work to avoid a round trip almost all of the time, while still >> giving >> > you >> > > enough info to save bw by deduping parents? >> > >> > As I suggested earlier, a package is fundamentally a compact block (or >> > block) announcement without the header. Compact block (BIP152) >> > announcement >> > is already well-defined and widely implemented. A node should never be >> > required to retain an orphan, and BIP152 ensures this is not required. >> > >> > Once a validated set of txs within the package has been obtained with >> > sufficient fee, a fee-optimal node would accept the largest subgraph of >> the >> > package that conforms to fee constraints and drop any peer that >> provides a >> > package for which the full graph does not. >> > >> > Let us not reinvent the wheel and/or introduce accidental complexity. I >> see >> > no reason why packaging is not simply BIP152 without the 'header' field, >> an >> > updated protocol version, and the following sort of changes to names: >> > >> > sendpkg >> > MSG_CMPCT_PKG >> > cmpctpkg >> > getpkgtxn >> > pkgtxn >> > >> >
Re: [bitcoin-dev] Package Relay Proposal
, 2022 1:52 PM > > To: 'Anthony Towns' ; 'Bitcoin Protocol Discussion' > > ; 'Gloria Zhao' > > > > Subject: RE: [bitcoin-dev] Package Relay Proposal > > > > > From: bitcoin-dev On > > Behalf > > > Of Anthony Towns via bitcoin-dev > > > Sent: Wednesday, May 25, 2022 11:56 AM > > > > > So the other thing is what happens if the peer announcing packages to > us > > is > > > dishonest? > > > > > > They announce pkg X, say X has parents A B C and the fee rate is > garbage. > > But > > > actually X has parent D and the fee rate is excellent. Do we request > the > > > package from another peer, or every peer, to double check? Otherwise > > we're > > > allowing the first peer we ask about a package to censor that tx from > us? > > > > > > I think the fix for that is just to provide the fee and weight when > > announcing > > > the package rather than only being asked for its info? Then if one peer > > makes > > > it sound like a good deal you ask for the parent txids from them, > dedupe, > > > request, and verify they were honest about the parents. > > > > Single tx broadcasts do not carry an advertised fee rate, however the' > > feefilter' message (BIP133) provides this distinction. This should be > > interpreted as applicable to packages. Given this message there is no > reason > > to send a (potentially bogus) fee rate with every package. It can only be > > validated by obtaining the full set of txs, and the only recourse is > > dropping (etc.) the peer, as is the case with single txs. Relying on the > > existing message is simpler, more consistent, and more efficient. > > > > > >> Is it plausible to add the graph in? > > > > > > Likewise, I think you'd have to have the graph info from many nodes if > > you're > > > going to make decisions based on it and don't want hostile peers to be > > able to > > > trick you into ignoring txs. > > > > > > Other idea: what if you encode the parent txs as a short hash of the > wtxid > > > (something like bip152 short ids? perhaps seeded per peer so collisions > > will > > > be different per peer?) and include that in the inv announcement? Would > > > that work to avoid a round trip almost all of the time, while still > giving > > you > > > enough info to save bw by deduping parents? > > > > As I suggested earlier, a package is fundamentally a compact block (or > > block) announcement without the header. Compact block (BIP152) > > announcement > > is already well-defined and widely implemented. A node should never be > > required to retain an orphan, and BIP152 ensures this is not required. > > > > Once a validated set of txs within the package has been obtained with > > sufficient fee, a fee-optimal node would accept the largest subgraph of > the > > package that conforms to fee constraints and drop any peer that provides > a > > package for which the full graph does not. > > > > Let us not reinvent the wheel and/or introduce accidental complexity. I > see > > no reason why packaging is not simply BIP152 without the 'header' field, > an > > updated protocol version, and the following sort of changes to names: > > > > sendpkg > > MSG_CMPCT_PKG > > cmpctpkg > > getpkgtxn > > pkgtxn > > > > > > For a maximum 25 transactions, > > > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package. > > > > > > If you're doing short ids that's maybe 25*4B=100B already, then the > above > > is > > > up to 36% overhead, I guess. Might be worth thinking more about, but > > maybe > > > more interesting with ancestors than just parents. > > > > > > >Also side note, since there are no size/count params, > > > > Size is restricted in the same manner as block and transaction > broadcasts, > > by consensus. If the fee rate is sufficient there would be no reason to > > preclude any valid size up to what can be mined in one block (packaging > > across blocks is not economically rational under the assumption that one > > miner cannot expect to mine multiple blocks in a row). Count is > incorporated > > into BIP152 as 'shortids_length'. > > > > > > wondering if we > > > >should just have "version" in "sendpackages" be a bit field instead of > > > >sending a message for each version. 32 versions should be enough > right
Re: [bitcoin-dev] Package Relay Proposal
Hi aj, answering slightly out of order: > what happens if the peer announcing packages to us is dishonest? > They announce pkg X, say X has parents A B C and the fee rate is garbage. But actually X has parent D and the fee rate is excellent. Do we request the package from another peer, or every peer, to double check? Otherwise we're allowing the first peer we ask about a package to censor that tx from us? Yes, providing false information shouldn't be worse than not announcing the package at all, otherwise we have a censorship vector. In general, the request logic should not let one peer prevent us from requesting a similar announcement from another peer. Yes I was indeed expecting that we would ask for package info from everyone who announces it until it accepts the package or has full information. I can see that it's a fair bit of messages (request pckginfo, oh it's low fee, request pckginfo from somebody else), but we also need to track announcements / potentially go through the same circle to handle "notfound"s, right? In normal running, the fee filter should stop a bunch of honest nodes from telling us packages that are low fee. > I think the fix for that is just to provide the fee and weight when announcing the package rather than only being asked for its info? Then if one peer makes it sound like a good deal you ask for the parent txids from them, dedupe, request, and verify they were honest about the parents. > Likewise, I think you'd have to have the graph info from many nodes if you're going to make decisions based on it and don't want hostile peers to be able to trick you into ignoring txs. I don't think providing more information up front can ever sufficiently resolve the censorship issue. If we want to prevent any one peer from being able to censor requests to other peers, we need to store all announcements and be prepared to request from everybody. Would it be better if we just took out the fee information and had "pckginfo" only consist of transaction ids? Sender tries its best to apply the fee filter? Presumably you have a txInventoryKnown of your peer based on what they've announced to you... just take the ancestor set of a transaction, subtract what they already have, and apply the fee filter to that? Or some kind of algorithm that ensures we don't underestimate? If it's imperfect, the worst case is the receiver downloads a few transactions and rejects them. Given that our goal is just to avoid this case, perhaps opting for simplicity is better than adding a topology graph serialization/deserialization + feerate assessment algorithm on top of this protocol...? >>We'd erroneously ask for A+B+C+X, but really we should only take A+B. >>But wouldn't A+B also be a package that was announced for B? > In theory, yes, but maybe it was announced earlier (while our node was down?) or had dropped from our mempool or similar, either way we don't have those txs yet. Hm. It's fine if they have Erlay, since a sender would know in advance that B is missing and announce it as a package. A potential tack-on solution would be to request package information whenever you have a "low fee" error on a parent and "missing inputs" on a child. Or we solve it at the validation level - instead of submitting each tx individually, we submit each ancestor subset. Do you think any of these is sufficient? At least the package properly propagates across nodes which are online when it is broadcasted... Best, Gloria On Wed, May 25, 2022 at 11:55 AM Anthony Towns wrote: > On 24 May 2022 5:05:35 pm GMT-04:00, Gloria Zhao via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >To clarify, in this situation, I'm imagining something like > >A: 0 sat, 100vB > >B: 1500 sat, 100vB > >C: 0 sat, 100vB > >X: 500 sat, 100vB > >feerate floor is 3sat/vB > > > >With the algo: > >> * is X alone above my fee rate? no, then forget it > >> * otherwise, s := X.size, f := X.fees, R := [X] > >> * for P = P1..Pn: > >> * do I already have P? then skip to the next parent > >> * s += P.size, f += P.fees, R += [P] > >> * if f/s above my fee rate floor? if so, request all the txs in R > > > >We'd erroneously ask for A+B+C+X, but really we should only take A+B. > >But wouldn't A+B also be a package that was announced for B? > > In theory, yes, but maybe it was announced earlier (while our node was > down?) or had dropped from our mempool or similar, either way we don't have > those txs yet. > > >Please lmk if you were imagining something different. I think I may be > >missing something. > > That's what I was thinking, yes. > > So the other thing is what happens if the peer announcing packages to us > is dishonest? > > They announce pkg X, say X has parents A B C and the fee rate is garbage. > But actually X has parent D and the fee rate is excellent. Do we request > the package from another peer, or every peer, to double check? Otherwise > we're allowing the first peer we ask about a package to censor that tx from > us?
Re: [bitcoin-dev] Package Relay Proposal
> From: bitcoin-dev On Behalf > Of Anthony Towns via bitcoin-dev > Sent: Wednesday, May 25, 2022 11:56 AM > So the other thing is what happens if the peer announcing packages to us is > dishonest? > > They announce pkg X, say X has parents A B C and the fee rate is garbage. But > actually X has parent D and the fee rate is excellent. Do we request the > package from another peer, or every peer, to double check? Otherwise we're > allowing the first peer we ask about a package to censor that tx from us? > > I think the fix for that is just to provide the fee and weight when announcing > the package rather than only being asked for its info? Then if one peer makes > it sound like a good deal you ask for the parent txids from them, dedupe, > request, and verify they were honest about the parents. Single tx broadcasts do not carry an advertised fee rate, however the' feefilter' message (BIP133) provides this distinction. This should be interpreted as applicable to packages. Given this message there is no reason to send a (potentially bogus) fee rate with every package. It can only be validated by obtaining the full set of txs, and the only recourse is dropping (etc.) the peer, as is the case with single txs. Relying on the existing message is simpler, more consistent, and more efficient. > >> Is it plausible to add the graph in? > > Likewise, I think you'd have to have the graph info from many nodes if you're > going to make decisions based on it and don't want hostile peers to be able to > trick you into ignoring txs. > > Other idea: what if you encode the parent txs as a short hash of the wtxid > (something like bip152 short ids? perhaps seeded per peer so collisions will > be different per peer?) and include that in the inv announcement? Would > that work to avoid a round trip almost all of the time, while still giving you > enough info to save bw by deduping parents? As I suggested earlier, a package is fundamentally a compact block (or block) announcement without the header. Compact block (BIP152) announcement is already well-defined and widely implemented. A node should never be required to retain an orphan, and BIP152 ensures this is not required. Once a validated set of txs within the package has been obtained with sufficient fee, a fee-optimal node would accept the largest subgraph of the package that conforms to fee constraints and drop any peer that provides a package for which the full graph does not. Let us not reinvent the wheel and/or introduce accidental complexity. I see no reason why packaging is not simply BIP152 without the 'header' field, an updated protocol version, and the following sort of changes to names: sendpkg MSG_CMPCT_PKG cmpctpkg getpkgtxn pkgtxn > > For a maximum 25 transactions, > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package. > > If you're doing short ids that's maybe 25*4B=100B already, then the above is > up to 36% overhead, I guess. Might be worth thinking more about, but maybe > more interesting with ancestors than just parents. > > >Also side note, since there are no size/count params, Size is restricted in the same manner as block and transaction broadcasts, by consensus. If the fee rate is sufficient there would be no reason to preclude any valid size up to what can be mined in one block (packaging across blocks is not economically rational under the assumption that one miner cannot expect to mine multiple blocks in a row). Count is incorporated into BIP152 as 'shortids_length'. > > wondering if we > >should just have "version" in "sendpackages" be a bit field instead of > >sending a message for each version. 32 versions should be enough right? Adding versioning to individual protocols is just a reflection of the insufficiency of the initial protocol versioning design, and that of the various ad-hoc changes to it (including yet another approach in this proposal) that have been introduced to compensate for it, though I'll address this in an independent post at some point. Best, e > Maybe but a couple of messages per connection doesn't really seem worth > arguing about? > > Cheers, > aj > > > -- > Sent from my phone. > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Package Relay Proposal
On 24 May 2022 5:05:35 pm GMT-04:00, Gloria Zhao via bitcoin-dev wrote: >To clarify, in this situation, I'm imagining something like >A: 0 sat, 100vB >B: 1500 sat, 100vB >C: 0 sat, 100vB >X: 500 sat, 100vB >feerate floor is 3sat/vB > >With the algo: >> * is X alone above my fee rate? no, then forget it >> * otherwise, s := X.size, f := X.fees, R := [X] >> * for P = P1..Pn: >> * do I already have P? then skip to the next parent >> * s += P.size, f += P.fees, R += [P] >> * if f/s above my fee rate floor? if so, request all the txs in R > >We'd erroneously ask for A+B+C+X, but really we should only take A+B. >But wouldn't A+B also be a package that was announced for B? In theory, yes, but maybe it was announced earlier (while our node was down?) or had dropped from our mempool or similar, either way we don't have those txs yet. >Please lmk if you were imagining something different. I think I may be >missing something. That's what I was thinking, yes. So the other thing is what happens if the peer announcing packages to us is dishonest? They announce pkg X, say X has parents A B C and the fee rate is garbage. But actually X has parent D and the fee rate is excellent. Do we request the package from another peer, or every peer, to double check? Otherwise we're allowing the first peer we ask about a package to censor that tx from us? I think the fix for that is just to provide the fee and weight when announcing the package rather than only being asked for its info? Then if one peer makes it sound like a good deal you ask for the parent txids from them, dedupe, request, and verify they were honest about the parents. >> Is it plausible to add the graph in? Likewise, I think you'd have to have the graph info from many nodes if you're going to make decisions based on it and don't want hostile peers to be able to trick you into ignoring txs. Other idea: what if you encode the parent txs as a short hash of the wtxid (something like bip152 short ids? perhaps seeded per peer so collisions will be different per peer?) and include that in the inv announcement? Would that work to avoid a round trip almost all of the time, while still giving you enough info to save bw by deduping parents? > For a maximum 25 transactions, >23*24/2 = 276, seems like 36 bytes for a child-with-parents package. If you're doing short ids that's maybe 25*4B=100B already, then the above is up to 36% overhead, I guess. Might be worth thinking more about, but maybe more interesting with ancestors than just parents. >Also side note, since there are no size/count params, wondering if we >should just have "version" in "sendpackages" be a bit field instead of >sending a message for each version. 32 versions should be enough right? Maybe but a couple of messages per connection doesn't really seem worth arguing about? Cheers, aj -- Sent from my phone. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Package Relay Proposal
The set of txs is the graph. Anything else would just reproduce the tx graph
which must be traversed in any case.
Similarly the set of txs is the fee, the sigops, the size, and the weight. The
only information required by packaging is the association of the txs with each
other for the purpose of aggregate (vs. individual) net reward consideration.
Since a package can only be reasonably considered for a single block, there is
a natural effective limit on acceptable package size. Since any number of
individual txs may be transmitted, and the size/weight/sigops of one tx is
bounded only by block validity, there is no reason to put any other constraints
on packages. A package is just a set of txs that may fit into a block and may
collectively be worth mining. A rational package is just a block or compact
block without the header. Making it any more than that is unnecessary
complexity.
If parts of a package satisfy profitability constraints, they will be
accepted/mined and if other parts do not, they will be rejected. There’s no
preventing this.
The only pertinent feature missing in the p2p protocol is the ability to
associate a set of txs for consideration, where the set (or subset) may satisfy
profitability constraints that would not be satisfied if the txs were
considered individually.
e
> On May 24, 2022, at 16:21, Gloria Zhao via bitcoin-dev
> wrote:
>
>
> Hi aj,
>
> > If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
> > below fee floor while A+B and A+B+C+X are above fee floor you have the
> > problem though.
>
> To clarify, in this situation, I'm imagining something like
> A: 0 sat, 100vB
> B: 1500 sat, 100vB
> C: 0 sat, 100vB
> X: 500 sat, 100vB
> feerate floor is 3sat/vB
>
> With the algo:
> > * is X alone above my fee rate? no, then forget it
> > * otherwise, s := X.size, f := X.fees, R := [X]
> > * for P = P1..Pn:
> > * do I already have P? then skip to the next parent
> > * s += P.size, f += P.fees, R += [P]
> > * if f/s above my fee rate floor? if so, request all the txs in R
>
> We'd erroneously ask for A+B+C+X, but really we should only take A+B.
> But wouldn't A+B also be a package that was announced for B?
> Please lmk if you were imagining something different. I think I may be
> missing something.
>
> > Is it plausible to add the graph in?
>
> Fun to think about. Most basic design would be to represent {spends, doesn’t
> spend} for a previous transaction in the package as a bit. Can think of it as
> a matrix where row i, column j tells you whether Tx j (directly) spends Tx i.
> But of course you can omit the last row, since the child spends all of them.
> And since topological ordering is a requirement, you only need as many bits
> as there are transactions preceding this one in the package.
> If you have up to 24 parents, you need 1 + 2 + ... + 23 bits to codify
> spending for the 2nd ... 24th parent. For a maximum 25 transactions, 23*24/2
> = 276, seems like 36 bytes for a child-with-parents package. A few more for
> tx-with-ancestors.
> Then you can split it up into sub-packages and everything. Still not sure if
> we really need to.
>
> Also side note, since there are no size/count params, wondering if we should
> just have "version" in "sendpackages" be a bit field instead of sending a
> message for each version. 32 versions should be enough right?
>
> Best,
> Gloria
>
>> On Tue, 24 May 2022 at 12:48 Anthony Towns wrote:
>> On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao
>> wrote:
>> >> If you're asking for the package for "D", would a response telling you:
>> >> txid_D (500 sat, 100vB)
>> >> txid_A (0 sat, 100vB)
>> >> txid_B (2000 sat, 100 vB)
>> >> be better, in that case? Then the receiver can maybe do the logic
>> >> themselves to figure out that they already have A in their mempool
>> >> so it's fine, or not?
>> >Right, I also considered giving the fees and sizes of each transaction in
>> >the package in “pckginfo1”. But I don’t think that information provides
>> >additional meaning unless you know the exact topology, i.e. also know if
>> >the parents have dependency relationships between them. For instance, in
>> >the {A, B, D} package there, even if you have the information listed, your
>> >decision should be different depending on whether B spends from A.
>>
>> I don't think that's true? We already know D is above our fee floor so if B
>> with A is also above the floor, we want them all, but also if B isn't above
>> the floor, but all of them combined are, then we also do?
>>
>> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
>> below fee floor while A+B and A+B+C+X are above fee floor you have the
>> problem though.
>>
>> Is it plausible to add the graph in?
>>
>> Cheers,
>> aj
>>
>>
>>
>> --
>> Sent from my phone.
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
>
Re: [bitcoin-dev] Package Relay Proposal
Hi aj,
> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
below fee floor while A+B and A+B+C+X are above fee floor you have the
problem though.
To clarify, in this situation, I'm imagining something like
A: 0 sat, 100vB
B: 1500 sat, 100vB
C: 0 sat, 100vB
X: 500 sat, 100vB
feerate floor is 3sat/vB
With the algo:
> * is X alone above my fee rate? no, then forget it
> * otherwise, s := X.size, f := X.fees, R := [X]
> * for P = P1..Pn:
> * do I already have P? then skip to the next parent
> * s += P.size, f += P.fees, R += [P]
> * if f/s above my fee rate floor? if so, request all the txs in R
We'd erroneously ask for A+B+C+X, but really we should only take A+B.
But wouldn't A+B also be a package that was announced for B?
Please lmk if you were imagining something different. I think I may be
missing something.
> Is it plausible to add the graph in?
Fun to think about. Most basic design would be to represent {spends,
doesn’t spend} for a previous transaction in the package as a bit. Can
think of it as a matrix where row i, column j tells you whether Tx j
(directly) spends Tx i.
But of course you can omit the last row, since the child spends all of
them. And since topological ordering is a requirement, you only need as
many bits as there are transactions preceding this one in the package.
If you have up to 24 parents, you need 1 + 2 + ... + 23 bits to codify
spending for the 2nd ... 24th parent. For a maximum 25 transactions,
23*24/2 = 276, seems like 36 bytes for a child-with-parents package. A few
more for tx-with-ancestors.
Then you can split it up into sub-packages and everything. Still not sure
if we really need to.
Also side note, since there are no size/count params, wondering if we
should just have "version" in "sendpackages" be a bit field instead of
sending a message for each version. 32 versions should be enough right?
Best,
Gloria
On Tue, 24 May 2022 at 12:48 Anthony Towns wrote:
> On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao
> wrote:
> >> If you're asking for the package for "D", would a response telling you:
> >> txid_D (500 sat, 100vB)
> >> txid_A (0 sat, 100vB)
> >> txid_B (2000 sat, 100 vB)
> >> be better, in that case? Then the receiver can maybe do the logic
> >> themselves to figure out that they already have A in their mempool
> >> so it's fine, or not?
> >Right, I also considered giving the fees and sizes of each transaction in
> >the package in “pckginfo1”. But I don’t think that information provides
> >additional meaning unless you know the exact topology, i.e. also know if
> >the parents have dependency relationships between them. For instance, in
> >the {A, B, D} package there, even if you have the information listed, your
> >decision should be different depending on whether B spends from A.
>
> I don't think that's true? We already know D is above our fee floor so if
> B with A is also above the floor, we want them all, but also if B isn't
> above the floor, but all of them combined are, then we also do?
>
> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
> below fee floor while A+B and A+B+C+X are above fee floor you have the
> problem though.
>
> Is it plausible to add the graph in?
>
> Cheers,
> aj
>
>
>
> --
> Sent from my phone.
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Package Relay Proposal
On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao wrote:
>> If you're asking for the package for "D", would a response telling you:
>> txid_D (500 sat, 100vB)
>> txid_A (0 sat, 100vB)
>> txid_B (2000 sat, 100 vB)
>> be better, in that case? Then the receiver can maybe do the logic
>> themselves to figure out that they already have A in their mempool
>> so it's fine, or not?
>Right, I also considered giving the fees and sizes of each transaction in
>the package in “pckginfo1”. But I don’t think that information provides
>additional meaning unless you know the exact topology, i.e. also know if
>the parents have dependency relationships between them. For instance, in
>the {A, B, D} package there, even if you have the information listed, your
>decision should be different depending on whether B spends from A.
I don't think that's true? We already know D is above our fee floor so if B
with A is also above the floor, we want them all, but also if B isn't above the
floor, but all of them combined are, then we also do?
If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is below
fee floor while A+B and A+B+C+X are above fee floor you have the problem though.
Is it plausible to add the graph in?
Cheers,
aj
--
Sent from my phone.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Package Relay Proposal
Hi aj,
> if you're writing a protocol that's
> dependent on people seeing that a package as a whole pays a competitive
> feerate, don't you want to know in advance what conditions the network
> is going to impose on your transactions in order to consider them as a
> package?
I do think unifying the size/count constraints would result in a more
stable/easier to reason about interface for L2 devs. Then the requirement
for propagation is just a path of nodes that support v1 package relay, and
it’s implied their mempool policy supports it as well. Also seems like it
could be a fingerprinting problem for nodes to give very specific
count/size limits.
> (… maybe core's defaults should be reconsidered rather than standardised
as-is)
> Worst case, you could presumably do a new package relay version with
> different constraints, if needed.
Maybe this was my actual concern. I think the defaults are safe but it’s
not like they’ve been proven to be optimal. This creates an obstacle to
changing them, especially if we want to make them smaller. But I think it’s
unlikely we’ll do that, and adding another version for new constraints
doesn’t seem too bad.
(Agreed with everything here, thanks for the feedback and clarifications!)
TLDR, making these changes:
- Count and size are implied by the version. Version 1 is specifically
child-with-unconfirmed-parents, where the whole package is at most 25
transactions and 101KvB.
- Announce sendpackages based on our own state. It’s ok to send
“sendpackages” if they sent fRelay=false.
- At verack, require fRelay=true and wtxidrelay if they sent sendpackages,
otherwise disconnect.
- If we get “getpckgtxns” or “pckgtxns” without having negotiated
“sendpackages” ahead of time, ignore, don’t disconnect. Emphasize that the
intention is to validate all of the transactions received through
“pckgtxns” together.
> If you're asking for the package for "D", would a response telling you:
> txid_D (500 sat, 100vB)
> txid_A (0 sat, 100vB)
> txid_B (2000 sat, 100 vB)
> be better, in that case? Then the receiver can maybe do the logic
> themselves to figure out that they already have A in their mempool
> so it's fine, or not?
Right, I also considered giving the fees and sizes of each transaction in
the package in “pckginfo1”. But I don’t think that information provides
additional meaning unless you know the exact topology, i.e. also know if
the parents have dependency relationships between them. For instance, in
the {A, B, D} package there, even if you have the information listed, your
decision should be different depending on whether B spends from A. The only
thing you know for sure about a child with direct parents is: if the
aggregate feerate is too low, you won’t want the child since it depends on
everyone else. If there’s a good-feerate transaction in there that doesn’t
have a dependency, you’re fine as long as someone sends it to you
individually.
Best,
Gloria
On Mon, May 23, 2022 at 2:34 PM Anthony Towns via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:
> On Wed, May 18, 2022 at 02:40:58PM -0400, Gloria Zhao via bitcoin-dev
> wrote:
> > > Does it make sense for these to be configurable, rather than implied
> > > by the version?
> > > … would it be better to either just not do sendpackages
> > > at all if you're limiting ancestors in the mempool incompatibly
> > Effectively: if you’re setting your ancestor/descendant limits lower than
> > the default, you can’t do package relay. I wonder if this might be
> > controversial, since it adds pressure to adhere to Bitcoin Core’s current
> > mempool policy? I would be happy to do it this way, though - makes things
> > easier to implement.
>
> How about looking at it the other way: if you're writing a protocol that's
> dependent on people seeing that a package as a whole pays a competitive
> feerate, don't you want to know in advance what conditions the network
> is going to impose on your transactions in order to consider them as a
> package? In that case, aren't the "depth" and "size" constraints things
> we should specify in a standard?
>
> (The above's not a rhetorical question; I'm not sure what the answer is.
> And even if it's "yes", maybe core's defaults should be reconsidered
> rather than standardised as-is)
>
> Worst case, you could presumably do a new package relay version with
> different constraints, if needed.
>
> > > > 5. If 'fRelay==false' in a peer's version message, the node must not
> > > >send "sendpackages" to them. If a "sendpackages" message is
> > > > received by a peer after sending `fRelay==false` in their version
> > > > message, the sender should be disconnected.
> > > Seems better to just say "if you set fRelay=false in your version
> > > message, you must not send sendpackages"? You already won't do packages
> > > with the peer if they don't also announce sendpackages.
> > I guess, theoretically, if you allow bloom filters with this peer, it’s
> > plausible they’re saying
Re: [bitcoin-dev] Package Relay Proposal
On Wed, May 18, 2022 at 02:40:58PM -0400, Gloria Zhao via bitcoin-dev wrote:
> > Does it make sense for these to be configurable, rather than implied
> > by the version?
> > … would it be better to either just not do sendpackages
> > at all if you're limiting ancestors in the mempool incompatibly
> Effectively: if you’re setting your ancestor/descendant limits lower than
> the default, you can’t do package relay. I wonder if this might be
> controversial, since it adds pressure to adhere to Bitcoin Core’s current
> mempool policy? I would be happy to do it this way, though - makes things
> easier to implement.
How about looking at it the other way: if you're writing a protocol that's
dependent on people seeing that a package as a whole pays a competitive
feerate, don't you want to know in advance what conditions the network
is going to impose on your transactions in order to consider them as a
package? In that case, aren't the "depth" and "size" constraints things
we should specify in a standard?
(The above's not a rhetorical question; I'm not sure what the answer is.
And even if it's "yes", maybe core's defaults should be reconsidered
rather than standardised as-is)
Worst case, you could presumably do a new package relay version with
different constraints, if needed.
> > > 5. If 'fRelay==false' in a peer's version message, the node must not
> > >send "sendpackages" to them. If a "sendpackages" message is
> > > received by a peer after sending `fRelay==false` in their version
> > > message, the sender should be disconnected.
> > Seems better to just say "if you set fRelay=false in your version
> > message, you must not send sendpackages"? You already won't do packages
> > with the peer if they don't also announce sendpackages.
> I guess, theoretically, if you allow bloom filters with this peer, it’s
> plausible they’re saying “fRelay=false, I’ll send you a bloom filter later,
> and I’ll also want to talk about packages.”
I was just meaning "it's okay to send VERSION fRelay=true then immediately
send WTXIDRELAY then immediately send SENDPACKAGES" without having to
first verify what the other guy's fRelay was set to. On the other hand,
you do already have to verify the other guy's version is high enough,
but it would be kind-of nice to move towards just announcing the features
you support, and not having to make it a multistep negotiation...
> > Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?
> Do you mean if we get a verack, and the peer sent “sendpackages” but not
> “wtxidrelay,” we should disconnect them?
Yes.
> I have it as: we send a PCKG INV when this transaction’s feerate is above
> the fee filter, but one or more of its parents don’t. I don’t think using
> ancestor feerate is better.
> See this counterexample:
> https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/abc_1parent_2kids.png
> A (0fee) has 2 kids, B (3sat/vB) and C (20sat/vB), everything’s the same
> vsize. Let’s say the fee filter is 3sat/vB.
> If we do it based on ancestor feerate, we won’t send B. But B is actually
> fine; C is paying for A.
But that only works if the receiver also has C, in which case they also
have A, and you don't need package relay to do anything with B? If they
didn't have C already, then relaying {A,B} would be a waste of time,
because {A,B} would be rejected as only paying 1.5sat/vB or whatever..
If you switch it to being:
A (0 sats, 200vB)
B (2000 sats, 200vB, spends A:0)
C (200 sats, 200vB)
D (1000 sats, 200vB, sepnds A:1, C:0)
then you get:
A alone = 0s/vB
B+A = 5s/vB
C alone = 1s/vB
D+C+A = 2s/vB
D+C = 3s/vB (B+A already at 5s/vB)
which I think recovers your point, while also having all the details
only be dealing with direct parents.
> > Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
> > just a general way to request a batch of transactions?
> > Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
> > be used more generally, potentially in ways unrelated to packages/cpfp?
> Indeed, it’s a general way to request a batch of transactions. I’ll
> highlight that it is “all or nothing,” i.e. if the sender is missing any of
> them, they’ll just send a notfound.
> The idea here was to avoid downloading any transactions that can’t be
> validated right away.
Right; maybe I should just be calling a "batch of packages to be validated
together" a "tx package" in the first place.
Maybe it would be worth emphasising that you should be expecting to
validate all the txs you receive as a response to getpckgtxns (getpkgtxs
:) all at the same time, and immediately upon receiving them?
> > The "only be sent if both peers agreed to do package relay" rule could
> > simply be dropped, I think.
> Wouldn’t we need some way of saying “hey I support batchtxns?” Otherwise
> you would have to guess by sending a request and waiting to see if it’s
> ignored?
Sure, perhaps I should have said leave
Re: [bitcoin-dev] Package Relay Proposal
(To everyone): I should have made it much clearer that version 1 is only supposed to solve 1 of the 2 use cases. I was a lot more focused on the fee-bumping use case, since it’s more important. Orphan-fetching was added to the motivation section last-minute because John Newbery mentioned to me “hey you could deal with orphans really easily with this.” Of course, child-with-unconfirmed-parents packages aren’t very useful for orphan-fetching since non-parent ancestors are quite common. Maybe a version 2 package for orphan-fetching could look like this: “pckginfo2” message contains a tx with all of its ancestors “MSG_PCKG2” inv type refers to a “pckginfo2” for a tx. You don’t send inv(MSG_PCKG2), but a node can request getdata(MSG_PCKG2) for a transaction they want the ancestors for, provided they sent sendpackages(version=2) ahead of time. It seems to me that orphan-fetching only ever needs to be receiver-initiated. Protocol flow would look like this: https://user-images.githubusercontent.com/25183001/168891185-1630f583-de47-4937-86b1-2652cf8852f2.png We don’t have a policy for dealing with anything more than a child with its direct parents, but I also don’t think anybody is relying on fee-bumping more than 2 generations, so the validation logic here could probably just submit them all individually. Maybe they can request a pckginfo1 if they see something that’s too-low-fee, and/or use the child-with-unconfirmed-parents logic opportunistically. Thanks aj for the feedback! Responding: > The "PCKG" abbreviation threw me for a loop; isn't the usual > abbreviation "PKG" ? Oh I didn't know that. I could change it if people feel strongly. > Does it make sense for these to be configurable, rather than implied > by the version? > … would it be better to either just not do sendpackages > at all if you're limiting ancestors in the mempool incompatibly Effectively: if you’re setting your ancestor/descendant limits lower than the default, you can’t do package relay. I wonder if this might be controversial, since it adds pressure to adhere to Bitcoin Core’s current mempool policy? I would be happy to do it this way, though - makes things easier to implement. > > 5. If 'fRelay==false' in a peer's version message, the node must not > >send "sendpackages" to them. If a "sendpackages" message is > > received by a peer after sending `fRelay==false` in their version > > message, the sender should be disconnected. > Seems better to just say "if you set fRelay=false in your version > message, you must not send sendpackages"? You already won't do packages > with the peer if they don't also announce sendpackages. I guess, theoretically, if you allow bloom filters with this peer, it’s plausible they’re saying “fRelay=false, I’ll send you a bloom filter later, and I’ll also want to talk about packages.” I don’t know if that’s a use case we want to support - my gut reaction is no. > Maybe: "You must not send sendpackages unless you also send wtxidrelay" ? Do you mean if we get a verack, and the peer sent “sendpackages” but not “wtxidrelay,” we should disconnect them? > As I understand it, the two cases for the protocol flow are "I received > an orphan, and I'd like its ancestors please" which seems simple enough, > and "here's a child you may be interested in, even though you possibly > weren't interested in the parents of that child". (Btw, please see my notes at the top of this email about separating those two use cases. sorry for the confusion). > I think the logic for the latter is: […] I have it as: we send a PCKG INV when this transaction’s feerate is above the fee filter, but one or more of its parents don’t. I don’t think using ancestor feerate is better. See this counterexample: https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/abc_1parent_2kids.png A (0fee) has 2 kids, B (3sat/vB) and C (20sat/vB), everything’s the same vsize. Let’s say the fee filter is 3sat/vB. If we do it based on ancestor feerate, we won’t send B. But B is actually fine; C is paying for A. > Are "getpckgtxns" / "pcktxns" really limited to packages, or are they > just a general way to request a batch of transactions? > Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to > be used more generally, potentially in ways unrelated to packages/cpfp? Indeed, it’s a general way to request a batch of transactions. I’ll highlight that it is “all or nothing,” i.e. if the sender is missing any of them, they’ll just send a notfound. The idea here was to avoid downloading any transactions that can’t be validated right away. With packages, this makes sense, because there are dependency relationships. But if you’re requesting multiple unrelated transactions, for example, it’s unnecessary. You might end up with even more transaction data that’s just sitting around waiting to be validated. > The "only be sent if both peers agreed to do package relay" rule could > simply be dropped, I think.
Re: [bitcoin-dev] Package Relay Proposal
On Tue, May 17, 2022 at 12:01:04PM -0400, Gloria Zhao via bitcoin-dev wrote:
> New Messages
> Three new protocol messages are added for use in any version of
> package relay. Additionally, each version of package relay must define
> its own inv type and "pckginfo" message version, referred to in this
> document as "MSG_PCKG" and "pckginfo" respectively. See
> BIP-v1-packages for a concrete example.
The "PCKG" abbreviation threw me for a loop; isn't the usual
abbreviation "PKG" ?
> =sendpackages=
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.
Does it make sense for these to be configurable, rather than implied
by the version?
I presume the idea is to cope with people specifying different values for
-limitancestorcount or -limitancestorsize, but if people are regularly
relaying packages around, it seems like it becomes hard to have those
values really be configurable while being compatible with that?
I guess I'm asking: would it be better to either just not do sendpackages
at all if you're limiting ancestors in the mempool incompatibly; or
alternatively, would it be better to do the package relay, then reject
the particular package if it turns out too big, and log that you've
dropped it so that the node operator has some way of realising "whoops,
I'm not relaying packages properly because of how I configured my node"?
> 5. If 'fRelay==false' in a peer's version message, the node must not
>send "sendpackages" to them. If a "sendpackages" message is
> received by a peer after sending `fRelay==false` in their version
> message, the sender should be disconnected.
Seems better to just say "if you set fRelay=false in your version
message, you must not send sendpackages"? You already won't do packages
with the peer if they don't also announce sendpackages.
> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>version, the peers should announce, request, and send package
> information to each other.
Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?
As I understand it, the two cases for the protocol flow are "I received
an orphan, and I'd like its ancestors please" which seems simple enough,
and "here's a child you may be interested in, even though you possibly
weren't interested in the parents of that child". I think the logic for
the latter is:
* if tx C's fee rate is less than the peer's feefilter, skip it
(will maybe treat it as a parent in some package later though)
* if tx C's ancestor fee rate is less than the peer's feefilter, skip
it?
* look at the lowest ancestor fee rate for any of C's in-mempool
parents
* if that is higher than the peer's fee filter, send a normal INV
* if it's lower than the peer's fee filter, send a PCKG INV
Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
just a general way to request a batch of transactions? Particularly in
the case of requesting the parents of an orphan tx you already have,
it seems hard for the node receiving getpckgtxns to validate that the
txs are related in some way; but also it doesn't seem very necessary?
Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
be used more generally, potentially in ways unrelated to packages/cpfp?
The "only be sent if both peers agreed to do package relay" rule could
simply be dropped, I think.
> 4. The reciever uses the package information to decide how to request
>the transactions. For example, if the receiver already has some of
> the transactions in their mempool, they only request the missing ones.
> They could also decide not to request the package at all based on the
> fee information provided.
Shouldn't the sender only be sending package announcements when they know
the recipient will be interested in the package, based on their feefilter?
> =pckginfo1=
> {|
> | Field Name || Type || Size || Purpose
> |-
> |blockhash || uint256 || 32 || The chain tip at which this package is
> defined.
> |-
> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in the
> package.
CAmount in consensus/amount.h is a int64_t so shouldn't this be 8
bytes? If you limit a package to 101kvB, an int32_t is enough to cover
any package with a fee rate of about 212 BTC/block or lower, though.
> |pckg_weight||int64_t||8|| The sum total weight of all transactions in the
> package.
The maximum block weight is 4M, and the default -limitancestorsize
presumably implies a max package weight of 404k; seems odd to provide
a uint64_t rather than an int32_t here, which easily allows either of
those values?
> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>of one transaction and its
Re: [bitcoin-dev] Package Relay Proposal
Hi Greg, Thanks for reading! >> A child-with-unconfirmed-parents package sent between nodes must abide by the rules below, otherwise the package is malformed and the sender should be disconnected. >> However, if the child has confirmed parents, they must not be in the package. > If my naive understanding is correct, this means things like otherwise common situations such as a new block will result in disconnects, say when > the sender doesn't hear about a new block which makes the relay package superfluous/irrelevant. Similar would be disconnection > when confirmed gets turned into unconfirmed, but those situations are extremely uncommon. The other rules are entirely under the control > of the sender, which leads me to wonder if it's appropriate. This is why the "pckginfo1" message includes the blockhash at which the package was defined. Also please see Clarifications - "Q: What if a new block arrives in between messages?'' section in the v1-packages portion. It covers both cases, i.e. a transaction going from unconfirmed->confirmed and confirmed->unconfirmed in a reorg. In case anybody is wondering "why don't we just allow confirmed parents?": Since we validate based on the UTXO set, when we see a recently-confirmed transaction, it just looks like it spends nonexistent inputs. In these cases, we don't really know if the input was recently spent in a block or just never existed, unless we plan on looking up transactions in past blocks. We do some guesswork when we deal with new blocks in normal transaction relay (e.g. we requested the tx before a block arrived): https://github.com/bitcoin/bitcoin/blob/d5d40d59f8d12cf53c5ad1ce9710f3f108cec386/src/validation.cpp#L780-L784 I believe it's cleaner to just explicitly say which blockhash you're on to avoid confusion. Thanks, Gloria On Tue, May 17, 2022 at 1:56 PM Greg Sanders wrote: > Hi Gloria, > > Thanks for working on this important proposal! > > Still a lot to digest, but I just had on area of comment/question: > > > A child-with-unconfirmed-parents package sent between nodes must abide by > the rules below, otherwise the package is malformed and the sender should > be disconnected. > > > However, if the child has confirmed parents, they must not be in the > package. > > If my naive understanding is correct, this means things like otherwise > common situations such as a new block will result in disconnects, say when > the sender doesn't hear about a new block which makes the relay package > superfluous/irrelevant. Similar would be disconnection > when confirmed gets turned into unconfirmed, but those situations are > extremely uncommon. The other rules are entirely under the control > of the sender, which leads me to wonder if it's appropriate. > > Cheers, > Greg > > On Tue, May 17, 2022 at 12:09 PM Gloria Zhao via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> Hi everybody, >> >> I’m writing to propose a set of p2p protocol changes to enable package >> relay, soliciting feedback on the design and approach. Here is a link >> to the most up-to-date proposal: >> >> https://github.com/bitcoin/bips/pull/1324 >> >> If you have concept or approach feedback, *please respond on the >> mailing list* to allow everybody to view and participate in the >> discussion. If you find a typo or inaccurate wording, please feel free >> to leave suggestions on the PR. >> >> I’m also working on an implementation for Bitcoin Core. >> >> >> The rest of this post will include the same contents as the proposal, >> with a bit of reordering and additional context. If you are not 100% >> up-to-date on package relay and find the proposal hard to follow, I >> hope you find this format more informative and persuasive. >> >> >> ==Background and Motivation== >> >> Users may create and broadcast transactions that depend upon, i.e. >> spend outputs of, unconfirmed transactions. A “package” is the >> widely-used term for a group of transactions representable by a >> connected Directed Acyclic Graph (where a directed edge exists between >> a transaction that spends the output of another transaction). >> >> Incentive-compatible mempool and miner policies help create a fair, >> fee-based market for block space. While miners maximize transaction >> fees in order to earn higher block rewards, non-mining users >> participating in transaction relay reap many benefits from employing >> policies that result in a mempool with the same contents, including >> faster compact block relay and more accurate fee estimation. >> Additionally, users may take advantage of mempool and miner policy to >> bump the priority of their transactions by attaching high-fee >> descendants (Child Pays for Parent or CPFP). Only considering >> transactions one at a time for submission to the mempool creates a >> limitation in the node's ability to determine which transactions have >> the highest feerates, since it cannot take into account descendants >> until all the transactions are in the mempool.
Re: [bitcoin-dev] Package Relay Proposal
Hi Gloria, Thanks for working on this important proposal! Still a lot to digest, but I just had on area of comment/question: > A child-with-unconfirmed-parents package sent between nodes must abide by the rules below, otherwise the package is malformed and the sender should be disconnected. > However, if the child has confirmed parents, they must not be in the package. If my naive understanding is correct, this means things like otherwise common situations such as a new block will result in disconnects, say when the sender doesn't hear about a new block which makes the relay package superfluous/irrelevant. Similar would be disconnection when confirmed gets turned into unconfirmed, but those situations are extremely uncommon. The other rules are entirely under the control of the sender, which leads me to wonder if it's appropriate. Cheers, Greg On Tue, May 17, 2022 at 12:09 PM Gloria Zhao via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Hi everybody, > > I’m writing to propose a set of p2p protocol changes to enable package > relay, soliciting feedback on the design and approach. Here is a link > to the most up-to-date proposal: > > https://github.com/bitcoin/bips/pull/1324 > > If you have concept or approach feedback, *please respond on the > mailing list* to allow everybody to view and participate in the > discussion. If you find a typo or inaccurate wording, please feel free > to leave suggestions on the PR. > > I’m also working on an implementation for Bitcoin Core. > > > The rest of this post will include the same contents as the proposal, > with a bit of reordering and additional context. If you are not 100% > up-to-date on package relay and find the proposal hard to follow, I > hope you find this format more informative and persuasive. > > > ==Background and Motivation== > > Users may create and broadcast transactions that depend upon, i.e. > spend outputs of, unconfirmed transactions. A “package” is the > widely-used term for a group of transactions representable by a > connected Directed Acyclic Graph (where a directed edge exists between > a transaction that spends the output of another transaction). > > Incentive-compatible mempool and miner policies help create a fair, > fee-based market for block space. While miners maximize transaction > fees in order to earn higher block rewards, non-mining users > participating in transaction relay reap many benefits from employing > policies that result in a mempool with the same contents, including > faster compact block relay and more accurate fee estimation. > Additionally, users may take advantage of mempool and miner policy to > bump the priority of their transactions by attaching high-fee > descendants (Child Pays for Parent or CPFP). Only considering > transactions one at a time for submission to the mempool creates a > limitation in the node's ability to determine which transactions have > the highest feerates, since it cannot take into account descendants > until all the transactions are in the mempool. Similarly, it cannot > use a transaction's descendants when considering which of two > conflicting transactions to keep (Replace by Fee or RBF). > > When a user's transaction does not meet a mempool's minimum feerate > and they cannot create a replacement transaction directly, their > transaction will simply be rejected by this mempool. They also cannot > attach a descendant to pay for replacing a conflicting transaction. > This limitation harms users' ability to fee-bump their transactions. > Further, it presents a security issue in contracting protocols which > rely on **presigned**, time-sensitive transactions to prevent cheating > (HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault > [4], Refund Transaction in Discreet Log Contracts [5], Updates in > eltoo [6]). In other words, a key security assumption of many > contracting protocols is that all parties can propagate and confirm > transactions in a timely manner. > > In the past few years, increasing attention [0][1][2][3][6] has been > brought to **pinning attacks**, a type of censorship in which the > attacker uses mempool policy restrictions to prevent a transaction > from being relayed or getting mined. TLDR: revocation transactions > must meet a certain confirmation target to be effective, but their > feerates are negotiated well ahead of broadcast time. If the > forecasted feerate was too low and no fee-bumping options are > available, attackers can steal money from their counterparties. I walk > through a concrete example for stealing Lightning HTLC outputs at > ~23:58 in this talk [7][8]. Note that most attacks are only possible > when the market for blockspace at broadcast time demands much higher > feerates than originally anticipated at signing time. Always > overestimating fees may sidestep this issue temporarily (while mempool > traffic is low and predictable), but this solution is not foolproof > and wastes users' money. The feerate
