Re: [bitcoin-dev] Packaged Transaction Relay
> -Original Message- > From: Anthony Towns > Subject: Re: [bitcoin-dev] Packaged Transaction Relay > > > > > > Protocol cannot be defined on an ad-hoc basis as a "courtesy" > > > > > BIPs are a courtesy in the first place. > > > > I suppose if you felt that you were the authority then this would > > > > be your perspective. > > > You seem to think that I'm arguing courtesy is not a good thing, or > > > that we couldn't use more of it? > > That is neither what I said nor implied. You were clearly dismissing > > the public process, not advocating for politeness. > > And that is neither what I said nor implied, nor something I believe. If you > think courtesy is something that can be ignored in a public process, I don't > think you should expect much success. "BIPs are a courtesy in the first place." > If you'd like to actually participate in public standards development, please > feel free to make some technical comments on my proposals, or others, or > make your own proposal, either here or on github, or heck, anywhere else. "RE: [bitcoin-dev] Packaged Transaction Relay" > I mean, that's what I'd suggest anyway; I'm not your boss. I promise to at > least be entertainingly surprised if you make any progress with your current > approach though. Grow up Anthony. e ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
On Sun, Oct 09, 2022 at 12:00:04AM -0700, e...@voskuil.org wrote: > On Sat, Oct 08, 2022, Anthony Towns via bitcoin-dev wrote: > > > > > Protocol cannot be defined on an ad-hoc basis as a "courtesy" > > > > BIPs are a courtesy in the first place. > > > I suppose if you felt that you were the authority then this would be > > > your perspective. > > You seem to think that I'm arguing courtesy is not a good thing, or that > we > > couldn't use more of it? > That is neither what I said nor implied. You were clearly dismissing the > public process, not advocating for politeness. And that is neither what I said nor implied, nor something I believe. If you think courtesy is something that can be ignored in a public process, I don't think you should expect much success. If you'd like to actually participate in public standards development, please feel free to make some technical comments on my proposals, or others, or make your own proposal, either here or on github, or heck, anywhere else. I mean, that's what I'd suggest anyway; I'm not your boss. I promise to at least be entertainingly surprised if you make any progress with your current approach though. Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
On Sat, Oct 08, 2022, Anthony Towns via bitcoin-dev wrote: > > > > Protocol cannot be defined on an ad-hoc basis as a "courtesy" > > > BIPs are a courtesy in the first place. > > I suppose if you felt that you were the authority then this would be > > your perspective. > > You seem to think that I'm arguing courtesy is not a good thing, or that we > couldn't use more of it? That is neither what I said nor implied. You were clearly dismissing the public process, not advocating for politeness. > > The BIP process was created by Amir specifically because Bitcoin > > standards were being discussed and developed behind closed doors. > > It definitely bothers me that Bitcoin development is not being discussed out > in the open as much as I would like, and to counter that, I try to encourage > people to post their ideas to this list, and write them up as a BIP; and likewise > try to do both myself as well. > > But how much value do you think anyone's actually getting from posting their > development ideas to this list these days? Do you really think people reading > your mail will be more inspired to discuss their ideas in the open, or that > they'll prefer to get in a room with their friends and allies, and close the > doors so they can work in peace? My comments have nothing to do with posting to this list. > > > There's no central authority to enforce some particular way of doing > > > things. > > As if reaching consensus with other people implies a singular authority. > > Reaching consensus with other people doesn't require putting a document in > some particular github repo, either. Which is a good thing, or the people in > control of that repo would become that singular authority. It is the public process that the community has clearly established. It has been challenged at times, which anyone is free to do - creating their own if they feel it becomes necessary. There is certainly no such issue in this case, so it is not at all clear what you mean to imply here. Is this just a blanket rejection of community standards development, or is it that you feel this community is limited to "friends and allies"? Developers of Bitcoin Core have stated countless times that they consider Bitcoin Core to be the protocol documentation, implying that their internal process is the process of arriving at community consensus. What was that you said about "some particular github repo" becoming the "singular authority"? > > > If you think that the version restriction should be part of the BIP, > > > why not do a pull request? The BIP is still marked as "Draft". > > I did not implement and ship a deviation from the posted proposal. > > You think BIP 155 is suboptimal, and would rather see it changed, no? The Bitcoin Core developers who deployed the deviation apparently also thought the BIP was suboptimal. Whether I agree with the change isn't relevant. > But if you won't put any effort into changing it (and how much effort do you > think a PR to change it document it as being gated by version 70016 would > be?), why do you imagine the people who are happy with the BIP as it is > would put any effort in? Yes, that's it. I'm lazy. It's all about effort, not about the process which, by your own measure, the owners of a single repo aim to be the "singular authority". > > > > I doubt that anyone who's worked with it is terribly fond of > > > > Bitcoin's P2P protocol versioning. I've spent some time on a > > > > proposal to update it, though it hasn't been a priority. If anyone > > > > is interested in collaborating on it please contact me directly. > > "contact me directly" and wanting something other than standards "being > discussed and developed behind closed doors" seems quite contradictory to > me. It's the public process that is at issue, and you of course know that - hence your varied attempts here to make it about something else. > > Your contributions notwithstanding, you are in no place to exhibit > > such arrogance. > > I don't understand what you think is arrogant about posting a public proposal > about how I think things should work, even if I had only put > 10 minutes thought into it. If that *is* arrogance, I guess I think we could use > more of it, as well as more courtesy... As if I was referring to this. "BIPs are a courtesy in the first place" says it all. Best, e ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
On Sat, Oct 08, 2022 at 12:58:35PM -0700, Eric Voskuil via bitcoin-dev wrote: > > > Protocol cannot be defined on an ad-hoc basis as a "courtesy" > > BIPs are a courtesy in the first place. > I suppose if you felt that you were the authority then this would be your > perspective. You seem to think that I'm arguing courtesy is not a good thing, or that we couldn't use more of it? If it helps: courtesy is a good thing, and we could use more of it. > The BIP process was created by Amir specifically because Bitcoin standards > were being discussed and developed behind closed doors. It definitely bothers me that Bitcoin development is not being discussed out in the open as much as I would like, and to counter that, I try to encourage people to post their ideas to this list, and write them up as a BIP; and likewise try to do both myself as well. But how much value do you think anyone's actually getting from posting their development ideas to this list these days? Do you really think people reading your mail will be more inspired to discuss their ideas in the open, or that they'll prefer to get in a room with their friends and allies, and close the doors so they can work in peace? > > There's no central authority to enforce some particular way of doing > > things. > As if reaching consensus with other people implies a singular authority. Reaching consensus with other people doesn't require putting a document in some particular github repo, either. Which is a good thing, or the people in control of that repo would become that singular authority. > > If you think that the version restriction should be part of the BIP, > > why not do a pull request? The BIP is still marked as "Draft". > I did not implement and ship a deviation from the posted proposal. You think BIP 155 is suboptimal, and would rather see it changed, no? But if you won't put any effort into changing it (and how much effort do you think a PR to change it document it as being gated by version 70016 would be?), why do you imagine the people who are happy with the BIP as it is would put any effort in? > > > I doubt that anyone who's worked with it is terribly fond of Bitcoin's > > > P2P protocol versioning. I've spent some time on a proposal to > > > update it, though it hasn't been a priority. If anyone is > > > interested in collaborating on it please contact me directly. "contact me directly" and wanting something other than standards "being discussed and developed behind closed doors" seems quite contradictory to me. (In my opinion, a big practical advantage of doing things in public is that it's easy for people to contribute, even if it's not a particular priority, and that it's also easy for someone new to take over, if the people previously working on it decide they no longer have time for that particular project) > > Bottlenecking a proposal on someone who doesn't see it as a priority > > doesn't seem smart? > I didn't realize I was holding you up. As far as I've been able to gather, > it hasn't been a priority for anyone. Yet somehow, on the same day that I > posted the fact that I was working on it, it became your top priority. It's not my top priority; it's just that writing a BIP and posting it publicly is fundamentally no harder than writing an email to bitcoin-dev. So since I'm willing to do one, why waste anyone's time by not also doing the other? Would've been even easier if I'd remembered Suhas had already written up a draft BIP two years ago... And if I'm going to suggest you should post a patch to a BIP you think is flawed, then not drafting a BIP to improve on a practice I think is flawed would be pretty hypocritical, no? (I didn't read what you said to imply that you were working on it, just that you'd spent time thinking about it, were interested, and might do more if people contacted you. If you have been working on it, why not do so in public? You already have a public bips fork at https://github.com/evoskuil/bips/branches -- how about just pushing your work-in-progress there?) (Ah, I also see now that I did contact you in Dec 2020/Jan 2021 on this topic, but never received a response. Apologies; the above was meant as a general statement in favour of just collaborating in public from the start for the practical advantages I outline above, not a personal dig) > > Here's what I think makes sense: > > https://github.com/ajtowns/bips/blob/202210-p2pfeatures/bip- > > p2pfeatures.mediawiki > Looks like you put about 10 minutes of thought into it. In your words, BIPs > are a courtesy - feel free to do what you want. So, you wrote a lot of stuff after this, but unless I missed it, it didn't include any substantive criticism of the proposal, or specific suggestions for changing it, or even any indication why you would have any difficulty supporting/implementing it in the software you care about. > Your contributions notwithstanding, you are in no place to exhibit such > arrogance. I don't understand what you
Re: [bitcoin-dev] Packaged Transaction Relay
> From: Anthony Towns > On Wed, Oct 05, 2022 at 09:32:29PM -0700, Eric Voskuil via bitcoin-dev wrote: > > Protocol cannot be defined on an ad-hoc basis as a "courtesy" > > BIPs are a courtesy in the first place. I suppose if you felt that you were the authority then this would be your perspective. However in the case of community software development, open standards are a tool to preempt such centralization. The BIP process was created by Amir specifically because Bitcoin standards were being discussed and developed behind closed doors. That process was being funded almost entirely by a corporate consortium (the Bitcoin Foundation). It was also clear that one implementation leads directly to this type of authority complex, which is why he also started libbitcoin. It's not surprising to learn that you feel this way, and it's nice of you to share those thoughts publicly. > There's no central authority to enforce some particular way of doing things. As if reaching consensus with other people implies a singular authority. > > - and it's not exactly a courtesy to keep yourself from getting dropped by > peers. It is not clear to me why such a comment would be accepted instead > of specifying this properly. > > If you think that the version restriction should be part of the BIP, why not do > a pull request? The BIP is still marked as "Draft". I did not implement and ship a deviation from the posted proposal. The developers who did so spent almost as much time writing a comment about the intentional deviation as they would have spent issuing a PR to the BIP. Presumably, given that years have passed, there has been enough time to correct that "mistake". At this point there are at least 5 implementations operating on mainnet that are inconsistent with Core. > > I doubt that anyone who's worked with it is terribly fond of Bitcoin's P2P > protocol versioning. I've spent some time on a proposal to update it, though > it hasn't been a priority. If anyone is interested in collaborating on it please > contact me directly. > > Bottlenecking a proposal on someone who doesn't see it as a priority doesn't > seem smart? I didn't realize I was holding you up. As far as I've been able to gather, it hasn't been a priority for anyone. Yet somehow, on the same day that I posted the fact that I was working on it, it became your top priority. > Here's what I think makes sense: > > https://github.com/ajtowns/bips/blob/202210-p2pfeatures/bip- > p2pfeatures.mediawiki Looks like you put about 10 minutes of thought into it. In your words, BIPs are a courtesy - feel free to do what you want. I'm well aware of your contributions to Bitcoin, but I find the arrogance off-putting. I have spent many years contributing to Bitcoin development and understanding, entirely on my own dime, even paying others to do so - as well as raising donations for them. I do this intentionally, not because I/we haven't had offers. Many corporate and state-funded Bitcoin Core developers have repeatedly, aggressively, openly and self-servingly worked to put a stop to such community efforts. To them the BIP process is a "courtesy" - just sometimes documenting what they happen to be doing in the protocols. And without actual alternatives, that's exactly what it is. So I'll just leave you with this: "MIT Digital Currency Initiative (DCI) announces research collaboration with the Bank of England on central bank digital currency The Bank of England announced an agreement to collaborate on a twelve-month Central Bank Digital Currency (CBDC) research project with MIT Digital Currency Initiative. The agreement supports and builds on DCI's ongoing research into CBDC, while also contributing to the Bank of England's wider research and exploration of central bank digital currencies. While no decision has been made on whether or not to introduce a CBDC in the UK, the work will investigate and experiment with potential CBDC technology designs and approaches, and evaluate key tradeoffs, opportunities, and risks. This type of research can help inform wider policy development by contributing important technical ideas and questions. As part of OpenCBDC, DCI's open-source codebase and research initiative, MIT DCI aims to fill this gap by engaging technologists, user researchers, central bankers, private sector leaders, and academics in service of a more accessible, trusted, fair, and resilient economy. We don't yet know if or in what contexts CBDCs can help improve the broader international monetary system, or how they might be best designed to do so, but we believe engaging in technical research is an important step in answering these questions." https://dci.mit.edu/research/2022/3/31/mit-digital-currency-initiative-dci-a nnounces-research-collaboration-with-the-bank-of-england-on-central-bank-dig ital-currency https://ras.mit.edu/finding-funding/find-funding/federal-funding https://dci.mit.edu/anthony-aj-towns Some might call this a conflict of interest.
Re: [bitcoin-dev] Packaged Transaction Relay
On Wed, Oct 05, 2022 at 09:32:29PM -0700, Eric Voskuil via bitcoin-dev wrote: > Protocol cannot be defined on an ad-hoc basis as a "courtesy" BIPs are a courtesy in the first place. There's no central authority to enforce some particular way of doing things. > - and it's not exactly a courtesy to keep yourself from getting dropped by > peers. It is not clear to me why such a comment would be accepted instead of > specifying this properly. If you think that the version restriction should be part of the BIP, why not do a pull request? The BIP is still marked as "Draft". > I doubt that anyone who's worked with it is terribly fond of Bitcoin's P2P > protocol versioning. I've spent some time on a proposal to update it, though > it hasn't been a priority. If anyone is interested in collaborating on it > please contact me directly. Bottlenecking a proposal on someone who doesn't see it as a priority doesn't seem smart? Here's what I think makes sense: https://github.com/ajtowns/bips/blob/202210-p2pfeatures/bip-p2pfeatures.mediawiki Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
>> ...sendaddrv2 messages are only sent to nodes advertising version 70016 or >> later (same as wtxidrelay) > I don’t see this constraint in BIP155. Do you mean that addrv2 support was > released in Core at the same time as wtxidrelay, or that it is an > undocumented version constraint implemented in Core? I see that it is the latter: // BIP155 defines addrv2 and sendaddrv2 for all protocol versions, but some // implementations reject messages they don't know. As a courtesy, don't send // it to nodes with a version before 70016, as no software is known to support // BIP155 that doesn't announce at least that protocol version number. https://github.com/bitcoin/bitcoin/pull/20564/files#diff-6875de769e90cec84d2e8a9c1b962cdbcda44d870d42e4215827e599e11e90e3R2366-R2370 The version string in the log message I posted implies it may not be a Core release. Yet it is BIP155 compliant. Protocol cannot be defined on an ad-hoc basis as a "courtesy" - and it's not exactly a courtesy to keep yourself from getting dropped by peers. It is not clear to me why such a comment would be accepted instead of specifying this properly. A new protocol cannot define a message for "all versions", it can only assume that older versions will disregard all unknown message traffic - or that implementers will patch it in this ad-hoc matter. I would suggest that authors update BIP155 and BIP330 (both still in Draft status), as well any pending proposals that may have picked up this pattern from BIP155. I doubt that anyone who's worked with it is terribly fond of Bitcoin's P2P protocol versioning. I've spent some time on a proposal to update it, though it hasn't been a priority. If anyone is interested in collaborating on it please contact me directly. e ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
>> [Regarding bandwidth waste: I've pointed out in years past that >> breaking the Bitcoin versioning scheme creates a requirement that any >> unknown message type be considered valid. Up until a recently-deployed >> protocol change, it had always been possible to validate messages by >> type. I noticed recently that validating nodes have been dropping peers >> at an increasing rate (a consequence of that deployment). Despite being >> an undocumented compatibility break, it is now unfortunately a matter >> of protocol that a peer must allow its peers to waste its bandwidth to >> remain compatible - something which we should eliminate.] > > The only message listed as not being preceded by a bumped version number > in: > > https://github.com/libbitcoin/libbitcoin-network/wiki/Protocol-Versioning Good find, still a work in progress. > is addrv2 (though addrv2 is gated on mutual exchange of sendaddrv2, so > it's presumably the sendaddrv2 message at issue), addrv2 is listed as the BIP title, the message that would cause the break is sendaddrv2 (quoted text). > however since [0] > sendaddrv2 messages are only sent to nodes advertising version 70016 or > later (same as wtxidrelay). I don’t see this constraint in BIP155. Do you mean that addrv2 support was released in Core at the same time as wtxidrelay, or that it is an undocumented version constraint implemented in Core? > ADDRV2 was introduced May 20 2020 after the > 0.20 branch, and SENDADDRV2 gating was merged Dec 9 2020 and included > from 0.21.0rc3 onwards. To clarify, there was no Core release of addrv2 without sendaddrv2 apart from 0.21 release candidates? > [0] https://github.com/bitcoin/bitcoin/pull/20564 > > I'm only seeing "bytesrecv_per_msg.*other*" entries for nodes advertising > a version of 0.17 and 0.18, > which I presume is due to REJECT messages (for taproot txs, perhaps?). Ideally you should not be seeing reject messages as protocol “other”, as these are valid messages as of protocol version 70002, and they are excluded by negotiated version before that. While there is no requirement to send them (BIP61 only defines a new message type), they remain defined messages until removed by a future protocol version. > Otherwise, I don't think there are any unexpected > messages you should be receiving when advertising version 70015 or lower. Yet nodes with an advertised protocol version of 70013 are receiving sendaddrv2. I've removed the IP address from the log extract below. 17:53:45.022347 DEBUG [network] Peer [x.x.x.x:8333] protocol version (70016) user agent: /Satoshi:0.21.0()/ 17:53:45.022377 DEBUG [network] Negotiated protocol version (70013) for [x.x.x.x.135:8333] 17:53:45.022767 INFO [network] Connected outbound channel [x.x.x.x.135:8333] 17:53:45.022913 DEBUG [node] Ask [x.x.x.x:8333] for headers after [0002e8c1c59fc86f721ba3a3294d2b1165597ddb910058e6] 17:53:45.023184 WARNING [network] Invalid sendaddrv2 payload from [x.x.x.x:8333] object does not exist 17:53:45.023317 DEBUG [network] Stop protocol version on [x.x.x.x:8333] object does not exist 17:53:45.023359 DEBUG [network] Outbound channel stopped [x.x.x.x:8333] success To my knowledge the only other time we've seen consistent invalid message traffic on the network was during the work on BIP150 (withdrawn), at which point BIP150 nodes were being deployed on mainnet. I made comments here on the issue at the time, which as I recall were generally rejected in favor of forcing nodes to allow all invalid traffic. In any case BIP150 was withdrawn and BIP324 proposed, which fixes this particular issue (using a service bit). Some argued at the time that allowance for invalid messages was a longstanding requirement in the protocol. I knew that this was not the case (except for BIP37, break documented in BIP60) because libbitcoin validates all messages, which led me to eventually document it. Recently I updated and posted that documentation (the github wiki link you found). This was a consequence of reviewing the Generic Package Relay proposal, which is also incompatible. In doing so I noticed this issue with BIP155 and BIP330 as well. This led us to check the logs for peer disconnects as a result of invalid messages, at which point the above was found to be an increasingly common occurrence. Best, e > Cheers, > aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
On Tue, Oct 04, 2022 at 05:01:04PM -0700, Eric Voskuil via bitcoin-dev wrote: > [Regarding bandwidth waste: I've pointed out in years past that > breaking the Bitcoin versioning scheme creates a requirement that any > unknown message type be considered valid. Up until a recently-deployed > protocol change, it had always been possible to validate messages by > type. I noticed recently that validating nodes have been dropping peers > at an increasing rate (a consequence of that deployment). Despite being > an undocumented compatibility break, it is now unfortunately a matter > of protocol that a peer must allow its peers to waste its bandwidth to > remain compatible - something which we should eliminate.] The only message listed as not being preceded by a bumped version number in: https://github.com/libbitcoin/libbitcoin-network/wiki/Protocol-Versioning is addrv2 (though addrv2 is gated on mutual exchange of sendaddrv2, so it's presumably the sendaddrv2 message at issue), however since [0] sendaddrv2 messages are only sent to nodes advertising version 70016 or later (same as wtxidrelay). ADDRV2 was introduced May 20 2020 after the 0.20 branch, and SENDADDRV2 gating was merged Dec 9 2020 and included from 0.21.0rc3 onwards. [0] https://github.com/bitcoin/bitcoin/pull/20564 I'm only seeing "bytesrecv_per_msg.*other*" entries for nodes advertising a version of 0.17 and 0.18, which I presume is due to REJECT messages (for taproot txs, perhaps?). Otherwise, I don't think there are any unexpected messages you should be receiving when advertising version 70015 or lower. Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
> Hi, > > Thanks for sharing your thoughts on packaged transaction relay. Hello, thanks for the reply. >> The sole objective, as expressed in the OP proposal, is to: >> "Propagate transactions that are incentive-compatible to mine, even >> if they don't meet minimum feerate alone." > > I actually do think there are additional goals we should include in any > protocol > change involving transaction relay, such as ensuring that we minimize > bandwidth waste as much as possible (as I mentioned in a previous message > in this thread). Yes - there is always the presumption of an optimally-performing protocol (not limited to bandwidth), this is just a restatement from the OP. The OP fails to eliminate orphan announcement, fails to prevent packages with insufficient fee from getting stuck in the same manner as txs (without explicitly re-announcing them again in an even larger package of higher feerate), and results in orphaned package announcements for the same reason (a static package is effectively just a larger tx). Due to the resulting orphaning, a node must allow its peer to continue to broadcast unverifiable orphans to it, potentially chasing ancestry. So in addition to bandwidth waste, there is also an inherent problem of bandwidth DOS. These are problems specifically addressed by packaged relay. [Regarding bandwidth waste: I've pointed out in years past that breaking the Bitcoin versioning scheme creates a requirement that any unknown message type be considered valid. Up until a recently-deployed protocol change, it had always been possible to validate messages by type. I noticed recently that validating nodes have been dropping peers at an increasing rate (a consequence of that deployment). Despite being an undocumented compatibility break, it is now unfortunately a matter of protocol that a peer must allow its peers to waste its bandwidth to remain compatible - something which we should eliminate.] > While I understand your proposal seeks to improve on an idea of static > packages in favor of dynamic package construction based on knowledge a > node should have of its peers, I think the main drawback of your proposal is > that it doesn't take into account the complexities of what a peer's "minimum > feerate" might mean. The consequence of this is that it's not generally > possible for a node to accurately guess whether a given transaction should > be sent in a package to a given peer, or not, and so in addition to any > "packaged transaction relay" mechanism that is implemented by a > transaction announcer, we'd still need to add protocol support for a receiving > peer to retrieve a package as well. It is certainly possible that there is ambiguity in BIP133 (and BIPs that modify it). However it's not clear to me which such ambiguity you are referring to. There is no guessing proposed. > First of all, a node's feerate is a dynamic value. BIP 133 allows for nodes > to > send feefilter messages at any time during the lifetime of a peer connection. > If we were to compare the feerate of ancestors of a relayed transaction to > the feerate in place at a peer as indicated by feefilter messages, and use > that > determine whether those ancestors would have been successfully relayed or > not, then doing so accurately would seem to require caching relay success for > each transaction, for each peer, at the time such transaction is relayed (or > perhaps caching feefilter values that are received from a peer?). This seems > likely to be undesirable, This is a possible implementation. What makes it undesirable? > and, at any rate, is more complex than merely comparing a pair of feerates. There are no necessary protocol changes (though a new INV type is ideal), so the relative complexity you are implying could only arise from implementation. While implementation considerations are relevant, achieving simplicity in the protocol is presumably the priority. Further, implementation complexity must be considered from what is necessary to actually achieve the objectives, and not from the perspective of any given implementation. Merely comparing a pair of feerates produces the problems described above, which includes not resolving the central problem, so this is an apples-to-oranges comparison. It's also more complex than doing nothing, but that also doesn't resolve the problem. > But even more fundamental than feerates being dynamic is that feerate > itself is not a well-defined concept when applied to arbitrary transaction > (sub-)graphs, and this is at the crux of the difficulty in coming up with > proposals that would meet the objective of ensuring that transactions which > are incentive-compatible to mine all get relayed successfully across the > network. Here are a couple examples that illustrate this: > > - Let A be a low feerate transaction (below any peer's minimum feerate). Let > B and C be descendants of A (but are otherwise unrelated). Suppose these >
Re: [bitcoin-dev] Packaged Transaction Relay
(Apologies for the double-post -- I'm resending this message to the list with much of the quoted text trimmed, because my first message was placed in the moderation queue for being too large) Hi, Thanks for sharing your thoughts on packaged transaction relay. The sole objective, as expressed in the OP proposal, is to: "Propagate transactions that are incentive-compatible to mine, even if they > don't meet minimum feerate alone." I actually do think there are additional goals we should include in any protocol change involving transaction relay, such as ensuring that we minimize bandwidth waste as much as possible (as I mentioned in a previous message in this thread). While I understand your proposal seeks to improve on an idea of static packages in favor of dynamic package construction based on knowledge a node should have of its peers, I think the main drawback of your proposal is that it doesn't take into account the complexities of what a peer's "minimum feerate" might mean. The consequence of this is that it's not generally possible for a node to accurately guess whether a given transaction should be sent in a package to a given peer, or not, and so in addition to any "packaged transaction relay" mechanism that is implemented by a transaction announcer, we'd still need to add protocol support for a receiving peer to retrieve a package as well. First of all, a node's feerate is a dynamic value. BIP 133 allows for nodes to send feefilter messages at any time during the lifetime of a peer connection. If we were to compare the feerate of ancestors of a relayed transaction to the feerate in place at a peer as indicated by feefilter messages, and use that determine whether those ancestors would have been successfully relayed or not, then doing so accurately would seem to require caching relay success for each transaction, for each peer, at the time such transaction is relayed (or perhaps caching feefilter values that are received from a peer?). This seems likely to be undesirable, and, at any rate, is more complex than merely comparing a pair of feerates. But even more fundamental than feerates being dynamic is that feerate itself is not a well-defined concept when applied to arbitrary transaction (sub-)graphs, and this is at the crux of the difficulty in coming up with proposals that would meet the objective of ensuring that transactions which are incentive-compatible to mine all get relayed successfully across the network. Here are a couple examples that illustrate this: - Let A be a low feerate transaction (below any peer's minimum feerate). Let B and C be descendants of A (but are otherwise unrelated). Suppose these transactions are relayed to a node in the order A, B, C. In the algorithm you proposed, I believe the determination for whether C should be announced to a given peer as a package (A, C) or as a singleton would either (1) depend on whether the package (A, B) was sufficient to meet the peer's feerate, or (2) waste bandwidth by engaging in packaged relay whenever A was already successfully relayed as part of a package. Both of these approaches seem undesirable. - Let A be a high fee, but low feerate transaction. Suppose B is a transaction that conflicts with A, has a high feerate, but lower total fee. In this situation, two different nodes that learned of these two transactions in opposite order [(A, B) vs (B, A)] might be expected to have differing mempools -- this at least would be the case in the BIP 125 algorithm (which requires that both feerate and total fee must increase when replacing an existing transaction), and at any rate it's not obvious from the information given which would be more optimal to include in a block, as that depends on factors that go beyond just these two transactions. Suppose further that a new transaction C is relayed on the network, which is a child of B and very high feerate, such that B + C have higher fee and higher feerate than A, when taken together. In this case we'd want our relay protocol to ensure that even nodes which saw A first should still have a chance to consider transaction C, but the packaging design you propose (which would compare transaction B's feerate to the peer's, and conclude packaging is unnecessary because B's feerate might already exceed the peer's feerate) would not facilitate this. To summarize, the point I'd make from these examples is that we should not expect that "feerate" (whatever that means) alone will be a sufficient predictor of what is in our peer's mempools. So while there may be some situations where a transaction relayer might be able to usefully package up a transaction with its dependencies (perhaps in narrowly defined situations), there will also always be situations where this isn't possible, and what I conclude from that is that it should be helpful to add to the protocol some way for the recipient of a transaction to request the dependencies directly. Taken together, I roughly understand
Re: [bitcoin-dev] Packaged Transaction Relay
Thanks again for the feedback. Comments inline. > On Sep 27, 2022, at 02:29, alicexbt wrote: > > Hi Eric, > > >> If by "range" you mean a connected tx subgraph, I don't see why not. But >> note that nodes only operate over signed txs. PSBT is a wallet workflow. > > Matt Corallo mentioned that pre-signed transactions created with low fee rate > become an issue when they are broadcasted after a long time and there is a > high demand for block space at that moment. Yes, I understood this. There are many ways that a fee may be created which is too low for propagation. > Example: > > Bob created PSBT1 in a multi party contract with fee rate 5 sat/vbyte however > its taking hours/days to confirm the transaction with such low fee rate. > > Carol created PSBT1 (5 sat/vbyte), PSBT2 (10 sat/vbyte) and PSBT3 (20 > sat/vbyte) for spending same inputs. She broadcasted PSBT3 which got > confirmed in a few minutes. > > >> Always. Only signed transactions are accepted. But assuming you are >> referring to a transaction that has been produced by a pre-signing workflow, >> I'm not sure how this would be distinct from any other tx. > > > `minfeefilter` for all peers of my node was 0.1000 at the time of writing > this email. I am assuming nobody creates pre-signed transaction with fee rate > below 1 sat/vbyte. How often does it happen that `minfeefilter` is above this > default value? I don’t consider node configuration relevant, regardless of its apparent consistency. >> I'm not sure I follow this, maybe you could reword it. But it seems that you >> are saying that CPFP fee-bumping is a problem scenario and the complexity of >> the proposed solutions are not justified by such scenarios. > > > Sorry that sentence was confusing. Yes complexity isn't justified for CPFP > fee-bumping txs below minimum fee rate. > > >> There are many node implementations used presently. And of course these are >> protocol proposals, which presumes more than one implementation. > > > Yes, a few implementations exist (knots, libbitcoin, btcd, bcoin etc.) > however they aren't used by lot of nodes. Based on this [chart][1] 98% nodes > use bitcoin core. Lot of bitcoin protocol proposals are influenced by bitcoin > core contributors and things could be different if even 30% nodes used other > implementations. I don’t consider such a measure relevant. This is a protocol consideration. Also consider that many nodes are not visible, and aspects of nodes, such as for p2p communication, are embedded into applications such as wallets - which could easily exceed the number of visible nodes. >> I don't consider this relevant to any protocol considerations. Miners should >> always be expected to select the most optimal set of txs available in the >> time available to do so. > > > Agree, miners should be expected to select most optimal set of txs. However, > according to one [comment][2] by Pieter Wuille, miners could affect the > security of some bitcoin projects with MEV. This would be a deficiency in such projects, by assuming economic irrationality. The fact that fees will become a greater percentage of the block reward is a surprise to no one. >> Over time we are likely to see that the only policies that remain in >> widespread application are those that are necessary for DOS protection (fee >> rate), as other restrictions are not economically rational and cannot be >> enforced. We've seen recent debate regarding dust policy, and op_return >> policy. "non-standard" txs are perfectly valid but get stuck very easily. >> I'll reiterate, any policy beyond what is published via the protocol will >> cause the above problems. > > I completely agree with this. > > > [1]: https://luke.dashjr.org/programs/bitcoin/files/charts/software.html > [2]: > https://bitcoin.stackexchange.com/questions/107787/front-running-in-bitcoin#comment123441_107796 > > > /dev/fd0 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Packaged Transaction Relay
Hi Eric, > If by "range" you mean a connected tx subgraph, I don't see why not. But note > that nodes only operate over signed txs. PSBT is a wallet workflow. Matt Corallo mentioned that pre-signed transactions created with low fee rate become an issue when they are broadcasted after a long time and there is a high demand for block space at that moment. Example: Bob created PSBT1 in a multi party contract with fee rate 5 sat/vbyte however its taking hours/days to confirm the transaction with such low fee rate. Carol created PSBT1 (5 sat/vbyte), PSBT2 (10 sat/vbyte) and PSBT3 (20 sat/vbyte) for spending same inputs. She broadcasted PSBT3 which got confirmed in a few minutes. > Always. Only signed transactions are accepted. But assuming you are referring > to a transaction that has been produced by a pre-signing workflow, I'm not > sure how this would be distinct from any other tx. `minfeefilter` for all peers of my node was 0.1000 at the time of writing this email. I am assuming nobody creates pre-signed transaction with fee rate below 1 sat/vbyte. How often does it happen that `minfeefilter` is above this default value? > I'm not sure I follow this, maybe you could reword it. But it seems that you > are saying that CPFP fee-bumping is a problem scenario and the complexity of > the proposed solutions are not justified by such scenarios. Sorry that sentence was confusing. Yes complexity isn't justified for CPFP fee-bumping txs below minimum fee rate. > There are many node implementations used presently. And of course these are > protocol proposals, which presumes more than one implementation. Yes, a few implementations exist (knots, libbitcoin, btcd, bcoin etc.) however they aren't used by lot of nodes. Based on this [chart][1] 98% nodes use bitcoin core. Lot of bitcoin protocol proposals are influenced by bitcoin core contributors and things could be different if even 30% nodes used other implementations. > I don't consider this relevant to any protocol considerations. Miners should > always be expected to select the most optimal set of txs available in the > time available to do so. Agree, miners should be expected to select most optimal set of txs. However, according to one [comment][2] by Pieter Wuille, miners could affect the security of some bitcoin projects with MEV. > Over time we are likely to see that the only policies that remain in > widespread application are those that are necessary for DOS protection (fee > rate), as other restrictions are not economically rational and cannot be > enforced. We've seen recent debate regarding dust policy, and op_return > policy. "non-standard" txs are perfectly valid but get stuck very easily. > I'll reiterate, any policy beyond what is published via the protocol will > cause the above problems. I completely agree with this. [1]: https://luke.dashjr.org/programs/bitcoin/files/charts/software.html [2]: https://bitcoin.stackexchange.com/questions/107787/front-running-in-bitcoin#comment123441_107796 /dev/fd0 Sent with Proton Mail secure email. --- Original Message --- On Tuesday, September 27th, 2022 at 2:49 AM, wrote: > > Hi Eric, > > > > This email wasn't answered by anyone on mailing list however I did some > > research about packages yesterday including this email and below are my > > observations, questions etc. > > > Hello, thanks for the reply. > > > > The sole objective, as expressed in the OP proposal, is to: > > > > > > "Propagate transactions that are incentive-compatible to mine, even if > > > they > > > don't meet minimum feerate alone." > > > > According to bitcoinops: Without package relay, it’s not possible to > > effectively CPFP fee bump a transaction that’s below the minimum feerate > > nodes accept. > > > Yes, the problem statement is not in question, just the mechanism of > resolution. The problem of stuck txs arises from minimum fee rate policy, > which is a necessary DOS guard. > > A secondary issue is that of orphan relay. As a node must allow receipt of > orphans, it has no means to differentiate a flood of unconfirmable txs from > those that are confirmable. > > > Matt Corallo's thoughts in a bitcoin core issue: > > > > "Matt Corallo recently wrote about an example on the bitcoin-dev mailing > > list > > involving lightning transactions, where pre-signed transactions might be > > broadcast to the blockchain long after they were generated, and thus not > > have been created with a fee that is sufficient to be confirmed quickly (or > > even be accepted to node mempools). In such situations, channel > > participants may need to use chained transactions (CPFP) in order to > > increase > > the confirmation speed of such transactions, and that implies we may need > > to introduce a mechanism for those parent transactions to be relayed along > > with their higher feerate children, even if the parent transaction would be > > rejected by itself." > >
Re: [bitcoin-dev] Packaged Transaction Relay
> Hi Eric,
>
>
> This email wasn't answered by anyone on mailing list however I did some
> research about packages yesterday including this email and below are my
> observations, questions etc.
Hello, thanks for the reply.
> > The sole objective, as expressed in the OP proposal, is to:
> >
> > "Propagate transactions that are incentive-compatible to mine, even if they
> don't meet minimum feerate alone."
>
> According to [bitcoinops][1]: Without package relay, it’s not possible to
> effectively CPFP fee bump a transaction that’s below the minimum feerate
> nodes accept.
Yes, the problem statement is not in question, just the mechanism of
resolution. The problem of stuck txs arises from minimum fee rate policy, which
is a necessary DOS guard.
A secondary issue is that of orphan relay. As a node must allow receipt of
orphans, it has no means to differentiate a flood of unconfirmable txs from
those that are confirmable.
> Matt Corallo's thoughts in a bitcoin core [issue][2]:
>
> "Matt Corallo recently wrote about an example on the bitcoin-dev mailing list
> involving lightning transactions, where pre-signed transactions might be
> broadcast to the blockchain long after they were generated, and thus not
> have been created with a fee that is sufficient to be confirmed quickly (or
> even be accepted to node mempools). In such situations, channel
> participants may need to use chained transactions (CPFP) in order to increase
> the confirmation speed of such transactions, and that implies we may need
> to introduce a mechanism for those parent transactions to be relayed along
> with their higher feerate children, even if the parent transaction would be
> rejected by itself."
While this is a valid scenario, the problems directly affect Bitcoin. Those
problems propagate to layers, but are not unique to layering.
> 1)Is it possible to have multiple pre-signed transactions with different fee
> rates in a range? Example: PSBT1: 5 sat/vbyte, PSBT2: 10 sat/vbyte, PSBT3: 20
> sat/vbyte and PSBT4: 100 sat/vbyte
If by "range" you mean a connected tx subgraph, I don't see why not. But note
that nodes only operate over signed txs. PSBT is a wallet workflow.
> 2)How would covenants affect this problem?
There are a good number of covenant proposals, though I assume they are all
implemented within script. If a tx is confirmable and satisfies fee rate (for
DOS protection), it is relayable. Covenants affect confirmability and should
not have any unique impact on relay.
> 3)How often does it happen that a pre-signed tx gets rejected by nodes
> because it did not meet the minimum fee rate? Is it predictable and could be
> managed in a different way?
Always. Only signed transactions are accepted. But assuming you are referring
to a transaction that has been produced by a pre-signing workflow, I'm not sure
how this would be distinct from any other tx.
> After reading several links related to packages and bitcoin core pull
> requests,
> I found it anti-bitcoin to introduce so much complexity because its not
> possible to CPFP fee bump a tx below minimum fee rate.
I'm not sure I follow this, maybe you could reword it. But it seems that you
are saying that CPFP fee-bumping is a problem scenario and the complexity of
the proposed solutions are not justified by such scenarios.
I would say that the problem is real, and that the least complex option is
generally preferred. There are always tradeoffs, and balancing these is part of
protocol development. But as a rule, complexity within a protocol
(communication) is to be avoided where possible.
> > Furthermore any tx that is "stuck" can be freed by simply sending another
> tx. The nodes at which the tx has become stuck will just package it up and
> relay it to peers. In other words, there is no impact on wallet implementation
> apart from raising the aggregate fee using a descendant transaction.
>
> It is easy to send another tx if there is only one user involved however
> packages are trying to fix issues in which multiple users and transaction pre-
> signed between them are involved. So, it will be difficult to coordinate and
> create new pre-signed transactions in some cases although it is possible for
> some use cases.
Given that nodes do not deal in presigned txs, this coordination difficulty
could not be increased in any scenario.
A node produces sets of txs ("packages") dynamically to satisfy its peer's
feerate. When a wallet broadcasts a tx/package to a node, it is operating as a
peer on the p2p network. The wallet simply implements the same dynamic
packaging algorithm as any peer - because it is a peer.
> > This is barely a protocol change - it's primarily implementation. All that
> should be required is an additional INV element type, such as
> MSG_TX_PACKAGE.
>
> > * All elements of MSG_TX_PACKAGE in one INV message MUST to be of
> the same package.
> > * A package MUST must define a set that can be mined into one block
>
Re: [bitcoin-dev] Packaged Transaction Relay
Hi Eric, This email wasn't answered by anyone on mailing list however I did some research about packages yesterday including this email and below are my observations, questions etc. > The sole objective, as expressed in the OP proposal, is to: > > "Propagate transactions that are incentive-compatible to mine, even if they > don't meet minimum feerate alone." According to [bitcoinops][1]: Without package relay, it’s not possible to effectively CPFP fee bump a transaction that’s below the minimum feerate nodes accept. Matt Corallo's thoughts in a bitcoin core [issue][2]: "Matt Corallo recently wrote about an example on the bitcoin-dev mailing list involving lightning transactions, where pre-signed transactions might be broadcast to the blockchain long after they were generated, and thus not have been created with a fee that is sufficient to be confirmed quickly (or even be accepted to node mempools). In such situations, channel participants may need to use chained transactions (CPFP) in order to increase the confirmation speed of such transactions, and that implies we may need to introduce a mechanism for those parent transactions to be relayed along with their higher feerate children, even if the parent transaction would be rejected by itself." 1)Is it possible to have multiple pre-signed transactions with different fee rates in a range? Example: PSBT1: 5 sat/vbyte, PSBT2: 10 sat/vbyte, PSBT3: 20 sat/vbyte and PSBT4: 100 sat/vbyte 2)How would covenants affect this problem? 3)How often does it happen that a pre-signed tx gets rejected by nodes because it did not meet the minimum fee rate? Is it predictable and could be managed in a different way? After reading several links related to packages and bitcoin core pull requests, I found it anti-bitcoin to introduce so much complexity because its not possible to CPFP fee bump a tx below minimum fee rate. > Furthermore any tx that is "stuck" can be freed by simply sending another tx. > The nodes at which the tx has become stuck will just package it up and relay > it to peers. In other words, there is no impact on wallet implementation > apart from raising the aggregate fee using a descendant transaction. It is easy to send another tx if there is only one user involved however packages are trying to fix issues in which multiple users and transaction pre-signed between them are involved. So, it will be difficult to coordinate and create new pre-signed transactions in some cases although it is possible for some use cases. > This is barely a protocol change - it's primarily implementation. All that > should be required is an additional INV element type, such as MSG_TX_PACKAGE. > * All elements of MSG_TX_PACKAGE in one INV message MUST to be of the same > package. > * A package MUST must define a set that can be mined into one block > (size/sigops constraint). > * A package SHOULD not contain confirmed txs (a race may cause this). > * A package MUST minimally satisfy peer.feerate. > * A partial tx order, as in the manner of the block.txs ordering, MUST be > imposed. > * A node SHOULD drop a peer that sends a package (or tx) below node.feerate. > * A node MAY drop a peer that sends a non-minimal package according to > node.feerate. This makes sense particularly if multiple node implementations are used in future. My other questions: a)If a package has tx1, tx2, tx3, tx4 and tx5 and miner just include tx1 and tx2 in the block, how does this affect the projects considered for packages proposal? b)How does changing the order of txs in a package affect these transactions? c)Do packages introduce more attack vectors in bitcoin for front running or MEV? MEV in bitcoin currently only affects the projects that are considered in packages proposal. d)What if the package contains a transactions with sanctioned address? e)Why would miners use packages if the existing scenario in terms of fees per block is beneficial for them? /dev/fd0 [1]: https://bitcoinops.org/en/topics/package-relay/ [2]: https://github.com/bitcoin/bitcoin/issues/14895 Sent with Proton Mail secure email. --- Original Message --- On Thursday, June 9th, 2022 at 4:13 AM, Eric Voskuil via bitcoin-dev wrote: > Hi Suhas/Gloria, > > Good questions. I've started a new thread because it became something else... > > Various ideas about packaging seem to be focused on the idea of an atomic > message that is gossiped around the network like a transaction or block. From > my perspective that seems to create a set of problems without good solutions, > and it is not a proper analogy to those atomic structures. It may be worth > taking the time to step back and take a close look at the underlying > objective. > > The sole objective, as expressed in the OP proposal, is to: > > "Propagate transactions that are incentive-compatible to mine, even if they > don't meet minimum feerate alone." > > Effectively producing this outcome with an
