Re: [bitcoin-dev] Bitcoin Transaction Relay over Nostr
Hi Joost, David, In my mind, weak blocks' main benefit would be that it improves block relay by giving PoW-hints on what are in miner's mempools. Non-standard transactions could even be cached(even if not validated until block inclusion), which would tolerate more heterogeneity in policies without drastically increasing relay times. Of course, it can also have the side effect of gossiping better transaction packages, though I think this would be a ton of work to really take advantage of. Perhaps we might be able to do better in a post-cluster-mempool world, gossiping chunks. At present I think energy would be best spent writing a weak blocks BIP proposal, since one has never been written before(?), and it would be fairly trivial to swap out p2p things with RPC calls if so desired for fast experimentation over alternative relays. Cheers, Greg On Tue, May 30, 2023 at 8:58 AM Joost Jager via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Hi David, > > >> A block template is an ordered list of raw transactions that can all be >> included in the next block (with some space reserved for a coinbase >> transaction). A full node can validate those transactions and calculate >> how much fee they pay. A Nostr relay can simply relay almost[1] any >> template that pays more fees than the previous best template it saw for >> the next block. That can be more flexible than the current >> implementation of submitblock with package relay which still enforces a >> lot of the rules that helps keep a regular relay node safe from DoS and >> a miner node able to select mineable transactions quickly. >> > > Interesting idea! This would also make it easy for external services to > try to do the best possible block building using advanced algorithms. > Miners would just select the best template available from various sources > including nostr. > > >> A weak block is a block whose header doesn't quite hash to low enough of >> a value to be included on the chain. It still takes an extraordinary >> amount of hashrate to produce, so it's inherently DoS resistant. If >> miners are producing block that include transactions not seen by typical >> relay nodes, that can reduce the efficiency and effectiveness of BIP152 >> compact block relay, which hurts the profitability of miners of custom >> blocks. To compensate, miners could relay weak blocks through Nostr to >> full nodes and other miners so that they could quickly relay and accept >> complete blocks that later included the same custom transactions. This >> would also help fee estimation and provide valuable insights to those >> trying to get their transactions included into the next block. >> > > I believe this would be useful right away, wouldn't it? Looking at > mempool.space's block audit, there are definitely blocks that have a > "surprising" content and might take long to download. > > The anti-dos measures that you describe for both weak blocks and block > templates seem very robust, but they would require a more intelligent nostr > relay to enforce. Not sure if it is still allowed to call it nostr at that > point. Perhaps it becomes more of a specialised bitcoin relay. btcstr - > "bitcoin stuff transmitted by relays". > > Regarding size, the block template and weak block could both be sent in >> BIP152 compact block format as a diff against the expected contents of a >> typical node, allowing Alice to send just a small amount of additional >> data for relay over what she'd have to send anyway for each transaction >> in a package. (Although it's quite possible that BetterHash or Stratum >> v2 have even better solutions, possibly already implemented.) >> > > Sounds like a great way to repurpose what already exists to reduce > resource usage for these additional message types. > > >> If nothing else, I think Nostr could provide an interesting playground >> for experimenting with various relay and mining ideas we've talked about >> for years, so thanks again for working on this! >> > > I think so too! The main question on my mind though is how to actually > make this work. There is a bit of a chicken-egg problem here with users and > miners possibly waiting for each other to adopt. > > Joost > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Bitcoin Transaction Relay over Nostr
Hi David, > A block template is an ordered list of raw transactions that can all be > included in the next block (with some space reserved for a coinbase > transaction). A full node can validate those transactions and calculate > how much fee they pay. A Nostr relay can simply relay almost[1] any > template that pays more fees than the previous best template it saw for > the next block. That can be more flexible than the current > implementation of submitblock with package relay which still enforces a > lot of the rules that helps keep a regular relay node safe from DoS and > a miner node able to select mineable transactions quickly. > Interesting idea! This would also make it easy for external services to try to do the best possible block building using advanced algorithms. Miners would just select the best template available from various sources including nostr. > A weak block is a block whose header doesn't quite hash to low enough of > a value to be included on the chain. It still takes an extraordinary > amount of hashrate to produce, so it's inherently DoS resistant. If > miners are producing block that include transactions not seen by typical > relay nodes, that can reduce the efficiency and effectiveness of BIP152 > compact block relay, which hurts the profitability of miners of custom > blocks. To compensate, miners could relay weak blocks through Nostr to > full nodes and other miners so that they could quickly relay and accept > complete blocks that later included the same custom transactions. This > would also help fee estimation and provide valuable insights to those > trying to get their transactions included into the next block. > I believe this would be useful right away, wouldn't it? Looking at mempool.space's block audit, there are definitely blocks that have a "surprising" content and might take long to download. The anti-dos measures that you describe for both weak blocks and block templates seem very robust, but they would require a more intelligent nostr relay to enforce. Not sure if it is still allowed to call it nostr at that point. Perhaps it becomes more of a specialised bitcoin relay. btcstr - "bitcoin stuff transmitted by relays". Regarding size, the block template and weak block could both be sent in > BIP152 compact block format as a diff against the expected contents of a > typical node, allowing Alice to send just a small amount of additional > data for relay over what she'd have to send anyway for each transaction > in a package. (Although it's quite possible that BetterHash or Stratum > v2 have even better solutions, possibly already implemented.) > Sounds like a great way to repurpose what already exists to reduce resource usage for these additional message types. > If nothing else, I think Nostr could provide an interesting playground > for experimenting with various relay and mining ideas we've talked about > for years, so thanks again for working on this! > I think so too! The main question on my mind though is how to actually make this work. There is a bit of a chicken-egg problem here with users and miners possibly waiting for each other to adopt. Joost ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Bitcoin Transaction Relay over Nostr
On 2023-05-22 21:19, Joost Jager via bitcoin-dev wrote: A notable advantage of this approach is that it delegates the responsibility of dealing with Denial-of-Service (DoS) threats to the relays themselves. They could, for example, require a payment to mitigate such concerns. Hi Joost, Thanks for working on this! One quick thought I had was that a possibly interesting avenue for exploration would be that, in addition to relaying individual transactions or packages, it might be worth relaying block templates and weak blocks as both of those provide inherent DoS resistance and can offer useful features. A block template is an ordered list of raw transactions that can all be included in the next block (with some space reserved for a coinbase transaction). A full node can validate those transactions and calculate how much fee they pay. A Nostr relay can simply relay almost[1] any template that pays more fees than the previous best template it saw for the next block. That can be more flexible than the current implementation of submitblock with package relay which still enforces a lot of the rules that helps keep a regular relay node safe from DoS and a miner node able to select mineable transactions quickly. A weak block is a block whose header doesn't quite hash to low enough of a value to be included on the chain. It still takes an extraordinary amount of hashrate to produce, so it's inherently DoS resistant. If miners are producing block that include transactions not seen by typical relay nodes, that can reduce the efficiency and effectiveness of BIP152 compact block relay, which hurts the profitability of miners of custom blocks. To compensate, miners could relay weak blocks through Nostr to full nodes and other miners so that they could quickly relay and accept complete blocks that later included the same custom transactions. This would also help fee estimation and provide valuable insights to those trying to get their transactions included into the next block. Regarding size, the block template and weak block could both be sent in BIP152 compact block format as a diff against the expected contents of a typical node, allowing Alice to send just a small amount of additional data for relay over what she'd have to send anyway for each transaction in a package. (Although it's quite possible that BetterHash or Stratum v2 have even better solutions, possibly already implemented.) If nothing else, I think Nostr could provide an interesting playground for experimenting with various relay and mining ideas we've talked about for years, so thanks again for working on this! -Dave [1] In addition to validating transactions, a relay would probably want to reject templates that contained transactions that took excessively long to validate (which could cause a block including them to become stale) or that included features reserved for upgrades (as a soft fork that happened before the relay's node was upgraded might make that block invalid). ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Bitcoin Transaction Relay over Nostr
Hi fd0, > - Transactions could be encrypted when published as nostr events initially > except size, fee rate and offer. This can be used by different clients to > show them as external mempool with transactions sorted by fee rate without > affecting privacy of users. > I don't think this will work because those encrypted transactions could all be fake and distort the view clients have on this 'mempool'? > - Mining pools will be incentivized to include these transaction in their > blocks if they are using a higher fee rate compared to transactions in > normal mempool used by bitcoin nodes or there is a mechanism to accept > published offers, NIP4 is used to privately coordinate everything between > user and pool. User can lock some sats in a 2of2 multisig and release it to > mining pool on confirmation. > I believe you are suggesting out-of-band payment in case the fee included in the transaction itself is insufficient and CPFP/RBF is impossible or impractical? The question is why would the miner trust you to indeed release after confirmation? The 2-of-2 presumably has a clawback clause with a timeout that could be used by the user to avoid paying. Joost ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Bitcoin Transaction Relay over Nostr
Hi Joost, Transaction relay over nostr sounds interesting. I have 2 suggestions: - Transactions could be encrypted when published as nostr events initially except size, fee rate and offer. This can be used by different clients to show them as external mempool with transactions sorted by fee rate without affecting privacy of users. - Mining pools will be incentivized to include these transaction in their blocks if they are using a higher fee rate compared to transactions in normal mempool used by bitcoin nodes or there is a mechanism to accept published offers, NIP4 is used to privately coordinate everything between user and pool. User can lock some sats in a 2of2 multisig and release it to mining pool on confirmation. /dev/fd0 floppy disk guy Sent with Proton Mail secure email. --- Original Message --- On Tuesday, May 23rd, 2023 at 12:49 PM, Joost Jager via bitcoin-dev wrote: > Hi, > > > > I write to get your thoughts on an alternative approach for Bitcoin > transaction relay, addressing some of the limitations in the current > peer-to-peer transaction relay system. To the best of my knowledge, the > credit for the original concept goes to Ben Carman. I felt it would be > beneficial to share the idea on this list to garner wider perspectives and > feedback. > > > > The existing peer-to-peer (P2P) transaction relay system comes with a set of > limitations that may negatively impact applications, notably those like > Lightning that make extensive use of pre-signed transactions. A key > limitation lies in the system's inability to relay transaction packages. This > constraint can lead to HTLCs expiring before being swept, thereby risking > fund losses. In addition, the P2P system falls short in supporting > non-standard transactions, despite an established demand for such > transactions in the marketplace. > > > > Nostr, an open and decentralized network of relays for public and ephemeral > messages between pseudonymous entities, could help address these > shortcomings. With the standards defined in NIP-89 [1], it becomes possible > to broadcast arbitrary Bitcoin transaction packages, overcoming one of the > key hurdles in the current relay system. > > > > In this proposed alternative relay mechanism, miners would listen for these > broadcasted transaction packages and insert the packages into their local > mempool. They can take advantage of the `submitpackage` RPC, limited to safe > topologies only - specifically child and direct parents, tree only [2]. This > feature could serve as an interim solution for package relay until it becomes > available through the traditional P2P method. > > > > A notable advantage of this approach is that it delegates the responsibility > of dealing with Denial-of-Service (DoS) threats to the relays themselves. > They could, for example, require a payment to mitigate such concerns. There > are in fact paid nostr relays already in operation. This partitioning would > result in a clear separation between the Bitcoin transaction layer and DoS > protection, introducing more flexibility in the system and potentially > boosting its resilience. > > > > Implementing Nostr as a relay mechanism also has the potential to democratize > access to miner mempools, thus leveling the playing field in the Bitcoin > network. In the current state, those with direct connections or certain > privileges can more readily submit transactions to miners, perhaps even > through means as informal as email. > > > > I have been working on a prototype of this concept (based on [3]) and have > captured its workings in a demonstration video [4]. > > > > Joost > > > > [1] https://github.com/nostr-protocol/nips/pull/476 > > [2] https://github.com/bitcoin/bitcoin/pull/27609#issuecomment-1544414801 > > [3] https://github.com/benthecarman/nostr-tx-broadcast > > [4] https://twitter.com/joostjgr/status/1658487013237211155 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] Bitcoin Transaction Relay over Nostr
Hi, I write to get your thoughts on an alternative approach for Bitcoin transaction relay, addressing some of the limitations in the current peer-to-peer transaction relay system. To the best of my knowledge, the credit for the original concept goes to Ben Carman. I felt it would be beneficial to share the idea on this list to garner wider perspectives and feedback. The existing peer-to-peer (P2P) transaction relay system comes with a set of limitations that may negatively impact applications, notably those like Lightning that make extensive use of pre-signed transactions. A key limitation lies in the system's inability to relay transaction packages. This constraint can lead to HTLCs expiring before being swept, thereby risking fund losses. In addition, the P2P system falls short in supporting non-standard transactions, despite an established demand for such transactions in the marketplace. Nostr, an open and decentralized network of relays for public and ephemeral messages between pseudonymous entities, could help address these shortcomings. With the standards defined in NIP-89 [1], it becomes possible to broadcast arbitrary Bitcoin transaction packages, overcoming one of the key hurdles in the current relay system. In this proposed alternative relay mechanism, miners would listen for these broadcasted transaction packages and insert the packages into their local mempool. They can take advantage of the `submitpackage` RPC, limited to safe topologies only - specifically child and direct parents, tree only [2]. This feature could serve as an interim solution for package relay until it becomes available through the traditional P2P method. A notable advantage of this approach is that it delegates the responsibility of dealing with Denial-of-Service (DoS) threats to the relays themselves. They could, for example, require a payment to mitigate such concerns. There are in fact paid nostr relays already in operation. This partitioning would result in a clear separation between the Bitcoin transaction layer and DoS protection, introducing more flexibility in the system and potentially boosting its resilience. Implementing Nostr as a relay mechanism also has the potential to democratize access to miner mempools, thus leveling the playing field in the Bitcoin network. In the current state, those with direct connections or certain privileges can more readily submit transactions to miners, perhaps even through means as informal as email. I have been working on a prototype of this concept (based on [3]) and have captured its workings in a demonstration video [4]. Joost [1] https://github.com/nostr-protocol/nips/pull/476 [2] https://github.com/bitcoin/bitcoin/pull/27609#issuecomment-1544414801 [3] https://github.com/benthecarman/nostr-tx-broadcast [4] https://twitter.com/joostjgr/status/1658487013237211155 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
