Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On Wed, Sep 15, 2021 at 08:24:43AM -0700, Matt Corallo via bitcoin-dev wrote: > > On Sep 13, 2021, at 21:56, Anthony Towns wrote: > > I'm not sure that's really the question you want answered? > Of course it is? I’d like to understand the initial thinking and design > analysis that went into this decision. That seems like an important question > to ask when seeking changes in an existing system :). Well, "are there any drawbacks to doing X instead, because that would make it easier for me to do Y" just seems like a more interesting question? Because: > > Mostly > > it's just "this is how mainnet works" plus "these are the smallest > > changes to have blocks be chosen by a signature, rather than entirely > > by PoW competition". doesn't seem like that interesting an answer... To be a bit more specific, it's not at all clear to me what you would be happy with? (I mean, beyond "something magic that works exactly how I want it, when I want it, even if I don't know what that is yet or change my mind later" which is obviously the desired behaviour for all software everywhere) You say you're happier with both mainnet and testnet3 than signet, but mainnet isn't any faster than signet, while (if you've got an ASIC) testnet3 will give you a block per second, especially if you don't mind your blocks getting reorged out. There's a lot of ground between those two extremes. > > For integration testing across many services, I think a ten-minute-average > > between blocks still makes sense -- protocols relying on CSV/CLTV to > > ensure there's a delay they can use to recover funds, if they specify > > that in blocks (as lightning's to_self_delay does), then significant > > surges of blocks will cause uninteresting bugs. > Hmm, why would blocks coming quicker lead to a bug? I certainly hope no one > has a bug if their block time is faster than per ten minutes. I presume here, > you mean something like “if the node can’t keep up with the block rate”, but > I certainly hope the benchmark for may isn’t 10 minutes, or really even one. The lightning to_self_delay is specified in blocks, but is meant to allow you to be offline for some real time period; if you specify 1000 blocks and are sure you'll be online every two days, that's fine on mainnet and signet as it stands, but broken on testnet. > > It would be easy enough to change things to target an average of 2 or > > 5 minutes, I suppose, but then you'd probably need to propogate that > > logic back into your apps that would otherwise think 144 blocks is around > > about a day. > Why? One useful thing for testing is compressing real time. Sure, but if you're compressing _real_ time you need to manipulate the nTime not just the number of blocks -- and that might be relevant for nLocktime or nSequence checks by mtp rather than height. But that's not something signet's appropriate for: you should be using regtest for that scenario. > > We could switch back to doing blocks exactly every 10 minutes, rather > > than a poisson-ish distribution in the range of 1min to 60min, but that > > doesn't seem like that huge a win, and makes it hard to test that things > > behave properly when blocks arrive in bursts. > Hmm, I suppose? If you want to test that the upper bound doesn’t > need to be 100 minutes, though, it could be 10. Mathematically, you can't have an average of 10 minutes and a max of 10 minutes without the minimum also being 10 minutes... > > Best of luck to you then? Nobody's trying to sell you on a subscription > > plan to using signet. > lol, yes, I’m aware of that, nor did I mean to imply that anything has to be > targeted at a specific person’s requirements. Rather, my point here is that > I’m really confused as to who the target user *is*, because we should be > building products with target users in mind, even if those targets are often > “me” for open source projects. I don't really think there's a definitive answer to that yet? My guess is "integration testing" is close to right; whether it be different services validating they interoperate, or users seeing if a service works the way they expect in a nearly-live environment. For private signets, the advantage over regtest is you don't risk some random participant causing major reorgs, and can reasonably use it over the internet without having to worry too much about securing things. For the default public signet, the advantage over regtest is probably that you've got additional infrastructure already setup (eg explorer.bc-2.jp and mempool.space/signet, perhaps eventually a decent lightning test network? there's signet-lightning.wakiyamap.dev) The advantage of a private signet vs the default public one is probably only that you can control the consensus rules to introduce and test a new soft fork if you want. The advantage of the default public signet over your own private one is probably mostly that it already has miners/explorers/faucets setup and you don't have to do that yourself. I
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> On Sep 13, 2021, at 21:56, Anthony Towns wrote: > I'm not sure that's really the question you want answered? Of course it is? I’d like to understand the initial thinking and design analysis that went into this decision. That seems like an important question to ask when seeking changes in an existing system :). > Mostly > it's just "this is how mainnet works" plus "these are the smallest > changes to have blocks be chosen by a signature, rather than entirely > by PoW competition". > > For integration testing across many services, I think a ten-minute-average > between blocks still makes sense -- protocols relying on CSV/CLTV to > ensure there's a delay they can use to recover funds, if they specify > that in blocks (as lightning's to_self_delay does), then significant > surges of blocks will cause uninteresting bugs. Hmm, why would blocks coming quicker lead to a bug? I certainly hope no one has a bug if their block time is faster than per ten minutes. I presume here, you mean something like “if the node can’t keep up with the block rate”, but I certainly hope the benchmark for may isn’t 10 minutes, or really even one. > It would be easy enough to change things to target an average of 2 or > 5 minutes, I suppose, but then you'd probably need to propogate that > logic back into your apps that would otherwise think 144 blocks is around > about a day. Why? One useful thing for testing is compressing real time. More broadly, the only issues that I’ve heard around block times in testnet3 are the inconsistency and, rarely software failing to keep up at all. > We could switch back to doing blocks exactly every 10 minutes, rather > than a poisson-ish distribution in the range of 1min to 60min, but that > doesn't seem like that huge a win, and makes it hard to test that things > behave properly when blocks arrive in bursts. Hmm, I suppose? If you want to test that the upper bound doesn’t need to be 100 minutes, though, it could be 10. > Best of luck to you then? Nobody's trying to sell you on a subscription > plan to using signet. lol, yes, I’m aware of that, nor did I mean to imply that anything has to be targeted at a specific person’s requirements. Rather, my point here is that I’m really confused as to who the target user *is*, because we should be building products with target users in mind, even if those targets are often “me” for open source projects. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On Sun, Sep 12, 2021 at 10:33:24PM -0700, Matt Corallo via bitcoin-dev wrote: > > On Sep 12, 2021, at 00:53, Anthony Towns wrote: > >> Why bother with a version bit? This seems substantially more complicated > >> than the original proposal that surfaced many times before signet launched > >> to just have a different reorg signing key. > > Yeah, that was the original idea, but there ended up being two problems > > with that approach. The simplest is that the signet block signature > > encodes the signet challenge, > But if that was the originally proposal, why is the challenge committed to in > the block? :) The answer to your question was in the text after the comma, that you deleted... > > Blocks on signet get mined at a similar rate to mainnet, so you'll always > > have to wait a little bit (up to an hour) -- if you don't want to wait > > at all, that's what regtest (or perhaps a custom signet) is for. > Can you explain the motivation for this? I'm not sure that's really the question you want answered? Mostly it's just "this is how mainnet works" plus "these are the smallest changes to have blocks be chosen by a signature, rather than entirely by PoW competition". For integration testing across many services, I think a ten-minute-average between blocks still makes sense -- protocols relying on CSV/CLTV to ensure there's a delay they can use to recover funds, if they specify that in blocks (as lightning's to_self_delay does), then significant surges of blocks will cause uninteresting bugs. It would be easy enough to change things to target an average of 2 or 5 minutes, I suppose, but then you'd probably need to propogate that logic back into your apps that would otherwise think 144 blocks is around about a day. We could switch back to doing blocks exactly every 10 minutes, rather than a poisson-ish distribution in the range of 1min to 60min, but that doesn't seem like that huge a win, and makes it hard to test that things behave properly when blocks arrive in bursts. > From where I sit, as far as I know, I should basically be a prime > example of the target market for public signet - someone developing > bitcoin applications with regular requirements to test those applications > with other developers without jumping through hoops to configure software > the same across the globe and set up miners. With blocks being slow and > irregular, I’m basically not benefited at all by signet and will stick > with testnet3/mainnet testing, which both suck. Best of luck to you then? Nobody's trying to sell you on a subscription plan to using signet. Signet's less expensive in fees (or risk) than mainnet, and takes far less time for IBD than testnet, but if those aren't blockers for you, that's great. Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> On Sep 13, 2021, at 05:30, Michael Folkson wrote: > > >> >> Can you explain the motivation for this? From where I sit, as far as I know, >> I should basically be > a prime example of the target market for public >> signet - someone developing bitcoin applications > with regular requirements >> to test those applications with other developers without >> jumping through hoops to configure software the same across the globe and >> set up miners. >> With blocks > being slow and irregular, I’m basically not benefited at all >> by signet and will >> stick with testnet3/mainnet testing, which both suck. > > On testnet3 you can realistically go days without blocks being found > (and conversely thousands of blocks can be found in a day), the block > discovery time variance is huge. Of course this is probabilistically > possible on mainnet too but the probability of this happening is close > to zero. Here[0] is an example of 16,000 blocks being found in a day > on testnet3. Blocks too fast isn’t generally an issue when waiting for blocks to test, and hooking up a miner is probably less work on testnet3 than creating a multi-party private signet with miners. In any case, you didn’t address the substance of the point - we can do better to make it a good platform for testing…. Why aren’t we? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> Can you explain the motivation for this? From where I sit, as far as I know, > I should basically be > a prime example of the target market for public > signet - someone developing bitcoin applications > with regular requirements > to test those applications with other developers without > jumping through hoops to configure software the same across the globe and set > up miners. > With blocks > being slow and irregular, I’m basically not benefited at all by > signet and will > stick with testnet3/mainnet testing, which both suck. On testnet3 you can realistically go days without blocks being found (and conversely thousands of blocks can be found in a day), the block discovery time variance is huge. Of course this is probabilistically possible on mainnet too but the probability of this happening is close to zero. Here[0] is an example of 16,000 blocks being found in a day on testnet3. On signet block discovery time variance mirrors mainnet. On mainnet you are risking Bitcoin with actual monetary value. If you don't mind doing this then you don't need testnet3, signet or anything else. In addition proposed soft forks may be activated on signet (and could also be on testnet3) well before they are considered for activation on mainnet for testing and experimentation purposes. [0] https://web.archive.org/web/20160910173004/https://blog.blocktrail.com/2015/04/in-the-darkest-depths-of-testnet3-16k-blocks-were-found-in-1-day/ -- Michael Folkson Email: michaelfolk...@gmail.com Keybase: michaelfolkson PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> On Sep 12, 2021, at 00:53, Anthony Towns wrote: > > On Thu, Sep 09, 2021 at 05:50:08PM -0700, Matt Corallo via bitcoin-dev wrote: >>> AJ proposed to allow SigNet users to opt-out of reorgs in case they >>> explicitly want to remain unaffected. This can be done by setting a >>> to-be-reorged version bit [...] >> Why bother with a version bit? This seems substantially more complicated >> than the original proposal that surfaced many times before signet launched >> to just have a different reorg signing key. > > Yeah, that was the original idea, but there ended up being two problems > with that approach. The simplest is that the signet block signature > encodes the signet challenge, But if that was the originally proposal, why is the challenge committed to in the block? :) > So using the RECENT_CONSENSUS_CHANGE behaviour that avoids the > discourage/disconnect logic seems the way to avoid that problem, and that > means making it so that nodes that that opt-out of reorgs can distinguish > valid-but-will-become-stale blocks from invalid blocks. Using a versionbit > seems like the easiest way of doing that. Sure, you could set that for invalid block signatures as well though. It’s not really a material DoS protection one way or the other. >>> The reorg-interval X very much depends on the user's needs. One could >>> argue that there should be, for example, three reorgs per day, each 48 >>> blocks apart. Such a short reorg interval allows developers in all time >>> zones to be awake during one or two reorgs per day. Developers don't >>> need to wait for, for example, a week until they can test their reorgs >>> next. However, too frequent reorgs could hinder other SigNet users. >> I see zero reason whatsoever to not simply reorg ~every block, or as often >> as is practical. If users opt in to wanting to test with reorgs, they should >> be able to test with reorgs, not wait a day to test with reorgs. > > Blocks on signet get mined at a similar rate to mainnet, so you'll always > have to wait a little bit (up to an hour) -- if you don't want to wait > at all, that's what regtest (or perhaps a custom signet) is for. Can you explain the motivation for this? From where I sit, as far as I know, I should basically be a prime example of the target market for public signet - someone developing bitcoin applications with regular requirements to test those applications with other developers without jumping through hoops to configure software the same across the globe and set up miners. With blocks being slow and irregular, I’m basically not benefited at all by signet and will stick with testnet3/mainnet testing, which both suck. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> Sometimes that reorg could be deeper if you would be lucky enough to get a block with more work than N following blocks combined Each block is credited for its contribution to total chainwork by the difficulty target, not the hash of the block itself. On Sun, Sep 12, 2021 at 10:42 PM vjudeu via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > > - 1 block reorgs: these are a regular feature on mainnet, everyone >should cope with them; having them happen multiple times a day to >make testing easier should be great > > Anyone can do 1 block reorg, because nonce is not signed, so anyone can > replace that with better value. For example, if you have block > 0086d6b2636cb2a392d45edc4ec544a10024d30141c9adf4bfd9de533b53 with > 0x0007f4cc nonce, you can replace that with 0x00110241 nonce and get > 00096a1c4239d994547185c80308a552cba85d5bd28a51e9dc583ae5eadb block, > where everything is identical, except the nonce. > > Sometimes that reorg could be deeper if you would be lucky enough to get a > block with more work than N following blocks combined. > > On 2021-09-08 09:59:29 user Anthony Towns via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > > On Tue, Sep 07, 2021 at 06:07:47PM +0200, 0xB10C via bitcoin-dev wrote: > > The reorg-interval X very much depends on the user's needs. One could > > argue that there should be, for example, three reorgs per day, each 48 > > blocks apart. > > Oh, wow, I think the last suggestion was every 100 blocks (every > ~16h40m). Once every ~8h sounds very convenient. > > > Such a short reorg interval allows developers in all time > > zones to be awake during one or two reorgs per day. > > And also for there to reliably be reorgs when they're not awake, which > might be a useful thing to be able to handle, too :) > > > Developers don't > > need to wait for, for example, a week until they can test their reorgs > > next. However, too frequent reorgs could hinder other SigNet users. > > Being able to run `bitcoind -signet -signetacceptreorg=0` and never > seeing any reorgs should presumably make this not a problem? > > For people who do see reorgs, having an average of 2 or 3 additional > blocks every 48 blocks is perhaps a 6% increase in storage/traffic. > > > # Scenario 1: Race between two chains > > > > For this scenario, at least two nodes and miner scripts need to be > > running. An always-miner A continuously produces blocks and rejects > > blocks with the to-be-reorged version bit flag set. And a race-miner R > > that only mines D blocks at the start of each interval and then waits X > > blocks. A and R both have the same hash rate. Assuming both are well > > connected to the network, it's random which miner will first mine and > > propagate a block. In the end, the A miner chain will always win the > race. > > I think this description is missing that all the blocks R mines have > the to-be-reorged flag set. > > > 3. How deep should the reorgs be on average? Do you want to test > >deeper reorgs (10+ blocks) too? > > Super interested in input on this -- perhaps we should get optech to > send a survey out to their members, or so? > > My feeling is: > > - 1 block reorgs: these are a regular feature on mainnet, everyone >should cope with them; having them happen multiple times a day to >make testing easier should be great > > - 2-3 block reorgs: good for testing the "your tx didn't get enough >confirms to be credited to your account" case, even though it barely >ever happens on mainnet > > - 4-6 block reorgs: likely to violate business assumptions, but >completely technically plausible, especially if there's an attack >against the network > > - 7-100 block reorgs: for this to happen on mainnet, it would probably >mean there was a bug and pools/miners agree the chain has to >be immediately reverted -- eg, someone discovers and exploits an >inflation bug, minting themselves free bitcoins and breaking the 21M >limit (eg, the 51 block reorg in Aug 2010); or someone discovers a >bug that splits the chain, and the less compatible chain is reverted >(eg, the 24 block reorg due to the bdb lock limit in Mar 2013); >or something similar. Obviously the bug would have to have been >discovered pretty quickly after it was exploited for the reorg to be >under a day's worth of blocks. > > - 100-2000+ block reorgs: severe bug that wasn't found quickly, or where >getting >50% of miners organised took more than a few hours. This will >start breaking protocol assumptions, like pool payouts, lightning's >relative locktimes, or liquid's peg-in confirmation requirements, and >result in hundres of MBs of changes to the utxo set > > Maybe it would be good to do reorgs of 15, 150 or 1500 blocks as a > special fire-drill event, perhaps once a month/quarter/year or so, > in some pre-announced window? > > I think sticking to 1-6 block reorgs initially is a fine way to
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> - 1 block reorgs: these are a regular feature on mainnet, everyone should cope with them; having them happen multiple times a day to make testing easier should be great Anyone can do 1 block reorg, because nonce is not signed, so anyone can replace that with better value. For example, if you have block 0086d6b2636cb2a392d45edc4ec544a10024d30141c9adf4bfd9de533b53 with 0x0007f4cc nonce, you can replace that with 0x00110241 nonce and get 00096a1c4239d994547185c80308a552cba85d5bd28a51e9dc583ae5eadb block, where everything is identical, except the nonce. Sometimes that reorg could be deeper if you would be lucky enough to get a block with more work than N following blocks combined. On 2021-09-08 09:59:29 user Anthony Towns via bitcoin-dev wrote: > On Tue, Sep 07, 2021 at 06:07:47PM +0200, 0xB10C via bitcoin-dev wrote: > The reorg-interval X very much depends on the user's needs. One could > argue that there should be, for example, three reorgs per day, each 48 > blocks apart. Oh, wow, I think the last suggestion was every 100 blocks (every ~16h40m). Once every ~8h sounds very convenient. > Such a short reorg interval allows developers in all time > zones to be awake during one or two reorgs per day. And also for there to reliably be reorgs when they're not awake, which might be a useful thing to be able to handle, too :) > Developers don't > need to wait for, for example, a week until they can test their reorgs > next. However, too frequent reorgs could hinder other SigNet users. Being able to run `bitcoind -signet -signetacceptreorg=0` and never seeing any reorgs should presumably make this not a problem? For people who do see reorgs, having an average of 2 or 3 additional blocks every 48 blocks is perhaps a 6% increase in storage/traffic. > # Scenario 1: Race between two chains > > For this scenario, at least two nodes and miner scripts need to be > running. An always-miner A continuously produces blocks and rejects > blocks with the to-be-reorged version bit flag set. And a race-miner R > that only mines D blocks at the start of each interval and then waits X > blocks. A and R both have the same hash rate. Assuming both are well > connected to the network, it's random which miner will first mine and > propagate a block. In the end, the A miner chain will always win the race. I think this description is missing that all the blocks R mines have the to-be-reorged flag set. > 3. How deep should the reorgs be on average? Do you want to test >deeper reorgs (10+ blocks) too? Super interested in input on this -- perhaps we should get optech to send a survey out to their members, or so? My feeling is: - 1 block reorgs: these are a regular feature on mainnet, everyone should cope with them; having them happen multiple times a day to make testing easier should be great - 2-3 block reorgs: good for testing the "your tx didn't get enough confirms to be credited to your account" case, even though it barely ever happens on mainnet - 4-6 block reorgs: likely to violate business assumptions, but completely technically plausible, especially if there's an attack against the network - 7-100 block reorgs: for this to happen on mainnet, it would probably mean there was a bug and pools/miners agree the chain has to be immediately reverted -- eg, someone discovers and exploits an inflation bug, minting themselves free bitcoins and breaking the 21M limit (eg, the 51 block reorg in Aug 2010); or someone discovers a bug that splits the chain, and the less compatible chain is reverted (eg, the 24 block reorg due to the bdb lock limit in Mar 2013); or something similar. Obviously the bug would have to have been discovered pretty quickly after it was exploited for the reorg to be under a day's worth of blocks. - 100-2000+ block reorgs: severe bug that wasn't found quickly, or where getting >50% of miners organised took more than a few hours. This will start breaking protocol assumptions, like pool payouts, lightning's relative locktimes, or liquid's peg-in confirmation requirements, and result in hundres of MBs of changes to the utxo set Maybe it would be good to do reorgs of 15, 150 or 1500 blocks as a special fire-drill event, perhaps once a month/quarter/year or so, in some pre-announced window? I think sticking to 1-6 block reorgs initially is a fine way to start though. > After enough testing, the default SigNet can start to do periodical > reorgs, too. FWIW, the only thing that concerns me about doing this on the default signet is making sure that nodes that set -signetacceptreorg=0 don't end up partitioning the p2p network due to either rejecting a higher work chain or rejecting txs due to double-spends across the two chains. A quick draft of code for -signetacceptreorg=0 is available at https://github.com/ajtowns/bitcoin/commits/202108-signetreorg Cheers, aj
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On Thu, Sep 09, 2021 at 05:50:08PM -0700, Matt Corallo via bitcoin-dev wrote: > > AJ proposed to allow SigNet users to opt-out of reorgs in case they > > explicitly want to remain unaffected. This can be done by setting a > > to-be-reorged version bit [...] > Why bother with a version bit? This seems substantially more complicated > than the original proposal that surfaced many times before signet launched > to just have a different reorg signing key. Yeah, that was the original idea, but there ended up being two problems with that approach. The simplest is that the signet block signature encodes the signet challenge, so if you have two different challenges, eg " CHECKSIG" "0 SWAP 1 2 CHECKMULTISIG" then while both challenges will accept a signature by normal as the block solution, the signature by "normal" will be different between the two. This is a fairly natural result of reusing the tx-signing code for the block signatures and not having a noinput/anyprevout tx-signing mode. More generally, though, this would mean that a node that's opting out of reorgs will see the to-be-reorged blocks as simply invalid due to a bad signature, and will follow the "this node sent me an invalid block" path in the p2p code, and start marking peers that are following reorgs as discouraged and worth disconnecting. I think that would make it pretty hard to avoid partitioning the network between peers that do and don't accept reorgs, and generally be a pain. So using the RECENT_CONSENSUS_CHANGE behaviour that avoids the discourage/disconnect logic seems the way to avoid that problem, and that means making it so that nodes that that opt-out of reorgs can distinguish valid-but-will-become-stale blocks from invalid blocks. Using a versionbit seems like the easiest way of doing that. > > The reorg-interval X very much depends on the user's needs. One could > > argue that there should be, for example, three reorgs per day, each 48 > > blocks apart. Such a short reorg interval allows developers in all time > > zones to be awake during one or two reorgs per day. Developers don't > > need to wait for, for example, a week until they can test their reorgs > > next. However, too frequent reorgs could hinder other SigNet users. > I see zero reason whatsoever to not simply reorg ~every block, or as often > as is practical. If users opt in to wanting to test with reorgs, they should > be able to test with reorgs, not wait a day to test with reorgs. Blocks on signet get mined at a similar rate to mainnet, so you'll always have to wait a little bit (up to an hour) -- if you don't want to wait at all, that's what regtest (or perhaps a custom signet) is for. I guess it would be super easy to say something like: - miner 1 ignores blocks marked for reorg - miner 2 marks its blocks for reorg, mines on top of the most work block - miner 2 never mines a block which would have (height % 10 == 1) - miner 1 and miner 2 have the same hashrate, but mine at randomly different times which would mean there's almost always a reorg being mined, people that follow reorgs will see fewer than 1.9x as many blocks as non-reorg nodes, and reorgs won't go on for more than 10 blocks. Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> Huh? Why would the goal be to match mainnet? The goal, as I understand it, is > to allow software to use SigNet without modification *to make testing simpler* - keep the header format the same to let SPV clients function without (significant) modification, etc. The point of the whole thing is to make testing as easy as possible, why would we do otherwise. I guess Kalle (and AJ) can answer this question better than me but my understanding is that the motivation for Signet was that testnet deviated erratically from mainnet behavior (e.g. long delays before any blocks were mined followed by a multitude of blocks mined in a short period of time) which meant it wasn't conducive to normal testing of applications. Why would you want a mainnet like chain? To check if your application works on a mainnet like chain without risking any actual value before moving to mainnet. The same purpose as testnet but more reliably resembling mainnet behavior. You are well within your rights to demand more than that but my preference would be to push some of those demands to custom signets rather than the default Signet. Testing out proposed soft forks in advance of them being considered for activation would already be introducing a dimension of complexity that is going to be hard to manage [0]. I'm generally of the view that if you are going to introduce a complexity dimension, keep the other dimensions as vanilla as possible. Otherwise you are battling complexity in multiple different dimensions and it becomes hard or impossible to maintain it and meet your initial objectives. But if this feature of extremely regular re-orgs is an in demand feature for testers I think the question then becomes what the default be (I would suggest re-orgs every 8 hours rather than no re-orgs at all) and then the alternative which you can switch to, re-orgs every block or every 6 blocks or whatever. > I believe my suggestion was not correctly understood. I'm not suggesting > *users* sign blocks or otherwise do anything manually here, only that the existing block producers each generate a new key, and we then only sign reorgs with *those* keys. Users will be able to set a flag to indicate "I want to accept sigs from either sets of keys, and see reorgs" or "I only want sigs from the non-reorg keys, and will consider the reorg keys-signed blocks invalid" Ah I did misunderstand, yes this makes more sense. Thanks for the correction. [0] https://bitcoin.stackexchange.com/questions/98642/can-we-experiment-on-signet-with-multiple-proposed-soft-forks-whilst-maintaining On Fri, Sep 10, 2021 at 7:24 PM Matt Corallo wrote: > > > > On 9/10/21 06:05, Michael Folkson wrote: > >> I see zero reason whatsoever to not simply reorg ~every block, or as often > >> as is practical. If users opt in to wanting to test with reorgs, they > >> should be able to test with reorgs, not wait a day to test with reorgs. > > > > One of the goals of the default Signet was to make the default Signet > > resemble mainnet as much as possible. (You can do whatever you want on > > a custom signet you set up yourself including manufacturing a re-org > > every block if you wish.) Hence I'm a bit wary of making the behavior > > on the default Signet deviate significantly from what you might > > experience on mainnet. Given re-orgs don't occur that often on mainnet > > I can see the argument for making them more regular (every 8 hours > > seems reasonable to me) on the default Signet but every block seems > > excessive. It makes the default Signet into an environment for purely > > testing whether your application can withstand various flavors of edge > > case re-orgs. You may want to test whether your application can > > withstand normal mainnet behavior (no re-orgs for long periods of > > time) first before you concern yourself with re-orgs. > > Huh? Why would the goal be to match mainnet? The goal, as I understand it, is > to allow software to > use SigNet without modification *to make testing simpler* - keep the header > format the same to let > SPV clients function without (significant) modification, etc. The point of > the whole thing is to > make testing as easy as possible, why would we do otherwise. > > Further, because one goal here is to enable clients to opt in or out of the > reorg chain at will > (presumably by just changing one config flag in bitcoin.conf), why would we > worry about making it > "similar to mainnet". If users want an experience "similar to mainnet", they > can simply turn off > reorgs and they'll see a consistent chain moving forward which never reorgs, > similar to the > practical experience of mainnet. > > Once you've opted into reorgs, you almost certainly are looking to *test* > reorgs - you just > restarted Bitcoin Core with the reorg flag set, waiting around for a reorg > after doing that seems > like the experience of testnet3 today, and the whole reason why we wanted > signet to begin with - > things happen sporadically and
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On Fri, Sep 10, 2021 at 11:24:15AM -0700, Matt Corallo via bitcoin-dev wrote: > I'm [...] suggesting [...] that the existing block producers each > generate a new key, and we then only sign reorgs with *those* keys. > Users will be able to set a flag to indicate "I want to accept sigs > from either sets of keys, and see reorgs" or "I only want sigs from > the non-reorg keys, and will consider the reorg keys-signed blocks > invalid" This seems pretty useful to me. I think we might want multiple sets of keys: 0. No reorgs 1. Periodic reorgs of small to moderate depth for ongoing testing without excessive disruption (e.g. the every 8 hours proposal). I think this probably ought to be the default-default `-signet` in Bitcoin Core and other nodes. 2. Either frequent reorgs (e.g. every block) or a webapp that generates reorgs on demand to further reduce testing delays. If we can only have two, I'd suggest dropping 0. I think it's already the case that too few people test their software with reorgs. -Dave signature.asc Description: PGP signature ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
Fwiw, your email client is broken and does not properly quote in the plaintext copy. I believe this is a known gmail bug, but I'd recommend avoiding gmail's web interface for list posting :). On 9/10/21 12:00, Michael Folkson wrote: Huh? Why would the goal be to match mainnet? The goal, as I understand it, is to allow software to use SigNet without modification *to make testing simpler* - keep the header format the same to let SPV clients function without (significant) modification, etc. The point of the whole thing is to make testing as easy as possible, why would we do otherwise. I guess Kalle (and AJ) can answer this question better than me but my understanding is that the motivation for Signet was that testnet deviated erratically from mainnet behavior (e.g. long delays before any blocks were mined followed by a multitude of blocks mined in a short period of time) which meant it wasn't conducive to normal testing of applications. Why would you want a mainnet like chain? To check if your application works on a mainnet like chain without risking any actual value before moving to mainnet. The same purpose as testnet but more reliably resembling mainnet behavior. You are well within your rights to demand more than that but my preference would be to push some of those demands to custom signets rather than the default Signet. Huh? You haven't made an argument here as to why such a chain is easier to test with, only that we should "match mainnet". Testing on mainnet sucks, 99% of the time testing on mainnet involves no reorgs, which *doesn't* match in-the-field reality of mainnet, with occasional reorgs. Matching mainnet's behavior is, in fact, a terrible way to test if your application will run fine on mainnet. My point is that the goal should be making it easier to test. I'm not entirely sure why there's debate here. I *regularly* have lunch late because I'm waiting for blocks either on mainnet or testnet3, and would quite like to avoid that in the future. It takes *forever* to test things on mainnet and testnet3, matching their behavior would mean its equally impossible to test things on mainnet and testnet3, why is that something we should stirve for? Testing out proposed soft forks in advance of them being considered for activation would already be introducing a dimension of complexity that is going to be hard to manage [0]. I'm generally of the view that if you are going to introduce a complexity dimension, keep the other dimensions as vanilla as possible. Otherwise you are battling complexity in multiple different dimensions and it becomes hard or impossible to maintain it and meet your initial objectives. Yep! Great reason to not have any probabilistic nonsense or try to match mainnet or something on signet, just make it deterministic, reorg once a block or twice an our or whatever and call it a day! Matt ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On 9/10/21 06:05, Michael Folkson wrote: I see zero reason whatsoever to not simply reorg ~every block, or as often as is practical. If users opt in to wanting to test with reorgs, they should be able to test with reorgs, not wait a day to test with reorgs. One of the goals of the default Signet was to make the default Signet resemble mainnet as much as possible. (You can do whatever you want on a custom signet you set up yourself including manufacturing a re-org every block if you wish.) Hence I'm a bit wary of making the behavior on the default Signet deviate significantly from what you might experience on mainnet. Given re-orgs don't occur that often on mainnet I can see the argument for making them more regular (every 8 hours seems reasonable to me) on the default Signet but every block seems excessive. It makes the default Signet into an environment for purely testing whether your application can withstand various flavors of edge case re-orgs. You may want to test whether your application can withstand normal mainnet behavior (no re-orgs for long periods of time) first before you concern yourself with re-orgs. Huh? Why would the goal be to match mainnet? The goal, as I understand it, is to allow software to use SigNet without modification *to make testing simpler* - keep the header format the same to let SPV clients function without (significant) modification, etc. The point of the whole thing is to make testing as easy as possible, why would we do otherwise. Further, because one goal here is to enable clients to opt in or out of the reorg chain at will (presumably by just changing one config flag in bitcoin.conf), why would we worry about making it "similar to mainnet". If users want an experience "similar to mainnet", they can simply turn off reorgs and they'll see a consistent chain moving forward which never reorgs, similar to the practical experience of mainnet. Once you've opted into reorgs, you almost certainly are looking to *test* reorgs - you just restarted Bitcoin Core with the reorg flag set, waiting around for a reorg after doing that seems like the experience of testnet3 today, and the whole reason why we wanted signet to begin with - things happen sporadically and inconsistently, making developers wait around forever. Please lets not replicate the "gotta wait for blocks before I can go to lunch" experience of testnet today on signet, I'm tired of eating lunch late. Why bother with a version bit? This seems substantially more complicated than the original proposal that surfaced many times before signet launched to just have a different reorg signing key. Thus, users who wish to follow reorgs can use a 1-of-2 (or higher multisig) and users who wish to not follow reorgs would use a 1-of-1 (or higher multisig), simply marking the reorg blocks as invalid without touching any header bits that non-full clients will ever see. If I understand this correctly this is introducing a need for users to sign blocks when currently with the default Signet the user does not need to concern themselves with signing blocks. That is entirely left to the network block signers of the default Signet (who were AJ and Kalle last time I checked). Again I don't think this additional complexity is needed on the default Signet when you can set up your own custom Signet if you want to test edge case scenarios that deviate significantly from what you are likely to experience on mainnet. A flag set via a configuration argument (the AJ, 0xB10C proposal) with no-reorgs (or 8 hour re-orgs) as the default seems to me like it would introduce no additional complexity to the casual (or alpha stage) tester experience though of course it introduces implementation complexity. To move the default Signet in the direction of resembling mainnet even closer would be to randomly generate batches of transactions to fill up blocks and create a fee market. It would be great to be able to test features like RBF and Lightning unhappy paths (justice transactions, perhaps even pinning attacks etc) on the default Signet in future. I believe my suggestion was not correctly understood. I'm not suggesting *users* sign blocks or otherwise do anything manually here, only that the existing block producers each generate a new key, and we then only sign reorgs with *those* keys. Users will be able to set a flag to indicate "I want to accept sigs from either sets of keys, and see reorgs" or "I only want sigs from the non-reorg keys, and will consider the reorg keys-signed blocks invalid" Matt ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
> I see zero reason whatsoever to not simply reorg ~every block, or as often as > is practical. If users opt in to wanting to test with reorgs, they should be > able to test with reorgs, not wait a day to test with reorgs. One of the goals of the default Signet was to make the default Signet resemble mainnet as much as possible. (You can do whatever you want on a custom signet you set up yourself including manufacturing a re-org every block if you wish.) Hence I'm a bit wary of making the behavior on the default Signet deviate significantly from what you might experience on mainnet. Given re-orgs don't occur that often on mainnet I can see the argument for making them more regular (every 8 hours seems reasonable to me) on the default Signet but every block seems excessive. It makes the default Signet into an environment for purely testing whether your application can withstand various flavors of edge case re-orgs. You may want to test whether your application can withstand normal mainnet behavior (no re-orgs for long periods of time) first before you concern yourself with re-orgs. > Why bother with a version bit? This seems substantially more complicated than > the original proposal that surfaced many times before signet launched to just > have a different reorg signing key. Thus, users who wish to follow reorgs can > use a 1-of-2 (or higher multisig) and users who wish to not follow reorgs > would use a 1-of-1 (or higher multisig), simply marking the reorg blocks as > invalid without touching any header bits that non-full clients will ever see. If I understand this correctly this is introducing a need for users to sign blocks when currently with the default Signet the user does not need to concern themselves with signing blocks. That is entirely left to the network block signers of the default Signet (who were AJ and Kalle last time I checked). Again I don't think this additional complexity is needed on the default Signet when you can set up your own custom Signet if you want to test edge case scenarios that deviate significantly from what you are likely to experience on mainnet. A flag set via a configuration argument (the AJ, 0xB10C proposal) with no-reorgs (or 8 hour re-orgs) as the default seems to me like it would introduce no additional complexity to the casual (or alpha stage) tester experience though of course it introduces implementation complexity. To move the default Signet in the direction of resembling mainnet even closer would be to randomly generate batches of transactions to fill up blocks and create a fee market. It would be great to be able to test features like RBF and Lightning unhappy paths (justice transactions, perhaps even pinning attacks etc) on the default Signet in future. -- Michael Folkson Email: michaelfolk...@gmail.com Keybase: michaelfolkson PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On 9/7/21 09:07, 0xB10C via bitcoin-dev wrote: Hello, tl;dr: We want to make reorgs on SigNet a reality and are looking for feedback on approach and parameters. Awesome! AJ proposed to allow SigNet users to opt-out of reorgs in case they explicitly want to remain unaffected. This can be done by setting a to-be-reorged version bit flag on the blocks that won't end up in the most work chain. Node operators could choose not to accept to-be-reorged SigNet blocks with this flag set via a configuration argument. Why bother with a version bit? This seems substantially more complicated than the original proposal that surfaced many times before signet launched to just have a different reorg signing key. Thus, users who wish to follow reorgs can use a 1-of-2 (or higher multisig) and users who wish to not follow reorgs would use a 1-of-1 (or higher multisig), simply marking the reorg blocks as invalid without touching any header bits that non-full clients will ever see. The reorg-interval X very much depends on the user's needs. One could argue that there should be, for example, three reorgs per day, each 48 blocks apart. Such a short reorg interval allows developers in all time zones to be awake during one or two reorgs per day. Developers don't need to wait for, for example, a week until they can test their reorgs next. However, too frequent reorgs could hinder other SigNet users. I see zero reason whatsoever to not simply reorg ~every block, or as often as is practical. If users opt in to wanting to test with reorgs, they should be able to test with reorgs, not wait a day to test with reorgs. We propose that the reorg depth D is deterministically random between a minimum and a maximum based on, e.g., the block hash or the nonce of the last block before the reorg. Compared to a local randint() based implementation, this allows reorg-handling tests and external tools to calculate the expected reorg depth. # Scenario 1: Race between two chains For this scenario, at least two nodes and miner scripts need to be running. An always-miner A continuously produces blocks and rejects blocks with the to-be-reorged version bit flag set. And a race-miner R that only mines D blocks at the start of each interval and then waits X blocks. A and R both have the same hash rate. Assuming both are well connected to the network, it's random which miner will first mine and propagate a block. In the end, the A miner chain will always win the race. # Scenario 2: Chain rollback This scenario only requires one miner and Bitcoin Core node but also works in a multiminer setup. The miners mine D blocks with the to-be-reorged version bit flag set at the start of the interval. After allowing the block at height X+D to propagate, they invalidate the block at height X+1 and start mining on block X again. This time without setting the to-be-reorged version bit flag. Non-miner nodes will reorg to the new tip at height X+D+1, and the first-seen branch stalls. Both seem reasonable. I'm honestly not sure what software cases would be hit differently between one or the other, as long as reorgs happen regularly and at random depth. Nodes should presumably only ever be following one chain. # Questions 1. How do you currently test your applications reorg handling? Do the two discussed scenarios (race and chain rollback) cover your needs? Are we missing something you'd find helpful? 2. How often should reorgs happen on the default SigNet? Should there be multiple reorgs a day (e.g., every 48 or 72 blocks assuming 144 blocks per day) as your engineers need to be awake? Do you favor less frequent reorgs (once per week or month)? Why? 3. How deep should the reorgs be on average? Do you want to test deeper reorgs (10+ blocks) too? 6 is the "standard" confirmation window for mainnet. Its arguably much too low, but for testing purposes we've gotta pick something, so that seems reasonable? Matt ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
On Tue, Sep 07, 2021 at 06:07:47PM +0200, 0xB10C via bitcoin-dev wrote: > The reorg-interval X very much depends on the user's needs. One could > argue that there should be, for example, three reorgs per day, each 48 > blocks apart. Oh, wow, I think the last suggestion was every 100 blocks (every ~16h40m). Once every ~8h sounds very convenient. > Such a short reorg interval allows developers in all time > zones to be awake during one or two reorgs per day. And also for there to reliably be reorgs when they're not awake, which might be a useful thing to be able to handle, too :) > Developers don't > need to wait for, for example, a week until they can test their reorgs > next. However, too frequent reorgs could hinder other SigNet users. Being able to run `bitcoind -signet -signetacceptreorg=0` and never seeing any reorgs should presumably make this not a problem? For people who do see reorgs, having an average of 2 or 3 additional blocks every 48 blocks is perhaps a 6% increase in storage/traffic. > # Scenario 1: Race between two chains > > For this scenario, at least two nodes and miner scripts need to be > running. An always-miner A continuously produces blocks and rejects > blocks with the to-be-reorged version bit flag set. And a race-miner R > that only mines D blocks at the start of each interval and then waits X > blocks. A and R both have the same hash rate. Assuming both are well > connected to the network, it's random which miner will first mine and > propagate a block. In the end, the A miner chain will always win the race. I think this description is missing that all the blocks R mines have the to-be-reorged flag set. > 3. How deep should the reorgs be on average? Do you want to test >deeper reorgs (10+ blocks) too? Super interested in input on this -- perhaps we should get optech to send a survey out to their members, or so? My feeling is: - 1 block reorgs: these are a regular feature on mainnet, everyone should cope with them; having them happen multiple times a day to make testing easier should be great - 2-3 block reorgs: good for testing the "your tx didn't get enough confirms to be credited to your account" case, even though it barely ever happens on mainnet - 4-6 block reorgs: likely to violate business assumptions, but completely technically plausible, especially if there's an attack against the network - 7-100 block reorgs: for this to happen on mainnet, it would probably mean there was a bug and pools/miners agree the chain has to be immediately reverted -- eg, someone discovers and exploits an inflation bug, minting themselves free bitcoins and breaking the 21M limit (eg, the 51 block reorg in Aug 2010); or someone discovers a bug that splits the chain, and the less compatible chain is reverted (eg, the 24 block reorg due to the bdb lock limit in Mar 2013); or something similar. Obviously the bug would have to have been discovered pretty quickly after it was exploited for the reorg to be under a day's worth of blocks. - 100-2000+ block reorgs: severe bug that wasn't found quickly, or where getting >50% of miners organised took more than a few hours. This will start breaking protocol assumptions, like pool payouts, lightning's relative locktimes, or liquid's peg-in confirmation requirements, and result in hundres of MBs of changes to the utxo set Maybe it would be good to do reorgs of 15, 150 or 1500 blocks as a special fire-drill event, perhaps once a month/quarter/year or so, in some pre-announced window? I think sticking to 1-6 block reorgs initially is a fine way to start though. > After enough testing, the default SigNet can start to do periodical > reorgs, too. FWIW, the only thing that concerns me about doing this on the default signet is making sure that nodes that set -signetacceptreorg=0 don't end up partitioning the p2p network due to either rejecting a higher work chain or rejecting txs due to double-spends across the two chains. A quick draft of code for -signetacceptreorg=0 is available at https://github.com/ajtowns/bitcoin/commits/202108-signetreorg Cheers, aj ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
If you make the to be reorged flag 2 bits, 1 bit can mark final block and the other can mark to be reorged. That way the nodes opting into reorg can see the reorg and ignore the final blocks (until a certain time? Or until it's via a reorg?), and the nodes wanting not to see reorgs get continuous service without disruption On Tue, Sep 7, 2021, 9:12 AM 0xB10C via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Hello, > > tl;dr: We want to make reorgs on SigNet a reality and are looking for > feedback on approach and parameters. > > One of the ideas for SigNet is the possibility for it to be reliably > unreliable, for example, planned chain reorganizations. These have not > been implemented yet. > > My summerofbitcoin.org mentee Nikhil Bartwal and I have been looking at > implementing support for reorgs on SigNet. We are looking for feedback > on which approach and parameters to use. Please consider answering the > questions below if you or your company is interested in chain > reorganizations on SigNet. > > With feedback from AJ and Kalle Alm (thanks again!), we came up with two > scenarios that could be implemented in the current SigNet miner script > [0]. Both would trigger automatically in a fixed block interval. > Scenario 1 simulates a race scenario where two chains compete for D > blocks. Scenario 2 simulates a chain rollback where the top D blocks get > replaced by a chain that outgrows the earlier branch. > > AJ proposed to allow SigNet users to opt-out of reorgs in case they > explicitly want to remain unaffected. This can be done by setting a > to-be-reorged version bit flag on the blocks that won't end up in the > most work chain. Node operators could choose not to accept to-be-reorged > SigNet blocks with this flag set via a configuration argument. > > The reorg-interval X very much depends on the user's needs. One could > argue that there should be, for example, three reorgs per day, each 48 > blocks apart. Such a short reorg interval allows developers in all time > zones to be awake during one or two reorgs per day. Developers don't > need to wait for, for example, a week until they can test their reorgs > next. However, too frequent reorgs could hinder other SigNet users. > > We propose that the reorg depth D is deterministically random between a > minimum and a maximum based on, e.g., the block hash or the nonce of the > last block before the reorg. Compared to a local randint() based > implementation, this allows reorg-handling tests and external tools to > calculate the expected reorg depth. > > # Scenario 1: Race between two chains > > For this scenario, at least two nodes and miner scripts need to be > running. An always-miner A continuously produces blocks and rejects > blocks with the to-be-reorged version bit flag set. And a race-miner R > that only mines D blocks at the start of each interval and then waits X > blocks. A and R both have the same hash rate. Assuming both are well > connected to the network, it's random which miner will first mine and > propagate a block. In the end, the A miner chain will always win the race. > > # Scenario 2: Chain rollback > > This scenario only requires one miner and Bitcoin Core node but also > works in a multiminer setup. The miners mine D blocks with the > to-be-reorged version bit flag set at the start of the interval. After > allowing the block at height X+D to propagate, they invalidate the block > at height X+1 and start mining on block X again. This time without > setting the to-be-reorged version bit flag. Non-miner nodes will reorg > to the new tip at height X+D+1, and the first-seen branch stalls. > > # Questions > > 1. How do you currently test your applications reorg handling? Do >the two discussed scenarios (race and chain rollback) cover your >needs? Are we missing something you'd find helpful? > > 2. How often should reorgs happen on the default SigNet? Should >there be multiple reorgs a day (e.g., every 48 or 72 blocks >assuming 144 blocks per day) as your engineers need to be awake? >Do you favor less frequent reorgs (once per week or month)? Why? > > 3. How deep should the reorgs be on average? Do you want to test >deeper reorgs (10+ blocks) too? > > > # Next Steps > > We will likely implement Scenario 1, the race between two chains, first. > We'll set up a public test-SigNet along with a faucet, block explorer, > and a block tree visualization. If there is interest in the second > approach, chain rollbacks can be implemented too. Future work will add > the possibility to include conflicting transactions in the two branches. > After enough testing, the default SigNet can start to do periodical > reorgs, too. > > Thanks, > 0xB10C > > [0]: https://github.com/bitcoin/bitcoin/blob/master/contrib/signet/miner > > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org >
[bitcoin-dev] Reorgs on SigNet - Looking for feedback on approach and parameters
Hello, tl;dr: We want to make reorgs on SigNet a reality and are looking for feedback on approach and parameters. One of the ideas for SigNet is the possibility for it to be reliably unreliable, for example, planned chain reorganizations. These have not been implemented yet. My summerofbitcoin.org mentee Nikhil Bartwal and I have been looking at implementing support for reorgs on SigNet. We are looking for feedback on which approach and parameters to use. Please consider answering the questions below if you or your company is interested in chain reorganizations on SigNet. With feedback from AJ and Kalle Alm (thanks again!), we came up with two scenarios that could be implemented in the current SigNet miner script [0]. Both would trigger automatically in a fixed block interval. Scenario 1 simulates a race scenario where two chains compete for D blocks. Scenario 2 simulates a chain rollback where the top D blocks get replaced by a chain that outgrows the earlier branch. AJ proposed to allow SigNet users to opt-out of reorgs in case they explicitly want to remain unaffected. This can be done by setting a to-be-reorged version bit flag on the blocks that won't end up in the most work chain. Node operators could choose not to accept to-be-reorged SigNet blocks with this flag set via a configuration argument. The reorg-interval X very much depends on the user's needs. One could argue that there should be, for example, three reorgs per day, each 48 blocks apart. Such a short reorg interval allows developers in all time zones to be awake during one or two reorgs per day. Developers don't need to wait for, for example, a week until they can test their reorgs next. However, too frequent reorgs could hinder other SigNet users. We propose that the reorg depth D is deterministically random between a minimum and a maximum based on, e.g., the block hash or the nonce of the last block before the reorg. Compared to a local randint() based implementation, this allows reorg-handling tests and external tools to calculate the expected reorg depth. # Scenario 1: Race between two chains For this scenario, at least two nodes and miner scripts need to be running. An always-miner A continuously produces blocks and rejects blocks with the to-be-reorged version bit flag set. And a race-miner R that only mines D blocks at the start of each interval and then waits X blocks. A and R both have the same hash rate. Assuming both are well connected to the network, it's random which miner will first mine and propagate a block. In the end, the A miner chain will always win the race. # Scenario 2: Chain rollback This scenario only requires one miner and Bitcoin Core node but also works in a multiminer setup. The miners mine D blocks with the to-be-reorged version bit flag set at the start of the interval. After allowing the block at height X+D to propagate, they invalidate the block at height X+1 and start mining on block X again. This time without setting the to-be-reorged version bit flag. Non-miner nodes will reorg to the new tip at height X+D+1, and the first-seen branch stalls. # Questions 1. How do you currently test your applications reorg handling? Do the two discussed scenarios (race and chain rollback) cover your needs? Are we missing something you'd find helpful? 2. How often should reorgs happen on the default SigNet? Should there be multiple reorgs a day (e.g., every 48 or 72 blocks assuming 144 blocks per day) as your engineers need to be awake? Do you favor less frequent reorgs (once per week or month)? Why? 3. How deep should the reorgs be on average? Do you want to test deeper reorgs (10+ blocks) too? # Next Steps We will likely implement Scenario 1, the race between two chains, first. We'll set up a public test-SigNet along with a faucet, block explorer, and a block tree visualization. If there is interest in the second approach, chain rollbacks can be implemented too. Future work will add the possibility to include conflicting transactions in the two branches. After enough testing, the default SigNet can start to do periodical reorgs, too. Thanks, 0xB10C [0]: https://github.com/bitcoin/bitcoin/blob/master/contrib/signet/miner ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
