[Bitcoin-development] Timed testing
I'm implementing a new testing mode that produces blocks periodically. You can get what I have so far here: https://github.com/jtimon/bitcoin/tree/timed It depends on pull request #3824 with some improvements on CChainParams, but after that the changes required to add this new mode are very small: https://github.com/jtimon/bitcoin/commit/445321928a143cb9a6f56777cbb7479dd32c3bcd I guess I need a new genesis block, different magic numbers, etc. So this is definitely not ready. You can run it like this: bitcoind -timedtest -gen=1 blocktime=2000 blocktime defaults to 1000 milliseconds for timedtest mode and 0 for the rest of the modes. What could this testing mode be useful for? Basically, simulations. For example, you could run several nodes implementing different mining policies. Let's say I want to mine 50% of the blocks with standard policy, 25% with policy A and 25% with policy B. I can run 1 one for each of one with block times 2000, 1000 and 1000 respectively. Maybe I want to detect performance bottlenecks by stressing this mode with as many transaction as can be processed, maybe removing the block size limits in the simulations. But this still doesn't serve for hardfork or double-spend attacks simulations without calculating any pow, which would be another interesting feature for a new testing mode. I would like to implement the new mode following as the concept of private chains described in freimarkets: http://freico.in/docs/freimarkets.pdf https://github.com/jtimon/freimarkets/blob/master/doc/freimarkets_specs.org#private-ledgers https://github.com/jtimon/freimarkets/blob/master/doc/freimarkets_specs.org#off-chain-transactions I know this could be considered a non-bitcoinish application and therefore be controversial as discussed in PR 3824, so I want to keep the conversation focused on testing use cases useful to bitcoin itself only: additional changes can be implemented elsewhere. One way I think you could support chain races simulations by using a private mode could be the following: 1) The private mode works like the timed mode in how often it produces blocks. 2) In private mode you replace the pow-related fields with a blockPubkeyScript and a lastBlockSigScript fields. In the genesis block, lastBlockSigScript is empty and the initial blockPubkeyScript can be an optional parameter like blocktime. You can set any valid script, probably p2sh, maybe with multisig to allow different nodes to sign. 3) In this context, longer chains mean more work. Another way to see it is that all blocks just contain work==1 in them. So let's say we want to simulate an attack using 50% standard and 50% attacker blocks. You set up the private mode script to be a 1 of 2 multisig and make each node sign always with the same private key (maybe an additional parameter). You make the attacker reject any blocks from height X that he hasn't signed himself to get the result you wanted: the standard node will produce blocks on top of the longest chain while the attacker will only hash on top of his own blocks. So my question to the community is, how invasive is this to bitcoin's source code? In my opinion, done properly could actually result (apart from getting the new features) in more readable code, not less, since you will probably need to make sure proof of work functionality is properly encapsulated during the implementation process (see PR 3839 for a first step on that direction). But, should I push a private mode to the core or just the timed one and implement the private mode elsewhere? Of course other comments on the parameters, defaults or any other design or implementation details are also welcomed. -- Jorge Timón http://freico.in/ -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Timed testing
So my question to the community is, how invasive is this to bitcoin's source code? I'd say not very considering you have regression testing mode. On Thu, Apr 17, 2014 at 8:25 AM, Jorge Timón jti...@monetize.io wrote: I'm implementing a new testing mode that produces blocks periodically. You can get what I have so far here: https://github.com/jtimon/bitcoin/tree/timed It depends on pull request #3824 with some improvements on CChainParams, but after that the changes required to add this new mode are very small: https://github.com/jtimon/bitcoin/commit/445321928a143cb9a6f56777cbb7479dd32c3bcd I guess I need a new genesis block, different magic numbers, etc. So this is definitely not ready. You can run it like this: bitcoind -timedtest -gen=1 blocktime=2000 blocktime defaults to 1000 milliseconds for timedtest mode and 0 for the rest of the modes. What could this testing mode be useful for? Basically, simulations. For example, you could run several nodes implementing different mining policies. Let's say I want to mine 50% of the blocks with standard policy, 25% with policy A and 25% with policy B. I can run 1 one for each of one with block times 2000, 1000 and 1000 respectively. Maybe I want to detect performance bottlenecks by stressing this mode with as many transaction as can be processed, maybe removing the block size limits in the simulations. But this still doesn't serve for hardfork or double-spend attacks simulations without calculating any pow, which would be another interesting feature for a new testing mode. I would like to implement the new mode following as the concept of private chains described in freimarkets: http://freico.in/docs/freimarkets.pdf https://github.com/jtimon/freimarkets/blob/master/doc/freimarkets_specs.org#private-ledgers https://github.com/jtimon/freimarkets/blob/master/doc/freimarkets_specs.org#off-chain-transactions I know this could be considered a non-bitcoinish application and therefore be controversial as discussed in PR 3824, so I want to keep the conversation focused on testing use cases useful to bitcoin itself only: additional changes can be implemented elsewhere. One way I think you could support chain races simulations by using a private mode could be the following: 1) The private mode works like the timed mode in how often it produces blocks. 2) In private mode you replace the pow-related fields with a blockPubkeyScript and a lastBlockSigScript fields. In the genesis block, lastBlockSigScript is empty and the initial blockPubkeyScript can be an optional parameter like blocktime. You can set any valid script, probably p2sh, maybe with multisig to allow different nodes to sign. 3) In this context, longer chains mean more work. Another way to see it is that all blocks just contain work==1 in them. So let's say we want to simulate an attack using 50% standard and 50% attacker blocks. You set up the private mode script to be a 1 of 2 multisig and make each node sign always with the same private key (maybe an additional parameter). You make the attacker reject any blocks from height X that he hasn't signed himself to get the result you wanted: the standard node will produce blocks on top of the longest chain while the attacker will only hash on top of his own blocks. So my question to the community is, how invasive is this to bitcoin's source code? In my opinion, done properly could actually result (apart from getting the new features) in more readable code, not less, since you will probably need to make sure proof of work functionality is properly encapsulated during the implementation process (see PR 3839 for a first step on that direction). But, should I push a private mode to the core or just the timed one and implement the private mode elsewhere? Of course other comments on the parameters, defaults or any other design or implementation details are also welcomed. -- Jorge Timón http://freico.in/ -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech___
Re: [Bitcoin-development] Timed testing
2) If I wanted to measure validation performance, to get the number of peak tps that could be processed without taking block sides or network latency into account, how would I do that? Has anybody tried this before? You can just reindex/replay the chain. It's been done many times. -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Timed testing
On 4/17/14, Mike Hearn m...@plan99.net wrote: 2) If I wanted to measure validation performance, to get the number of peak tps that could be processed without taking block sides or network latency into account, how would I do that? Has anybody tried this before? You can just reindex/replay the chain. It's been done many times. Yes, thank you. I guess that's what everybody is doing to measure validation performance. So I guess the timedtest mode doesn't make much sense, at most only as the blocktime parameter defaulting to zero. If bool MineBlocksOnDemand() gets refactored out of ChainParams into a parameter (maybe just use genproclimit ?), you can have the periodic block generation and the generation on demand reusing the same regtest mode. So it seems a new mode only makes sense if the -private mode makes sense, which in turn only makes sense to include in bitcoind if it's useful enough for the network attack simulations, which remains the open question. -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Timed testing
Not necessarily. Running a private server involves listening to the p2p network for incoming transactions, performing validation on receipt and organizing a mempool, performing transaction selection, and relaying blocks to auditors - none of which is tested in a reindex. A reindex would give you an optimistic upper bound though, if that's all you care about. On 04/17/2014 08:49 AM, Mike Hearn wrote: 2) If I wanted to measure validation performance, to get the number of peak tps that could be processed without taking block sides or network latency into account, how would I do that? Has anybody tried this before? You can just reindex/replay the chain. It's been done many times. -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Timed testing
On Thu, Apr 17, 2014 at 12:09 PM, Jorge Timón jti...@monetize.io wrote: So it seems a new mode only makes sense if the -private mode makes sense, which in turn only makes sense to include in bitcoind if it's useful enough for the network attack simulations, which remains the open question. Unless I misunderstood what your private mode does, you can get the same effect with -regtest by just controlling nodes connectivity. For example: Start 2 nodes, connected to each other. Mine a -regtest chain they both agree on. Restart them so they're not connected. Have one mine normally, have the other mine... however you like to simulate some attack (deep chain re-org, double-spend, whatever). To simulate launching the attack, connect them together again, let the two chains compete and see what happens. -- -- Gavin Andresen -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Timed testing
Thank you for all the explanations on how to use regtest to reproduce the example scenarios. It seems like a private mode wouldn't be particularly helpful for testing so I won't create a pull request and will just work on the private chains separately from bitcoind. Going back to chainparam modes in general, I've heard Sipa complain some times about regtest being too specific, arguing that some of the behaviors should be specified as independent parameters instead of chainparams attributes. One example could be bool CChainPrams::MineBlocksOnDemand() (see https://github.com/jtimon/bitcoin/commit/5bd4bc7f3694e46568c9276f0cb26402dfb99718 ). If that was an independent parameter that people set to true when using regtest, the blocktime param I was proposing for -timedtest could also be implemented as an independent parameter without any need for a new mode. It's not clear to me if the timedtest parameter would be useful for bitcoind testing or not, but it's just something I've noticed while playing with this part of the code. Sipa, is this the kind of thing you refer to when you say regtest is too specific? Do you have any suggestion on how to solve the issue as part of PR 3824? Well, any suggestions from anyone on how to improve PR 3824 are welcomed, I was just asking specifically to sipa because as said it is my understanding that he had some complaints about regtest and I think this is something simple enough for me to start contributing to bitcoind. Specially given that I was going to review all that part of the code to externally write the private mode anyway. -- Jorge Timón http://freico.in/ -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
