[Bitcoin-development] Implementing batch processing for -blocknotify
Hello dear list I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. This means the that bitcoind might keep executing my application even though the previous instance hasn't finished, and that's fairly inefficient resource-wise, as many instances of the application will be running simultaneously. I've discussed this with wumpus on bitcoin-dev, and we figured out a solution that might be better. It could replace -blocknotify or we could put it in a new function called -batchblocknotify The idea is that when bitcoind is executed with the -batchblocknotify option, and it's missing a lot of blocks, upon the first incoming block, the command specified by -batchblocknotify is executed, and if additional blocks come in while this command is still running, we add the block hashes to a list instead of executing the command again. When the previous command finishes, we execute it again and pass two parameters to it: 1. the first block hash in the list of queued blocks, and 2. the number of blocks that have come in while the last command was executing. This prevents bitcoind from fork bombing the system, and allows the command to handle incoming blocks in batches. Would this make sense as an approach? I've been looking at the code and I'm not sure how to implement it. As far as I can see, I need to pass an object - whose state is retained between calls - to the thread function (runCommand) that runs the command, which contains a variable that keeps track of whether a previously executed command is still running, and that contains a list of block hashes that haven't been processed. And I'm not sure how to do this. The runCommand thread is started in SetBestChain() in main.cpp. SetBestChain() is executed by ConnectBestBlock() in main.cpp. ConnectBestBlock() is executed by CBlock::AddToBlockIndex() in main.cpp. CBlock::AddToBlockIndex() is executed by CBlock::AcceptBlock() in main.cpp. CBlock::AcceptBlock() is executed by ProcessBlock() in main.cpp. ProcessBlock() is executed by ProcessMessage() in main.cpp. And so on, and so forth. What's the right way to create an object that can be passed to the runCommand thread, whose state is retained, so it can hold information about whether the -batchblocknotify command is still executing, and contain a list of blocks that are waiting to be passed to the -batchblocknotify command? I assume I shouldn't add a new parameter to the ProcessMessage() function, which passes it to ProcessBlock(), which passes it to AcceptBlock() which passes it to AddToBlockIndex()... and so on. Would it be appropriate to store this object inside the CValidationState class that is passed to SetBestChain()? I'm not quite so how to go about this. /Rune -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen runesv...@gmail.comwrote: I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. In a similar circumstance, I changed my -blocknotify script to quickly append necessary information to a queue and immediately exit. A separate script runs at all times monitoring this queue for work and performs the labor intensive calculations. I hope that helps. -- Michael -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks mich...@ndrix.orgwrote: On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen runesv...@gmail.comwrote: I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. In a similar circumstance, I changed my -blocknotify script to quickly append necessary information to a queue and immediately exit. A separate script runs at all times monitoring this queue for work and performs the labor intensive calculations. I've thought about this as well. It just seems somewhat clunky to me. I'd really prefer having bitcoind put out messages in batches, if it's doable, that is. I'd run into a lot of concurrency issues, as far as I can see, where I can't be sure that the queue isn't written to while, for example, it is opened by the program that needs to process the queue items. What if a disk operation takes a long time to finish, and a two queue operations want to add to the queue simultaneously? This really brings forward all the horrors of concurrent programming. On Fri, May 31, 2013 at 2:17 PM, Jeremy Spilman jer...@taplink.co wrote: Would it work to just block the bitcoind thread until your process exits? I don't think that's optimal, no. That would slow down synchronization drastically. It would be really nimble for bitcoind to be able to synchronize at full speed, and only send out events when necessary, batching together previously queued items. -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 7:56 AM, Rune Kjær Svendsen runesv...@gmail.com wrote: Hello dear list I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. This means the that bitcoind might keep executing my application even though the previous instance hasn't finished, and that's fairly inefficient resource-wise, as many instances of the application will be running simultaneously. I've discussed this with wumpus on bitcoin-dev, and we figured out a solution that might be better. It could replace -blocknotify or we could put it in a new function called -batchblocknotify Similar to other suggestions in this thread, If your -blocknotify execution is too slow, then create a solution that simply queues work. There is no need to add this code to bitcoind itself. Another solution is modifying pynode to directly listen to a trusted node (bitcoind), and monitor blocks as they arrive and are announced. That does not fix the problem of slow block processing on your side, but is another way to implement -blocknotify-like behavior. -- Jeff Garzik Senior Software Engineer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Decentralizing mining
I just posted the following to bitcointalk. https://bitcointalk.org/index.php?topic=221164.0 Right now between two to four running the largest pools control Bitcoin in the short term. That's a lot of hashing power in the hands of very, very few people. In addition pools have little incentive to run secure operations, and many pools have been hacked with their funds stolen. Those hacks could just have easily been used to attack the network itself. This needs to change. Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were discussing at the conference two weeks ago. (credit goes to Greg and Luke; I was mostly just listening) Basically the idea is that miners with mining equipment run a local Bitcoin node and use that node to construct the blocks they mine - the same as if they were solo mining. The pools job is then to only track shares and organize payouts. If the pool gets hacked the worst that can happen is miners are ripped off, rather than Bitcoin itself being attacked. With pooled-solo mining even a pool with a majority of hashing power wouldn't be able to do much harm to Bitcoin. (depending on the implementation they may be able to blacklist specific transactions - the pool needs to know what transactions are in the share to credit fees properly) Tech-wise Luke created getblocktemplate last year as a means to audit mining pools. I'm sure Greg and Luke can explain the nitty gritty details better than I can, but essentially the plan is to take getblocktemplate and extend it as required for pooled-solo mining. This will include pool and miner software initially, and later improvements to GUIs and what not to make the whole process easier. With the success of my recent video project I also want to make this Keep Bitcoin Free's next project, specifically funding a developer (likely Luke) to make this happen. Additionally once software is written and easily usable a good follow-up would be a video and other media to promote the idea to miners. No guarantees we'll be able to come up with commercially competitive remuneration, but we can at least come up with a Thank you tip. But first lets discuss the technical requirements to get an idea of what the scope is. Finally, for the record, a big part of the reason why myself and other Keep Bitcoin Free supporters are interested in doing this is very much to take power over the direction of the network from big pools and put it into the hands of thousands of individual miners. It's much easier to convince people that changes to Bitcoin, like increasing the blocksize, are directly impacting decentralization when individual miners are seeing that happen to themselves. -- 'peter'[:-1]@petertodd.org 00c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b signature.asc Description: Digital signature -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Decentralizing mining
I like this idea a lot. To add: I think it is a bug and security risk if pooled-solo or (current pooled miners) do not add randomness to their extraNonce2 (like 128-bits of it). For privacy and to avoid various hostile-pre-mining attacks it should be done this way. Lack of the self-chosen challenge field is the reason Satoshi's first year mining is marked (plus forgetting to reset the counter). (Bitcoind I believe considered the direct miners key as defense enough as a stand in for self-chosen challenge, which has a few problems). The base counter I think is only 32-bits, the extranonce2 itself being random can be incremented while still looking random. But incrementing extranonce directy while initializing it to 0 is not good (per previous mining extranone marked coins bug - is that even fixed?) (You dont want to reveal the miners power in his pool shares, if the full counter is revealed with no randomness it also reveals how many iterations he can do since the block start). Adam On Fri, May 31, 2013 at 12:57:58PM -0400, Peter Todd wrote: I just posted the following to bitcointalk. https://bitcointalk.org/index.php?topic=221164.0 Right now between two to four running the largest pools control Bitcoin in the short term. That's a lot of hashing power in the hands of very, very few people. In addition pools have little incentive to run secure operations, and many pools have been hacked with their funds stolen. Those hacks could just have easily been used to attack the network itself. This needs to change. Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were discussing at the conference two weeks ago. (credit goes to Greg and Luke; I was mostly just listening) Basically the idea is that miners with mining equipment run a local Bitcoin node and use that node to construct the blocks they mine - the same as if they were solo mining. The pools job is then to only track shares and organize payouts. If the pool gets hacked the worst that can happen is miners are ripped off, rather than Bitcoin itself being attacked. With pooled-solo mining even a pool with a majority of hashing power wouldn't be able to do much harm to Bitcoin. (depending on the implementation they may be able to blacklist specific transactions - the pool needs to know what transactions are in the share to credit fees properly) Tech-wise Luke created getblocktemplate last year as a means to audit mining pools. I'm sure Greg and Luke can explain the nitty gritty details better than I can, but essentially the plan is to take getblocktemplate and extend it as required for pooled-solo mining. This will include pool and miner software initially, and later improvements to GUIs and what not to make the whole process easier. With the success of my recent video project I also want to make this Keep Bitcoin Free's next project, specifically funding a developer (likely Luke) to make this happen. Additionally once software is written and easily usable a good follow-up would be a video and other media to promote the idea to miners. No guarantees we'll be able to come up with commercially competitive remuneration, but we can at least come up with a Thank you tip. But first lets discuss the technical requirements to get an idea of what the scope is. Finally, for the record, a big part of the reason why myself and other Keep Bitcoin Free supporters are interested in doing this is very much to take power over the direction of the network from big pools and put it into the hands of thousands of individual miners. It's much easier to convince people that changes to Bitcoin, like increasing the blocksize, are directly impacting decentralization when individual miners are seeing that happen to themselves. -- 'peter'[:-1]@petertodd.org 00c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 8:37 AM, Rune Kjær Svendsen runesv...@gmail.com wrote: I've thought about this as well. It just seems somewhat clunky to me. I'd really prefer having bitcoind put out messages in batches, if it's doable, that is. I'd run into a lot of concurrency issues, as far as I can see, where I can't be sure that the queue isn't written to while, for example, it is opened by the program that needs to process the queue items. What if a disk operation takes a long time to finish, and a two queue operations want to add to the queue simultaneously? This really brings forward all the horrors of concurrent programming. This is not a compelling need to update bitcoind for this. The vast majority of systems are currently capable of processing a block, before another block arrives. As for parallel processing, your what if has been a solved problem for decade(s) now. -- Jeff Garzik Senior Software Engineer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen runesv...@gmail.com wrote: I'm not quite so how to go about this. As others have said, queuing outside of bitcoind is a better approach. I use zeromq for this situation. blocknotify runs a program which uses zeromq's pub/sub to queue and the application subscribes to this to get notified of the data. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
Chris, Using zmq is a great fit for high-speed notifications such as this. Have you seen the pull request to integrate zmq directly into bitcoind, so that you don't even need -blocknotify? https://github.com/bitcoin/bitcoin/pull/2415 If not: we could use some testing there! Wladimir On Sat, Jun 1, 2013 at 12:20 AM, Chris Double chris.dou...@double.co.nzwrote: On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen runesv...@gmail.com wrote: I'm not quite so how to go about this. As others have said, queuing outside of bitcoind is a better approach. I use zeromq for this situation. blocknotify runs a program which uses zeromq's pub/sub to queue and the application subscribes to this to get notified of the data. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Sat, Jun 1, 2013 at 11:29 AM, Wladimir laa...@gmail.com wrote: Using zmq is a great fit for high-speed notifications such as this. Have you seen the pull request to integrate zmq directly into bitcoind, so that you don't even need -blocknotify? https://github.com/bitcoin/bitcoin/pull/2415 If not: we could use some testing there! I hadn't seen this, thanks! I've integrated zmq myself for block notifications in bitcoin and the alt coins I use it on my bitparking merge mining pool. I would love to see something official. I'll try out the patch in the pull request. Chris. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
