Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen runesv...@gmail.comwrote: I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. In a similar circumstance, I changed my -blocknotify script to quickly append necessary information to a queue and immediately exit. A separate script runs at all times monitoring this queue for work and performs the labor intensive calculations. I hope that helps. -- Michael -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks mich...@ndrix.orgwrote: On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen runesv...@gmail.comwrote: I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. In a similar circumstance, I changed my -blocknotify script to quickly append necessary information to a queue and immediately exit. A separate script runs at all times monitoring this queue for work and performs the labor intensive calculations. I've thought about this as well. It just seems somewhat clunky to me. I'd really prefer having bitcoind put out messages in batches, if it's doable, that is. I'd run into a lot of concurrency issues, as far as I can see, where I can't be sure that the queue isn't written to while, for example, it is opened by the program that needs to process the queue items. What if a disk operation takes a long time to finish, and a two queue operations want to add to the queue simultaneously? This really brings forward all the horrors of concurrent programming. On Fri, May 31, 2013 at 2:17 PM, Jeremy Spilman jer...@taplink.co wrote: Would it work to just block the bitcoind thread until your process exits? I don't think that's optimal, no. That would slow down synchronization drastically. It would be really nimble for bitcoind to be able to synchronize at full speed, and only send out events when necessary, batching together previously queued items. -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 7:56 AM, Rune Kjær Svendsen runesv...@gmail.com wrote: Hello dear list I have an application that wants to keep up with new blocks as they come in. For that I can use the -blocknotify option with bitcoind, which will execute my application for each new block. The problem is that my app isn't necessarily quick enough to finish its work before a new block comes in and the app is executed again. This means the that bitcoind might keep executing my application even though the previous instance hasn't finished, and that's fairly inefficient resource-wise, as many instances of the application will be running simultaneously. I've discussed this with wumpus on bitcoin-dev, and we figured out a solution that might be better. It could replace -blocknotify or we could put it in a new function called -batchblocknotify Similar to other suggestions in this thread, If your -blocknotify execution is too slow, then create a solution that simply queues work. There is no need to add this code to bitcoind itself. Another solution is modifying pynode to directly listen to a trusted node (bitcoind), and monitor blocks as they arrive and are announced. That does not fix the problem of slow block processing on your side, but is another way to implement -blocknotify-like behavior. -- Jeff Garzik Senior Software Engineer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 8:37 AM, Rune Kjær Svendsen runesv...@gmail.com wrote: I've thought about this as well. It just seems somewhat clunky to me. I'd really prefer having bitcoind put out messages in batches, if it's doable, that is. I'd run into a lot of concurrency issues, as far as I can see, where I can't be sure that the queue isn't written to while, for example, it is opened by the program that needs to process the queue items. What if a disk operation takes a long time to finish, and a two queue operations want to add to the queue simultaneously? This really brings forward all the horrors of concurrent programming. This is not a compelling need to update bitcoind for this. The vast majority of systems are currently capable of processing a block, before another block arrives. As for parallel processing, your what if has been a solved problem for decade(s) now. -- Jeff Garzik Senior Software Engineer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen runesv...@gmail.com wrote: I'm not quite so how to go about this. As others have said, queuing outside of bitcoind is a better approach. I use zeromq for this situation. blocknotify runs a program which uses zeromq's pub/sub to queue and the application subscribes to this to get notified of the data. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
Chris, Using zmq is a great fit for high-speed notifications such as this. Have you seen the pull request to integrate zmq directly into bitcoind, so that you don't even need -blocknotify? https://github.com/bitcoin/bitcoin/pull/2415 If not: we could use some testing there! Wladimir On Sat, Jun 1, 2013 at 12:20 AM, Chris Double chris.dou...@double.co.nzwrote: On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen runesv...@gmail.com wrote: I'm not quite so how to go about this. As others have said, queuing outside of bitcoind is a better approach. I use zeromq for this situation. blocknotify runs a program which uses zeromq's pub/sub to queue and the application subscribes to this to get notified of the data. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Implementing batch processing for -blocknotify
On Sat, Jun 1, 2013 at 11:29 AM, Wladimir laa...@gmail.com wrote: Using zmq is a great fit for high-speed notifications such as this. Have you seen the pull request to integrate zmq directly into bitcoind, so that you don't even need -blocknotify? https://github.com/bitcoin/bitcoin/pull/2415 If not: we could use some testing there! I hadn't seen this, thanks! I've integrated zmq myself for block notifications in bitcoin and the alt coins I use it on my bitparking merge mining pool. I would love to see something official. I'll try out the patch in the pull request. Chris. -- http://www.bluishcoder.co.nz -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development