Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)
You'd want to create and get merged patches in the following order: 1) Be able to store just block headers in the blk.dat files instead of full block contents. At this point you are still *downloading* full blocks, but they are not being stored. The contents are still sent to the wallet for extracting relevant transactions though (see SyncWithWallets). You also need to disable listening and addr announcements to the P2P network at this point. You need to be able to re-org and do all the usual things without storing block contents. You also need to short-circuit the leveldbs so they aren't created or used. All that needs to be unit tested. You need to also rewrite the mempool logic so it throws out irrelevant transactions. The RPC interface needs to adjust itself so you can't try to start mining, query the utxo set, etc. At this point you have an SPV node, albeit one that still downloads the entire block chain. However total disk storage used will be much lower. Getting this written and reviewed is a big chunk of work but is the hardest part. Once it's done you can breath easy. 2) Next step, use getheaders to catch up with the chain until the min(wallet birthdays) is reached. You can see in Satoshi's patch where he adds support for receiving "headers" messages. Because key times are recorded as dates and you don't know the dates of blocks in advance, you need to download headers until you see one that goes past the key birthday minus some slack period, then throw out the headers you downloaded and switch to downloading full blocks again from that point onwards. 3) Next step, implement client side support for Bloom filtering. Switch from downloading full blocks to filteredblocks, verify the Merkle branches then apply them to the wallet. Watch out for accidental re-orderings of transactions here from block order (e.g. if you accidentally insert them into a std::map or other unordered collection it can lead to bugs). Come up with some way to decide on a FP rate. Probably you want a fairly high FP rate for desktop wallets. 4) Next step (optional), implement monitoring of broadcast propagation for transactions that are received. SPV clients cannot verify unconfirmed transactions so you can either just give up entirely and accept any old garbage, or assume a non-MITMd internet connection and use network propagation as a rough proxy for "likely to be valid and mined upon". 4) Optimize! How much you need to optimize really depends on a lot of things. I found that to be competitive with Electrum/blockchain.info I had to do a ton of optimizations including very aggressive checkpointing so new users don't have to download more than a month or twos worth of headers, as downloading all the headers was becoming a bottleneck. You'd need to download about 16mb+ of data at the moment to grab all the headers and on a weakass mobile phone with a weak Dalvik VM and 3G internet this was way too much. I also had to spend some time profiling to ensure we weren't accidentally thrashing the UI due to too-fast updates, we weren't bottlenecking on updating last seen block data in the wallet, we weren't accidentally de/reserializing messages redundantly etc. After about 3-4 evenings of non-stop profiling and optimising I ended up with a relatively flat profile whilst doing initial catchup and chain sync. On a desktop I bet you can get away with much less optimisation because your CPUs, network and disk tend to be much stronger. On Tue, Jul 16, 2013 at 4:16 PM, Wendell wrote: > Hello everyone, > > In the previous thread, I expressed interest in seeing an SPV bitcoind, > further stating that I would fund such work. Mike Hearn followed up with > some of Satoshi's old code for this, which is now quite broken. The offer > and interest on my side still stand, as more diversity in SPV options seems > like the right way to go. > > Time-permitting, I would really appreciate feedback from knowledgable > parties about the possible approaches to an SPV bitcoind. We at Hive > ideally want to see something that could one be merge into master, rather > than a fork. > > -wendell > > grabhive.com | twitter.com/grabhive > > > > -- > See everything from the browser to the database with AppDynamics > Get end-to-end visibility with application monitoring from AppDynamics > Isolate bottlenecks and diagnose root cause in seconds. > Start your free trial of AppDynamics Pro today! > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk > ___ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isol
[Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)
Hello everyone, In the previous thread, I expressed interest in seeing an SPV bitcoind, further stating that I would fund such work. Mike Hearn followed up with some of Satoshi's old code for this, which is now quite broken. The offer and interest on my side still stand, as more diversity in SPV options seems like the right way to go. Time-permitting, I would really appreciate feedback from knowledgable parties about the possible approaches to an SPV bitcoind. We at Hive ideally want to see something that could one be merge into master, rather than a fork. -wendell grabhive.com | twitter.com/grabhive -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
I for one would be interested in taking a look at that. In San Jose I was asking around about the possibility of hiring someone to complete such a patch. Pieter Wuille introduced me to Eric Lombrozo, who expressed interest, but has since gotten quite busy. So we haven't arrived at a detailed estimate of what it would involve. Maybe it would be better to start a completely new thread for this topic, but I would very much be interested in an open dissection of what adding SPV support to bitcoind would take. I am willing to fund or (ideally) co-fund this endeavor, if I can ever get my head around it. I'm super interested in all of these possibilities (including micro-stripped-VMs and transpilation), but would simply like to encourage the proliferation of _options_ whenever possible. -wendell grabhive.com | twitter.com/grabhive On Jul 16, 2013, at 11:51 AM, Mike Hearn wrote: > If you wanted to implement SPV mode in bitcoind, Gavin or I could send you > Satoshi's old patch although of course it is no longer usable. It would > indicate the basic cut lines though. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Let's re-add the list as this is a topic of general interest. On Tue, Jul 16, 2013 at 11:36 AM, Jonas Schnelli wrote: > What do you think about extending the BitcoinKit.framework (based on > bitcoind) so that the kit could handle the very basic SPV concept > (getHeaders aka fast catchup, wallet-birthday, bloom filters)? > Maybe it would be possible to only port some of the bitcoinj work to c++ > (and use it for extending BitcoinKit or bitcoind)? > Maybe then it could also be a starting point for someone who likes to > create a SPV mode for bitcoind? > Making bitcoind/Bitcoin-Qt support SPV mode was the original plan some years ago, Satoshi even sent me some code he wrote that did the first parts, but it was incomplete. At the time, I decided to do a separate implementation for a few different reasons. One is that my understanding of his code wasn't so good back then and I lacked confidence to change it. Especially as there were no unit tests back then (and still aren't any for most of it), making invasive changes to the core validation code was and is highly risky. A separate code base seemed to reduce the risk a lot. Another reason is that Satoshi encouraged me to write a simple re-implementation that people could learn from. And I wanted a documented, object oriented API that people could use to build a variety of apps. Yet another reason was bitcoind is security critical code that scrapes complex data structures from untrusted sources on the internet, and it's written in an unmanaged language. Ordinarily this would be a recipe for disaster as a single overflow or memory management error could lead to hacking and theft on a massive scale. It's like taking a chainsaw and using it to carve an ice sculpture. Satoshi, incredibly, pulled it off, mostly by using advanced C++ features that made his code hard to read for many people and by being very, very careful. I was not convinced I could do such a good job and was worried about accidentally introducing vulnerabilities. A final reason is that it was clear that the bitcoind codebase would need serious changes for mobiles, beyond that required for ordinary SPV support. For example, Satoshi's code assumes it has access to block headers via a std::map and that assumption is made in a lot of places. On Android phones, you can't fit all the block headers in RAM. bitcoinj uses a circular ring buffer of the last N thousand headers for this reason. It's quite different to how bitcoind works. All that said, it was a ton of work and it's still unclear that it was the right call. Anyway, your situation is a little different. Firstly you don't care about mobiles, your app is intended for desktops. So the changes required are less invasive. Also, there are more unit tests and more people with a good understanding of the code these days, so perhaps the risk of introducing bugs is lower. And these days we have some nice APIs for building apps so that need is already met. If you wanted to implement SPV mode in bitcoind, Gavin or I could send you Satoshi's old patch although of course it is no longer usable. It would indicate the basic cut lines though. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
> Clear. Your right. C++ would give us more flexibility (other platforms) > and also android compatibility (through NDK). > I'm a bit confused I'm afraid. bitcoinj already runs SPV wallets on Android on top of Dalvik. In fact that's what it's designed for. The NDK is not necessary to work with Bitcoin at any point. > That's a great idea. > Let me look into the quality of j2c's output. > There's an example of what it looks like here: https://code.google.com/a/eclipselabs.org/p/j2c/wiki/Examples If you're serious about playing with j2c let me know. It's an amazing piece of work BUT it was written for fun, and as such isn't really documented at all. It took me a little while to figure out how to make it work properly. I'm now fixing bugs in it and making various improvements along with filling out the native stubs (a.k.a. portability layer). If you want to catch up to where I'm at, I can send you some notes because otherwise you might waste a lot of time on blind alleys. The main things be aware of so far are: - Lots of explicit null pointer checks are generated. The reason is that the output is meant to be entirely portable, so Jacek doesn't want to rely on platform specific stuff like signals or SEH. Simplest solution is just to disable npc() generation entirely because normal C++ libraries just segfault if a null pointer gets in the wrong place, they don't throw exceptions. Losing the Java behaviour would not be a downgrade for people used to C++. - Array accesses don't seem to be properly bounds-checked. That's a part of the Java security model - bitcoinj is written on the assumption that buffer and heap overflows aren't possible because they're caught by the runtime. If those checks go missing then it'd likely become possible to hack your program by exploiting buffer overflows. So that needs to be fixed. - Generated code doesn't use the STL of course, it can't because the Java library has more features than the STL. However as the way j2c works is you transpile your code alongside a copy of the (open source) Java class library, you can go in and modify the generated code for java::lang::String or java::util::List and so on to add helper methods for converting to various other forms. On Linux you'd have implicit c'tors to go back and forth between std::string, on MacOS X you'd have conversions for NSString, you could add code for QStrings or raw C strings too. Once the code has been generated you can extend or patch it to make the API more convenient. - Obviously, the resulting code requires the Boehm GC because there are no explicit delete calls anywhere. This is a safety feature though, it avoids use-after-free and double-free bugs that can create security holes. - The code generator doesn't do dependency tracing, so you end up with generated code that isn't used anywhere. It's up to the linker to do a dead code elimination pass. Otherwise the resulting binaries can be huge. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
