Re: [Bitcoin-development] Introducing BitcoinKit.framework
As an FYI, I've sent Wendell and co some example code for how to use CPPJVM to use bitcoinj from native code. A rather rough Hello World app looks like this: https://github.com/mikehearn/cppjvm/blob/master/mytest/bcj-hello-world.cpp So, fairly C++ like. Further discussion of this should take place on the bitcoinj mailing list. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
On Tue, Jul 16, 2013 at 12:59:56PM +0200, Mike Hearn wrote: I think that's a great approach. Here is the patch Satoshi sent me back in 2010. All the code has changed since but it can be a source of inspiration. From Satoshi: *The simplified payment verification in the paper imagined you would receive transactions directly, as with sending to IP address which nobody uses, or a node would index all transactions by public key and you could download them like downloading mail from a mail server. I'm currently working on headers-first sync, which I believe is generally very useful (it fixes tons of edge-cases block synchronization currently experiences), but it's also a first step towards SPV mode. So headers-first sync means you first synchronize just the headers, and then, when you already know (or have strong evidence for a guess on) the best chain, start requesting blocks along that best chain - potentially in parallel from different peers. SPV mode is basically headers-first sync, but never do the full block sync step, and replace it with a bloom/birthday/...-based fetching of blocks interesting to the associated wallets. In SPV you'll also need to disable the mempool though, and there will be more small changes, but I think the separate headers-sync phase will be most of the work. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Actually bitcoinj typically doesn't download all the headers (just from the last checkpoint) and it throws away headers that are very old. By now there's quite a lot of difference in how they manage things and I guess it will diverge from bitcoind even more in future. For instance we're going to start only storing relevant outputs in the wallet and doing other things to try and save memory. Some people managed to get themselves wallets that don't actually fit in ram :( On 21 Jul 2013 17:55, Pieter Wuille pieter.wui...@gmail.com wrote: On Tue, Jul 16, 2013 at 12:59:56PM +0200, Mike Hearn wrote: I think that's a great approach. Here is the patch Satoshi sent me back in 2010. All the code has changed since but it can be a source of inspiration. From Satoshi: *The simplified payment verification in the paper imagined you would receive transactions directly, as with sending to IP address which nobody uses, or a node would index all transactions by public key and you could download them like downloading mail from a mail server. I'm currently working on headers-first sync, which I believe is generally very useful (it fixes tons of edge-cases block synchronization currently experiences), but it's also a first step towards SPV mode. So headers-first sync means you first synchronize just the headers, and then, when you already know (or have strong evidence for a guess on) the best chain, start requesting blocks along that best chain - potentially in parallel from different peers. SPV mode is basically headers-first sync, but never do the full block sync step, and replace it with a bloom/birthday/...-based fetching of blocks interesting to the associated wallets. In SPV you'll also need to disable the mempool though, and there will be more small changes, but I think the separate headers-sync phase will be most of the work. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Clear. Your right. C++ would give us more flexibility (other platforms) and also android compatibility (through NDK). I'm a bit confused I'm afraid. bitcoinj already runs SPV wallets on Android on top of Dalvik. In fact that's what it's designed for. The NDK is not necessary to work with Bitcoin at any point. That's a great idea. Let me look into the quality of j2c's output. There's an example of what it looks like here: https://code.google.com/a/eclipselabs.org/p/j2c/wiki/Examples If you're serious about playing with j2c let me know. It's an amazing piece of work BUT it was written for fun, and as such isn't really documented at all. It took me a little while to figure out how to make it work properly. I'm now fixing bugs in it and making various improvements along with filling out the native stubs (a.k.a. portability layer). If you want to catch up to where I'm at, I can send you some notes because otherwise you might waste a lot of time on blind alleys. The main things be aware of so far are: - Lots of explicit null pointer checks are generated. The reason is that the output is meant to be entirely portable, so Jacek doesn't want to rely on platform specific stuff like signals or SEH. Simplest solution is just to disable npc() generation entirely because normal C++ libraries just segfault if a null pointer gets in the wrong place, they don't throw exceptions. Losing the Java behaviour would not be a downgrade for people used to C++. - Array accesses don't seem to be properly bounds-checked. That's a part of the Java security model - bitcoinj is written on the assumption that buffer and heap overflows aren't possible because they're caught by the runtime. If those checks go missing then it'd likely become possible to hack your program by exploiting buffer overflows. So that needs to be fixed. - Generated code doesn't use the STL of course, it can't because the Java library has more features than the STL. However as the way j2c works is you transpile your code alongside a copy of the (open source) Java class library, you can go in and modify the generated code for java::lang::String or java::util::List and so on to add helper methods for converting to various other forms. On Linux you'd have implicit c'tors to go back and forth between std::string, on MacOS X you'd have conversions for NSString, you could add code for QStrings or raw C strings too. Once the code has been generated you can extend or patch it to make the API more convenient. - Obviously, the resulting code requires the Boehm GC because there are no explicit delete calls anywhere. This is a safety feature though, it avoids use-after-free and double-free bugs that can create security holes. - The code generator doesn't do dependency tracing, so you end up with generated code that isn't used anywhere. It's up to the linker to do a dead code elimination pass. Otherwise the resulting binaries can be huge. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Let's re-add the list as this is a topic of general interest. On Tue, Jul 16, 2013 at 11:36 AM, Jonas Schnelli jonas.schne...@include7.ch wrote: What do you think about extending the BitcoinKit.framework (based on bitcoind) so that the kit could handle the very basic SPV concept (getHeaders aka fast catchup, wallet-birthday, bloom filters)? Maybe it would be possible to only port some of the bitcoinj work to c++ (and use it for extending BitcoinKit or bitcoind)? Maybe then it could also be a starting point for someone who likes to create a SPV mode for bitcoind? Making bitcoind/Bitcoin-Qt support SPV mode was the original plan some years ago, Satoshi even sent me some code he wrote that did the first parts, but it was incomplete. At the time, I decided to do a separate implementation for a few different reasons. One is that my understanding of his code wasn't so good back then and I lacked confidence to change it. Especially as there were no unit tests back then (and still aren't any for most of it), making invasive changes to the core validation code was and is highly risky. A separate code base seemed to reduce the risk a lot. Another reason is that Satoshi encouraged me to write a simple re-implementation that people could learn from. And I wanted a documented, object oriented API that people could use to build a variety of apps. Yet another reason was bitcoind is security critical code that scrapes complex data structures from untrusted sources on the internet, and it's written in an unmanaged language. Ordinarily this would be a recipe for disaster as a single overflow or memory management error could lead to hacking and theft on a massive scale. It's like taking a chainsaw and using it to carve an ice sculpture. Satoshi, incredibly, pulled it off, mostly by using advanced C++ features that made his code hard to read for many people and by being very, very careful. I was not convinced I could do such a good job and was worried about accidentally introducing vulnerabilities. A final reason is that it was clear that the bitcoind codebase would need serious changes for mobiles, beyond that required for ordinary SPV support. For example, Satoshi's code assumes it has access to block headers via a std::map and that assumption is made in a lot of places. On Android phones, you can't fit all the block headers in RAM. bitcoinj uses a circular ring buffer of the last N thousand headers for this reason. It's quite different to how bitcoind works. All that said, it was a ton of work and it's still unclear that it was the right call. Anyway, your situation is a little different. Firstly you don't care about mobiles, your app is intended for desktops. So the changes required are less invasive. Also, there are more unit tests and more people with a good understanding of the code these days, so perhaps the risk of introducing bugs is lower. And these days we have some nice APIs for building apps so that need is already met. If you wanted to implement SPV mode in bitcoind, Gavin or I could send you Satoshi's old patch although of course it is no longer usable. It would indicate the basic cut lines though. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
I for one would be interested in taking a look at that. In San Jose I was asking around about the possibility of hiring someone to complete such a patch. Pieter Wuille introduced me to Eric Lombrozo, who expressed interest, but has since gotten quite busy. So we haven't arrived at a detailed estimate of what it would involve. Maybe it would be better to start a completely new thread for this topic, but I would very much be interested in an open dissection of what adding SPV support to bitcoind would take. I am willing to fund or (ideally) co-fund this endeavor, if I can ever get my head around it. I'm super interested in all of these possibilities (including micro-stripped-VMs and transpilation), but would simply like to encourage the proliferation of _options_ whenever possible. -wendell grabhive.com | twitter.com/grabhive On Jul 16, 2013, at 11:51 AM, Mike Hearn wrote: If you wanted to implement SPV mode in bitcoind, Gavin or I could send you Satoshi's old patch although of course it is no longer usable. It would indicate the basic cut lines though. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Introducing BitcoinKit.framework
Hi devs, Just wanted to cross-post this here since it seems very relevant. We're launching BitcoinKit.framework, a Cocoa framework that allows developers to write Bitcoin wallet apps for Mac OS X. BitcoinKit uses bitcoind, and serves a small and tidy API for developer use. Support for other Bitcoin implementations (libcbitcoin, etc) is soon to follow. BitcoinKit's first application is as the backbone of a new Mac wallet app called Hive, which will be released soon at www.grabhive.com. Grab the source here: https://github.com/grabhive/BitcoinKit Support is available via GitHub issues and this Bitcointalk thread: https://bitcointalk.org/index.php?topic=256583.msg2733523 A sample GUI app is also included: http://imgur.com/FzqA00X Cheers everyone! -Wendell signature.asc Description: Message signed with OpenPGP using GPGMail -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
That's great! I'm all for more wallets, especially user friendly UIs. However being based on bitcoind means it will take a very long time to synchronize for new users. We know a lot of users drop out. The best fix for this is SPV mode. Do you have any plans in this direction? So far, the only SPV mode implementation I know about is bitcoinj. I am experimenting with trans-piling bitcoinj to C++ to make it usable from Objective-C++ exactly with your use case in mind. On Mon, Jul 15, 2013 at 12:07 PM, Wendell w...@grabhive.com wrote: Hi devs, Just wanted to cross-post this here since it seems very relevant. We're launching BitcoinKit.framework, a Cocoa framework that allows developers to write Bitcoin wallet apps for Mac OS X. BitcoinKit uses bitcoind, and serves a small and tidy API for developer use. Support for other Bitcoin implementations (libcbitcoin, etc) is soon to follow. BitcoinKit's first application is as the backbone of a new Mac wallet app called Hive, which will be released soon at www.grabhive.com. Grab the source here: https://github.com/grabhive/BitcoinKit Support is available via GitHub issues and this Bitcointalk thread: https://bitcointalk.org/index.php?topic=256583.msg2733523 A sample GUI app is also included: http://imgur.com/FzqA00X Cheers everyone! -Wendell -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Hi Mike, You are absolutely right about the synchronize time, it's one of our main frustration points right now and we clearly won't deliver the kind of user experience we want, without fixing this. Actually we were thinking of extending Jeff Garzik's picocoin as time permits, but the plan is far from concrete at the moment. What you say about trans-piling bitcoinj is _really_ appealing. We discounted Java simply because an OS X JVM is no longer guaranteed, but otherwise bitcoinj is ideal for our purposes. How can we assist or otherwise accelerate such an effort? -w On Jul 15, 2013, at 3:19 PM, Mike Hearn wrote: That's great! I'm all for more wallets, especially user friendly UIs. However being based on bitcoind means it will take a very long time to synchronize for new users. We know a lot of users drop out. The best fix for this is SPV mode. Do you have any plans in this direction? So far, the only SPV mode implementation I know about is bitcoinj. I am experimenting with trans-piling bitcoinj to C++ to make it usable from Objective-C++ exactly with your use case in mind. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Introducing BitcoinKit.framework
Oracle provide an OSX JVM and will do so for the forseeable future, it's also open source, so the community could carry on if they stopped. The primary problem with the Oracle JVM is lack of retina support for Swing, but if you'd write a Cocoa UI yourself then it doesn't matter of course as Java won't handle any GUI stuff. Retina support for JavaFX2 (the current-gen gui toolkit) is available in Java 8 so it's definitely being actively developed, it's not abandoned or anything. So the question then becomes, which is better: a) Take bitcoinj completely out of the Java world via native compilation or transpilation to C++ b) Embed the JVM and link the two worlds together? (b) is much less ambitious, especially if you're OK with writing a bit of Java code to keep the interface thin. Basically the Java side calls into your app when interesting user-visible things happen, like new transactions appearing, then your GUI can call into the java side to send money. There are auto-translators that make the glue work easy, like https://code.google.com/p/javacpp/. You probably wouldn't want to expose the entire bitcoinj API that way because it's very large, but the code needed to bring up a wallet app is very small. I knocked one up this weekend in about one evenings worth of coding, completed with nice animations. The interfaces you'd need are basically some Objective-C++ methods that receive information from the Bitcoin side, like the balance having changed, a list of transactions, etc, and then a callback into the Java side to send money. If you look at the javacpp site you can see example code for making calls both ways. If I were in your shoes, I'd go for (b) because it is the most well trodden path and will let you achieve the best user visible results quickly. The JVM can be bundled with your app and stripped down if you're worried about download size. If it's unclear how the code would look, let me know and I'll try and knock up a really simple prototype. There's also (a). I'm investigating transpilation for a few reasons, one of which is to do with a private project. I'm working with the author of j2c: https://code.google.com/a/eclipselabs.org/p/j2c/. It's a rather sophisticated transpiler that converts Java to clean, readable C++11 that looks much like code a human would write. It's complete enough to transpile the entire standard Java class library, including all the GUI toolkits and other things - so, pretty amazing piece of code. However it's incomplete because where the Java code calls native methods (that would be provided by the JVM) it just spits out stubs you're expected to fill out yourself, for starting threads and so on. As there's no JVM it's just like using a C++ library that is missing a portability layer. I'm working on this myself and don't really need much help at the moment, I'm just making steady progress towards getting something up and running. I can let you know once I reach some interesting milestones. The point of this is that whilst you don't need access to most of the API to write a wallet app, I'd like to make every kind of app easy from C++, not just GUI wallets. Then the compile-to-C++ approach is much more appealing, even though it's also more work. On Mon, Jul 15, 2013 at 4:39 PM, Wendell w...@grabhive.com wrote: Hi Mike, You are absolutely right about the synchronize time, it's one of our main frustration points right now and we clearly won't deliver the kind of user experience we want, without fixing this. Actually we were thinking of extending Jeff Garzik's picocoin as time permits, but the plan is far from concrete at the moment. What you say about trans-piling bitcoinj is _really_ appealing. We discounted Java simply because an OS X JVM is no longer guaranteed, but otherwise bitcoinj is ideal for our purposes. How can we assist or otherwise accelerate such an effort? -w On Jul 15, 2013, at 3:19 PM, Mike Hearn wrote: That's great! I'm all for more wallets, especially user friendly UIs. However being based on bitcoind means it will take a very long time to synchronize for new users. We know a lot of users drop out. The best fix for this is SPV mode. Do you have any plans in this direction? So far, the only SPV mode implementation I know about is bitcoinj. I am experimenting with trans-piling bitcoinj to C++ to make it usable from Objective-C++ exactly with your use case in mind. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net
Re: [Bitcoin-development] Introducing BitcoinKit.framework
You can cut down the JVM to be a few megabytes if you're aggressive about it. But for a desktop app I'm not sure it's really necessary these days. A few megabytes used to make a noticeable difference to success rates but bandwidth improved a lot since then. Portability to android is a given, it's already Java based. IOS is a non starter until apple is convinced to allow wallet apps into the App store, language is not the issue there. There is no point manually rewriting bitcoinj to c++ when j2c does such a great job already. You would want to at last start from what it generates even if you fork from there. On 15 Jul 2013 20:19, Jonas Schnelli jonas.schne...@include7.ch wrote: The embedded JVM is a way. But the binary will be huge. And how about the portability (like iOS and Android)? If i would have unlimited resources and like to make a perfect native client i would see two solutions: a) add SPV features to bitcoind and go on with BitcoinKit.framework (maybe first SPV features are only available through API's and not for bitcoind as runnable binary) b) rewrite bitcoinj in c++ (*auto-port the unit-tests* and try to rewrite line by line to c++) also a mix of a) and b) is possible. Like extending bitcoind with the SPVBlockstore, bloom filter, etc. from bitcoinj (rewritten in c++). The wallet birthday must also be added somehow. both a) and b) (or the mix) is a lot of work and might take much longer as Mike's JVM idea. But it then might end up in a stable, portable and extendable pice of code. /jonas Oracle provide an OSX JVM and will do so for the forseeable future, it's also open source, so the community could carry on if they stopped. The primary problem with the Oracle JVM is lack of retina support for Swing, but if you'd write a Cocoa UI yourself then it doesn't matter of course as Java won't handle any GUI stuff. Retina support for JavaFX2 (the current-gen gui toolkit) is available in Java 8 so it's definitely being actively developed, it's not abandoned or anything. So the question then becomes, which is better: a) Take bitcoinj completely out of the Java world via native compilation or transpilation to C++ b) Embed the JVM and link the two worlds together? (b) is much less ambitious, especially if you're OK with writing a bit of Java code to keep the interface thin. Basically the Java side calls into your app when interesting user-visible things happen, like new transactions appearing, then your GUI can call into the java side to send money. There are auto-translators that make the glue work easy, like https://code.google.com/p/javacpp/. You probably wouldn't want to expose the entire bitcoinj API that way because it's very large, but the code needed to bring up a wallet app is very small. I knocked one up this weekend in about one evenings worth of coding, completed with nice animations. The interfaces you'd need are basically some Objective-C++ methods that receive information from the Bitcoin side, like the balance having changed, a list of transactions, etc, and then a callback into the Java side to send money. If you look at the javacpp site you can see example code for making calls both ways. If I were in your shoes, I'd go for (b) because it is the most well trodden path and will let you achieve the best user visible results quickly. The JVM can be bundled with your app and stripped down if you're worried about download size. If it's unclear how the code would look, let me know and I'll try and knock up a really simple prototype. There's also (a). I'm investigating transpilation for a few reasons, one of which is to do with a private project. I'm working with the author of j2c: https://code.google.com/a/eclipselabs.org/p/j2c/. It's a rather sophisticated transpiler that converts Java to clean, readable C++11 that looks much like code a human would write. It's complete enough to transpile the entire standard Java class library, including all the GUI toolkits and other things - so, pretty amazing piece of code. However it's incomplete because where the Java code calls native methods (that would be provided by the JVM) it just spits out stubs you're expected to fill out yourself, for starting threads and so on. As there's no JVM it's just like using a C++ library that is missing a portability layer. I'm working on this myself and don't really need much help at the moment, I'm just making steady progress towards getting something up and running. I can let you know once I reach some interesting milestones. The point of this is that whilst you don't need access to most of the API to write a wallet app, I'd like to make every kind of app easy from C++, not just GUI wallets. Then the compile-to-C++ approach is much more appealing, even though it's also more work. On Mon, Jul 15, 2013 at 4:39 PM, Wendell w...@grabhive.com wrote: Hi Mike, You are absolutely right about the synchronize time, it's one of our main
