Re: [Bitcoin-development] Plans to separate wallet from core
So first bitcoind will support SPV mode then we separate the wallet? Are the core and the wallet share any code (say, the p2p messages via a sub-repo or something)? bitcoind already supports SPV mode, that's how bitcoinj clients work. However the current wallet code doesn't use it, it integrates directly with the full mode main loop and doesn't talk P2P internally. Which is the fine and obvious way to implement the wallet feature. I'm not totally convinced it should become an SPV wallet given the complexity of doing that. But if you did want to separate the wallet code from the full node then that'd be the way to do it. The question is; what does this buy us, and is it worth the potentially huge amount of time it could take? My gut feeling is we have bigger fish to fry. There's plenty of work to do just on the core consensus code, making Bitcoin Core into a competitive wallet as well would be an additional burden. However I may be quite biased, as I am the maintainer of what is primarily a wallet library :) I'm sorry, but I still don't know what Electrum has to do with all this. People use Electrum as shorthand to mean something a bit like the P2P network, but with trusted remote servers which build additional databases and thus support additional commands. -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
The question is; what does this buy us, and is it worth the potentially huge amount of time it could take? My gut feeling is we have bigger fish to fry. There's plenty of work to do just on the core consensus code, making Bitcoin Core into a competitive wallet as well would be an additional burden. I don't intend to work on that myself but that's up to the people that want to contribute to that. Once it's a separate project it could either be a big success, or it could slowly wither away. It can have a release cycle separate from the node. Likely faster. The organizational reason to split off the wallet is to get rid of that responsibility (and code) from the bitcoind repo. Maintaining a wallet should not be part of maintaining the core infrastructure. But just deleting it would be unreasonable. However I may be quite biased, as I am the maintainer of what is primarily a wallet library :) Hah. I've thought about that migration path as well. From my experience the main thing people are missing with BitcoinJ is a quick and easy way to set up a wallet as a daemon, to use the functionality from non-java through RPC. But there are other interesting upcoming wallet projects as well, for example CoinVault. Wladimir -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
From my experience the main thing people are missing with BitcoinJ is a quick and easy way to set up a wallet as a daemon, to use the functionality from non-java through RPC. Yes. I'd love to have a mostly Core compatible JSON-RPC frontend. Most of my current users are happy using it as a library though. A lot of popular languages can run directly on the JVM these days. The big ones we miss are C++ and PHP, I think. But you can use JavaScript, Python 2.7, Lisp, Ruby, along with other less well known ones. The other good reason to have JSON-RPC support would be to reuse the Core regression tests. Anyway, this is off topic :) -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On 6/24/14, Mike Hearn m...@plan99.net wrote: bitcoind already supports SPV mode, that's how bitcoinj clients work. However the current wallet code doesn't use it, it integrates directly with the full mode main loop and doesn't talk P2P internally. Which is the fine and obvious way to implement the wallet feature. I'm not totally convinced it should become an SPV wallet given the complexity of doing that. But if you did want to separate the wallet code from the full node then that'd be the way to do it. The question is; what does this buy us, and is it worth the potentially huge amount of time it could take? My gut feeling is we have bigger fish to fry. There's plenty of work to do just on the core consensus code, making Bitcoin Core into a competitive wallet as well would be an additional burden. Exactly, this is part of my point, the qt-wallet does not support SPV operation at this point, and that complex work should be done after the wallet is separated. Thus the first version of the separated wallet should be functionally equivalent to today's wallet, that is, a full node wallet (even though I understand Wladimir's arguments against full-node wallets). I'm sorry, but I still don't know what Electrum has to do with all this. People use Electrum as shorthand to mean something a bit like the P2P network, but with trusted remote servers which build additional databases and thus support additional commands. Ok, thanks. So a bitcoin core node which maintains and serves additional indexes (say, an utxo index via rpc or something else) is referred to as an electrum even though it doesn't use electrum-server. Strange, but now I get it. So in summary: 1) I accept the library approach over the core-wallet protocol. 2) That doesn't necessarily mean that optionally maintaining additional indexes in the core is not interesting for some use cases (interesting for bitcoind, I don't care much if electrum-server currently does this and more [with more dependencies]). Although Pieter thinks that should also be separated into an index node too, but I think that's another discussion. 3) The wallet doesn't currently operate as SPV and that work should be done (if there's enough interest) only after the wallet is separated from the core. -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On Tue, Jun 24, 2014 at 1:29 PM, Jorge Timón jti...@monetize.io wrote: On 6/24/14, Mike Hearn m...@plan99.net wrote: ou did want to separate the wallet code from the full node then that'd be the way to do it. Exactly, this is part of my point, the qt-wallet does not support SPV operation at this point, and that complex work should be done after the wallet is separated. Thus the first version of the separated wallet should be functionally equivalent to today's wallet, that is, a full node wallet (even though I understand Wladimir's arguments against full-node wallets). Do mind that some of the steps on the path of bitcoind towards SPV are also useful in general. For example, headers-first allows parallel block download, which would solve a lot of sync issues people are having - a much higher priority than the wallet. But anyhow I'm describing how I would do it. If you're going to do it, you can do it in any order that you want. As we're talking about a separate project here it's not even clear who will be maintainer. 2) That doesn't necessarily mean that optionally maintaining additional indexes in the core is not interesting for some use cases (interesting for bitcoind, I don't care much if electrum-server currently does this and more [with more dependencies]). Although Pieter thinks that should also be separated into an index node too, but I think that's another discussion. I don't understand your argument against Electrum here. Dependencies? Come on, that's a matter of software distribution. If that really bothers you I suppose you could contribute to Electrum server so that it has less deps. It doesn't make the protocol worth any less. Although Pieter and I disagree with regard to issue #4351, we agree on wanting to keep (or at least making) bitcoind as lean as possible. Maintaining extra indices for others doesn't fit in there - that's also why the address index patch was not merged. An 'index node' could be a different animal. Wladimir -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
I think there are three typical uses: 1. Building consensus on the block chain. This is what the core is for. 2. Single user wallets. This is where SPV alone is good. 3. Services e.g. exchange, payment processor This is where core + indexing server talking SPV to core is the right choice Regards, Tamás Blummer Founder, CEO http://bitsofproof.com signature.asc Description: Message signed with OpenPGP using GPGMail -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On Tue, Jun 24, 2014 at 2:16 PM, Mike Hearn m...@plan99.net wrote: priority. So a single unified program that just figures it out automatically rather than expecting users to assemble a bag of parts seems a goal worth striving for. As I've said before -- and I think we disagree here - I like moving towards a bag of parts of specialized tools, maintained by people that specialize in those tools, instead of a single project that aims to do and know everything. This encourages experimentation and makes competition possible and I think that is healthy in this space. Bitcoin has a strict need for consensus in the block chain format, scripting system and validation. Outside of those, innovation should be possible without any gatekeeper bottleneck or even widespread agreement. Wallets, what data to store on disk, what indices to maintain. But even P2P message extensions, as long as it doesn't interfere with the rest of the network. After an experiment is successful it could always be merged into bitcoin core. But then the 'what-ifers' have less ammo, as it has been tested in the real world. For user convenience it's still possible to package pre-assembled bags. But that doesn't need to figure into how things are developed. Wladimir -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
Le 24/06/2014 11:44, Wladimir a écrit : But IMO this is a passed stage. SPV wallets w/ Bloom filtering can work without any special servers, so why require a 'close binding' to a trusted bitcoin core? To clarify (and not piss off ThomasV :-): I do not think the idea of having servers with a reputation of their own is a passed stage. There are many things that cannot be done at SPV level security with just the P2P protocol yet. So having fewer but more trusted Electrum servers is a reasonable compromise. Thanks for that :) Note that my goal is to make the Electrum servers as trustless as possible, and not to rely on some sort of 'reputation'. Thomas -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On 6/24/14, Tamas Blummer ta...@bitsofproof.com wrote: 3. Services e.g. exchange, payment processor This is where core + indexing server talking SPV to core is the right choice I think this is my main question, what's the advantage of having the processes talking via the p2p protocol instead of something more direct when you control both processes? Wladimir, of course headers-first is generally good, and of course nobody will be force to separate the code in a way he doesn't like, I was just testing the waters to see what people had in mind, since I realized the ideas could be more different than I had assumed. I don't have any issues with electrum, I'm just not convinced by the arguments that say that the indexes must be necessarily out of the core, specially when some of them could be committed in the future. So I'm all in favor of modularity and competing codebases, I'm just not convinced that the core full-node must be necessarily restricted to validation only (for example, I think it should maintain a minimal and non-optimized mining functionality,even if it's only used for testing purposes). So far it is great that everybody seems to agree that the wallet code needs to be separated. Thanks everyone for sharing your views on the subject. -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
I'd like to point out that there is quite a difference between what core nodes should be like and what the codebase core nodes are built from must support. Given sufficiently modularized code (which I think everyone seems to be in favor of, regardless of the goals), you can likely build a binary that does full verification and maintains some indexes of some sort. I still believe that what we push for to run as the core nodes of the network should aim for purely verification and relay, and nothing else, but people can and will do things differently if the source code allows it. And that's fine. On Tue, Jun 24, 2014 at 3:26 PM, Jorge Timón jti...@monetize.io wrote: I think this is my main question, what's the advantage of having the processes talking via the p2p protocol instead of something more direct when you control both processes? IMHO, maintaining a correct view of the current state of the chain (excluding blocks, just headers) is already sufficiently hard (I hope that everyone who ever implemented an SPV wallet can agree). You simplify things a bit by not needing to verify what the peer claims if you trust them, but not much. You still need to support reorganizations, counting confirmations, making sure you stay up-to-date. These are functions the (SPV) P2P protocol has already shown to do well, and there are several codebases out there that implement it. No need to reinvent the wheel with a marginally more efficient protocol, if it means starting over everything else. -- Pieter -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/24/2014 09:07 AM, Wladimir wrote: My main argument for the split is that full nodes and wallets have completely different usage scenarios: - A wallet should be online as little as possible, ideally only when you do transactions or want to check for them. - A full node should be online 24/7 or it is virtually useless to the network. I think btcd has done this right. A wallet is a daemon that runs constantly in the background, just like the full node. The GUI (which is distinct from the wallet) runs as little as possible. Presumably there's no need for a 1:1 relationship between wallets and GUIs. - -- Support online privacy by using email encryption whenever possible. Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJTqZpVAAoJEMP3uyY4RQ21E48H/0XNYBzR8QZjfku/MeL5IbwL A56jrlWe2EZTabwfKdGx12L5oeBXe3DOeLsTizqIu0vijcl7qQryU49AjrVe91Rx OdEi4lmaiXdkM3lWeWUxLoLLHR1B+1f8T18Mrnzo+yasyrywPb+6H79VN5KERdA2 5yHYCZyHXdNoXpzyf38GC2PdYJmAZdrRfAGyC5+OXSwE3XLhpRBrSBh/mrx0ct5M ghkCKtsmJCJJ6sR2TbFxaj71kPp0J0tp8JVvjVEqC2uB4Ih0NY+79kz8L81TaNmw ol1o6p7TypIk7QRQ4ES3Fq0ALh2aQ/tX4rc0GC0ed+xLi+dHJ2wGBI37HoyGIyg= =Nn9X -END PGP SIGNATURE- 0x38450DB5.asc Description: application/pgp-keys -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On 6/24/14, Justus Ranvier justusranv...@gmail.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/24/2014 09:07 AM, Wladimir wrote: My main argument for the split is that full nodes and wallets have completely different usage scenarios: - A wallet should be online as little as possible, ideally only when you do transactions or want to check for them. - A full node should be online 24/7 or it is virtually useless to the network. I think btcd has done this right. A wallet is a daemon that runs constantly in the background, just like the full node. The GUI (which is distinct from the wallet) runs as little as possible. Presumably there's no need for a 1:1 relationship between wallets and GUIs. I think he means that the wallet shouldn't be running as much as it is currently doing. But yes, I think you're right about wallets and GUIs not necessarily mapping 1:1. -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Plans to separate wallet from core
I know there are plans to separate the wallet from the core code and I think it's a great idea that will result in cleaner and more modular software. But it seems like my assumptions on how this would be done may be incorrect. I was assuming that the wallet would consume data from a trusted bitcoind core node using rpc or a better interface like an http rest api (see PR #2844). So the core would take care of the hard consensus stuff, and the wallet would maintain its own database with private keys, addresses, balances, etc. and would consume some data contained in bitcoind's database. I also assumed that the interface between wallet and core would include queries to the UTXO (see PR #4351) and maybe TXO (see PR #3652) for getting the historic balances. As said, I'm not sure these assumptions are true anymore so I ask. Is this the plan? Is the plan that the wallet will use the p2p directly and maintain its own chain database? Well, it's something that generally interests me so if anyone can detail the steps for separating the wallet a little bit, maybe I can help with some of the steps. Maybe there's no roadway yet. In that case I would like to advance that discussion in this thread. -- Jorge Timón -- HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing Easy Data Exploration http://p.sf.net/sfu/hpccsystems ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Plans to separate wallet from core
On Mon, Jun 23, 2014 at 11:50 AM, Jorge Timón jti...@monetize.io wrote: I know there are plans to separate the wallet from the core code and I think it's a great idea that will result in cleaner and more modular software. But it seems like my assumptions on how this would be done may be incorrect. I was assuming that the wallet would consume data from a trusted bitcoind core node using rpc or a better interface like an http rest api (see PR #2844). It's least surprising if the wallet works as a SPV client by default. Then, users can use it without first setting up a core. Thus the idea would be to use P2P primarily. There could be a mode to use a trusted core by RPC for mempool/conflicted transaction validation and such. But I'm not sure about this - as we've seen, pure-SPV wallets work pretty well. If you want it to act as an edge router you can point a SPV wallet at your trusted core as well. There are no plans for adding Electrum-like functionality to bitcoind. There is already Electrum. Let's not reinvent any wheels. So the core would take care of the hard consensus stuff, and the wallet would maintain its own database with private keys, addresses, balances, etc. and would consume some data contained in bitcoind's database. Right, the wallet would keep track of those. I also assumed that the interface between wallet and core would include queries to the UTXO (see PR #4351) and maybe TXO (see PR #3652) for getting the historic balances. As said, I'm not sure these assumptions are true anymore so I ask. Is this the plan? Is the plan that the wallet will use the p2p directly and maintain its own chain database? It does not need to keep a full chain database. But it needs its own record of the chain, headers-only + what concerns the keys in the wallet. Wladimir -- HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing Easy Data Exploration http://p.sf.net/sfu/hpccsystems ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
