Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Mike Hearn

 So first bitcoind will support SPV mode then we separate the wallet?
 Are the core and the wallet share any code (say, the p2p messages via
 a sub-repo or something)?


bitcoind already supports SPV mode, that's how bitcoinj clients work.
However the current wallet code doesn't use it, it integrates directly with
the full mode main loop and doesn't talk P2P internally. Which is the fine
and obvious way to implement the wallet feature. I'm not totally convinced
it should become an SPV wallet given the complexity of doing that. But if
you did want to separate the wallet code from the full node then that'd be
the way to do it.

The question is; what does this buy us, and is it worth the potentially
huge amount of time it could take? My gut feeling is we have bigger fish to
fry. There's plenty of work to do just on the core consensus code, making
Bitcoin Core into a competitive wallet as well would be an additional
burden.

However I may be quite biased, as I am the maintainer of what is primarily
a wallet library :)


 I'm sorry, but I still don't know what Electrum has to do with all this.


People use Electrum as shorthand to mean something a bit like the P2P
network, but with trusted remote servers which build additional databases
and thus support additional commands.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Wladimir
 The question is; what does this buy us, and is it worth the potentially huge
 amount of time it could take? My gut feeling is we have bigger fish to fry.
 There's plenty of work to do just on the core consensus code, making Bitcoin
 Core into a competitive wallet as well would be an additional burden.

I don't intend to work on that myself but that's up to the people that
want to contribute to that. Once it's a separate project it could
either be a big success, or it could slowly wither away. It can have a
release cycle separate from the node. Likely faster.

The organizational reason to split off the wallet is to get rid of
that responsibility (and code) from the bitcoind repo. Maintaining a
wallet should not be part of maintaining the core infrastructure. But
just deleting it would be unreasonable.

 However I may be quite biased, as I am the maintainer of what is primarily a 
 wallet library :)

Hah. I've thought about that migration path as well.

From my experience the main thing people are missing with BitcoinJ is
a quick and easy way to set up a wallet as a daemon, to use the
functionality from non-java through RPC.

But there are other interesting upcoming wallet projects as well, for
example CoinVault.

Wladimir

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Mike Hearn

 From my experience the main thing people are missing with BitcoinJ is
 a quick and easy way to set up a wallet as a daemon, to use the
 functionality from non-java through RPC.


Yes. I'd love to have a mostly Core compatible JSON-RPC frontend. Most of
my current users are happy using it as a library though. A lot of popular
languages can run directly on the JVM these days. The big ones we miss are
C++ and PHP, I think. But you can use JavaScript, Python 2.7, Lisp, Ruby,
along with other less well known ones.

The other good reason to have JSON-RPC support would be to reuse the Core
regression tests.

Anyway, this is off topic :)
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Jorge Timón
On 6/24/14, Mike Hearn m...@plan99.net wrote:
 bitcoind already supports SPV mode, that's how bitcoinj clients work.
 However the current wallet code doesn't use it, it integrates directly with
 the full mode main loop and doesn't talk P2P internally. Which is the fine
 and obvious way to implement the wallet feature. I'm not totally convinced
 it should become an SPV wallet given the complexity of doing that. But if
 you did want to separate the wallet code from the full node then that'd be
 the way to do it.

 The question is; what does this buy us, and is it worth the potentially
 huge amount of time it could take? My gut feeling is we have bigger fish to
 fry. There's plenty of work to do just on the core consensus code, making
 Bitcoin Core into a competitive wallet as well would be an additional
 burden.

Exactly, this is part of my point, the qt-wallet does not support SPV
operation at this point, and that complex work should be done after
the wallet is separated. Thus the first version of the separated
wallet should be functionally equivalent to today's wallet, that is, a
full node wallet (even though I understand Wladimir's arguments
against full-node wallets).

 I'm sorry, but I still don't know what Electrum has to do with all this.


 People use Electrum as shorthand to mean something a bit like the P2P
 network, but with trusted remote servers which build additional databases
 and thus support additional commands.

Ok, thanks. So a bitcoin core node which maintains and serves
additional indexes (say, an utxo index via rpc or something else) is
referred to as an electrum even though it doesn't use
electrum-server. Strange, but now I get it.

So in summary:

1) I accept the library approach over the core-wallet protocol.

2) That doesn't necessarily mean that optionally maintaining
additional indexes in the core is not interesting for some use cases
(interesting for bitcoind, I don't care much if electrum-server
currently does this and more [with more dependencies]). Although
Pieter thinks that should also be separated into an index node too,
but I think that's another discussion.

3) The wallet doesn't currently operate as SPV and that work should be
done (if there's enough interest) only after the wallet is separated
from the core.

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Wladimir
On Tue, Jun 24, 2014 at 1:29 PM, Jorge Timón jti...@monetize.io wrote:
 On 6/24/14, Mike Hearn m...@plan99.net wrote:
ou did want to separate the wallet code from the full node then that'd be
 the way to do it.

 Exactly, this is part of my point, the qt-wallet does not support SPV
 operation at this point, and that complex work should be done after
 the wallet is separated. Thus the first version of the separated
 wallet should be functionally equivalent to today's wallet, that is, a
 full node wallet (even though I understand Wladimir's arguments
 against full-node wallets).

Do mind that some of the steps on the path of bitcoind towards SPV are
also useful in general. For example, headers-first allows parallel
block download, which would solve a lot of sync issues people are
having - a much higher priority than the wallet.

But anyhow I'm describing how I would do it. If you're going to do it,
you can do it in any order that you want. As we're talking about a
separate project here it's not even clear who will be maintainer.

 2) That doesn't necessarily mean that optionally maintaining
 additional indexes in the core is not interesting for some use cases
 (interesting for bitcoind, I don't care much if electrum-server
 currently does this and more [with more dependencies]). Although
 Pieter thinks that should also be separated into an index node too,
 but I think that's another discussion.

I don't understand your argument against Electrum here. Dependencies?
Come on, that's a matter of software distribution. If that really
bothers you I suppose you could contribute to Electrum server so that
it has less deps. It doesn't make the protocol worth any less.

Although Pieter and I disagree with regard to issue #4351, we agree on
wanting to keep (or at least making) bitcoind as lean as possible.
Maintaining extra indices for others doesn't fit in there - that's
also why the address index patch was not merged. An 'index node' could
be a different animal.

Wladimir

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Tamas Blummer
I think there are three typical uses:

1. Building consensus on the block chain. This is what the core is for.
2. Single user wallets. This is where SPV alone is good.
3. Services e.g. exchange, payment processor  This is where core + indexing 
server talking SPV to core is the right choice

Regards,

Tamás Blummer
Founder, CEO

http://bitsofproof.com




signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Wladimir
On Tue, Jun 24, 2014 at 2:16 PM, Mike Hearn m...@plan99.net wrote:
 priority. So a single unified program that just figures it out automatically
 rather than expecting users to assemble a bag of parts seems a goal worth
 striving for.

As I've said before -- and I think we disagree here - I like moving
towards a bag of parts of specialized tools, maintained by people that
specialize in those tools, instead of a single project that aims to do
and know everything. This encourages experimentation  and makes
competition possible and I think that is healthy in this space.

Bitcoin has a strict need for consensus in the block chain format,
scripting system and validation. Outside of those, innovation should
be possible without any gatekeeper bottleneck or even widespread
agreement. Wallets, what data to store on disk, what indices to
maintain. But even P2P message extensions, as long as it doesn't
interfere with the rest of the network.

After an experiment is successful it could always be merged into
bitcoin core. But then the 'what-ifers' have less ammo, as it has been
tested in the real world.

For user convenience it's still possible to package pre-assembled
bags. But that doesn't need to figure into how things are developed.

Wladimir

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Thomas Voegtlin


Le 24/06/2014 11:44, Wladimir a écrit :
 But IMO this is a passed stage. SPV wallets w/ Bloom filtering can
 work without any special servers, so why require a 'close binding' to
 a trusted bitcoin core?
 
 To clarify (and not piss off ThomasV :-): I do not think the idea of
 having servers with a reputation of their own is a passed stage. There
 are many things that cannot be done at SPV level security with just
 the P2P protocol yet. So having fewer but more trusted Electrum
 servers is a reasonable compromise.
 


Thanks for that :)

Note that my goal is to make the Electrum servers as trustless as
possible, and not to rely on some sort of 'reputation'.

Thomas

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Jorge Timón
On 6/24/14, Tamas Blummer ta...@bitsofproof.com wrote:
 3. Services e.g. exchange, payment processor  This is where core +
 indexing server talking SPV to core is the right choice

I think this is my main question, what's the advantage of having the
processes talking via the p2p protocol instead of something more
direct when you control both processes?

Wladimir, of course headers-first is generally good, and of course
nobody will be force to separate the code in a way he doesn't like, I
was just testing the waters to see what people had in mind, since I
realized the ideas could be more different than I had assumed.
I don't have any issues with electrum, I'm just not convinced by the
arguments that say that the indexes must be necessarily out of the
core, specially when some of them could be committed in the future.
So I'm all in favor of modularity and competing codebases, I'm just
not convinced that the core full-node must be necessarily restricted
to validation only (for example, I think it should maintain a minimal
and non-optimized mining functionality,even if it's only used for
testing purposes).

So far it is great that everybody seems to agree that the wallet code
needs to be separated.

Thanks everyone for sharing your views on the subject.

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Pieter Wuille
I'd like to point out that there is quite a difference between what
core nodes should be like and what the codebase core nodes are built
from must support.

Given sufficiently modularized code (which I think everyone seems to
be in favor of, regardless of the goals), you can likely build a
binary that does full verification and maintains some indexes of some
sort.

I still believe that what we push for to run as the core nodes of the
network should aim for purely verification and relay, and nothing
else, but people can and will do things differently if the source code
allows it. And that's fine.

On Tue, Jun 24, 2014 at 3:26 PM, Jorge Timón jti...@monetize.io wrote:
 I think this is my main question, what's the advantage of having the
 processes talking via the p2p protocol instead of something more
 direct when you control both processes?

IMHO, maintaining a correct view of the current state of the chain
(excluding blocks, just headers) is already sufficiently hard (I hope
that everyone who ever implemented an SPV wallet can agree). You
simplify things a bit by not needing to verify what the peer claims if
you trust them, but not much. You still need to support
reorganizations, counting confirmations, making sure you stay
up-to-date. These are functions the (SPV) P2P protocol has already
shown to do well, and there are several codebases out there that
implement it. No need to reinvent the wheel with a marginally more
efficient protocol, if it means starting over everything else.

-- 
Pieter

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Justus Ranvier
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/24/2014 09:07 AM, Wladimir wrote:
 My main argument for the split is that full nodes and wallets have 
 completely different usage scenarios:
 
 - A wallet should be online as little as possible, ideally only
 when you do transactions or want to check for them.
 
 - A full node should be online 24/7 or it is virtually useless to
 the network.

I think btcd has done this right.

A wallet is a daemon that runs constantly in the background, just like
the full node.

The GUI (which is distinct from the wallet) runs as little as
possible. Presumably there's no need for a 1:1 relationship between
wallets and GUIs.


- -- 
Support online privacy by using email encryption whenever possible.
Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJTqZpVAAoJEMP3uyY4RQ21E48H/0XNYBzR8QZjfku/MeL5IbwL
A56jrlWe2EZTabwfKdGx12L5oeBXe3DOeLsTizqIu0vijcl7qQryU49AjrVe91Rx
OdEi4lmaiXdkM3lWeWUxLoLLHR1B+1f8T18Mrnzo+yasyrywPb+6H79VN5KERdA2
5yHYCZyHXdNoXpzyf38GC2PdYJmAZdrRfAGyC5+OXSwE3XLhpRBrSBh/mrx0ct5M
ghkCKtsmJCJJ6sR2TbFxaj71kPp0J0tp8JVvjVEqC2uB4Ih0NY+79kz8L81TaNmw
ol1o6p7TypIk7QRQ4ES3Fq0ALh2aQ/tX4rc0GC0ed+xLi+dHJ2wGBI37HoyGIyg=
=Nn9X
-END PGP SIGNATURE-


0x38450DB5.asc
Description: application/pgp-keys
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-24 Thread Jorge Timón
On 6/24/14, Justus Ranvier justusranv...@gmail.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 06/24/2014 09:07 AM, Wladimir wrote:
 My main argument for the split is that full nodes and wallets have
 completely different usage scenarios:

 - A wallet should be online as little as possible, ideally only
 when you do transactions or want to check for them.

 - A full node should be online 24/7 or it is virtually useless to
 the network.

 I think btcd has done this right.

 A wallet is a daemon that runs constantly in the background, just like
 the full node.

 The GUI (which is distinct from the wallet) runs as little as
 possible. Presumably there's no need for a 1:1 relationship between
 wallets and GUIs.

I think he means that the wallet shouldn't be running as much as it is
currently doing.
But yes, I think you're right about wallets and GUIs not necessarily
mapping 1:1.

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Plans to separate wallet from core

2014-06-23 Thread Wladimir
On Mon, Jun 23, 2014 at 11:50 AM, Jorge Timón jti...@monetize.io wrote:
 I know there are plans to separate the wallet from the core code and I
 think it's a great idea that will result in cleaner and more modular
 software.
 But it seems like my assumptions on how this would be done may be incorrect.

 I was assuming that the wallet would consume data from a trusted
 bitcoind core node using rpc or a better interface like an http rest
 api (see PR #2844).

It's least surprising if the wallet works as a SPV client by default.
Then, users can use it without first setting up a core. Thus the idea
would be to use P2P primarily.

There could be a mode to use a trusted core by RPC for
mempool/conflicted transaction validation and such. But I'm not sure
about this - as we've seen, pure-SPV wallets work pretty well. If you
want it to act as an edge router you can point a SPV wallet at your
trusted core as well.

There are no plans for adding Electrum-like functionality to bitcoind.
There is already Electrum. Let's not reinvent any wheels.

 So the core would take care of the hard consensus stuff, and the
 wallet would maintain its own database with private keys, addresses,
 balances, etc. and would consume some data contained in bitcoind's
 database.

Right, the wallet would keep track of those.

 I also assumed that the interface between wallet and core would
 include queries to the UTXO (see PR #4351) and maybe TXO (see PR
 #3652) for getting the historic balances.

 As said, I'm not sure these assumptions are true anymore so I ask.
 Is this the plan?
 Is the plan that the wallet will use the p2p directly and maintain its
 own chain database?

It does not need to keep a full chain database. But it needs its own
record of the chain, headers-only + what concerns the keys in the
wallet.

Wladimir

--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing  Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development