Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Fri, Apr 11, 2014 at 5:54 PM, Gregory Maxwell gmaxw...@gmail.com wrote: For the non-error-coded case I believe nodes with random spans of blocks works out asymptotically to the same failure rates as random. If each block is really 512 blocks in sequence, then each slot is more likely to be hit. It effectively reduces the number of blocks by the minimum run lengths. ECC seemed cooler though. (The conversation Peter Todd was referring to was one where I was pointing out that with suitable error coding you also get an anti-censorship effect where its very difficult to provide part of the data without potentially providing all of it) Interesting too. I think in the network we have today and for the foreseeable future we can reasonably count on there being a reasonable number of nodes that store all the blocks... quite likely not enough to satisfy the historical block demand from the network alone, but easily enough to supply blocks that have otherwise gone missing. That's true. Scaling up the transactions per second increases the chance of data lost. With side/tree chains, the odds of data loss in the less important chains increases (though they are by definition lower value chains) -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple to use. Nothing to install. Get started now for free. http://p.sf.net/sfu/SauceLabs___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Thu, Apr 10, 2014 at 10:30 AM, Tier Nolan tier.no...@gmail.com wrote: Error correction is an interesting suggestion. Though I mentioned it, it was in jest— I think right now it would be an over-design at least for the basic protocol. Also, storing 'random' blocks has some locality problems, when verifying blocks you need to obtain them contiguously, and so we can take advantage of the locality of reference. For the non-error-coded case I believe nodes with random spans of blocks works out asymptotically to the same failure rates as random. One thing that I like to point out is that there is absolutely no need for the entire network to use the same p2p protocol. Diversity here would be very good. I think it would be really good for someone to have an alternative p2p protocol using these techniques even though I think they aren't yet compelling enough to be table stakes in the basic protocol. There are some very helpful things you can do with forward error correction for faster and more efficient block relaying too: https://en.bitcoin.it/wiki/User:Gmaxwell/block_network_coding (The conversation Peter Todd was referring to was one where I was pointing out that with suitable error coding you also get an anti-censorship effect where its very difficult to provide part of the data without potentially providing all of it) If there was 1 nodes and each stored 0.1% of the blocks, at random, then the odds of a block not being stored is 45 in a million. I think in the network we have today and for the foreseeable future we can reasonably count on there being a reasonable number of nodes that store all the blocks... quite likely not enough to satisfy the historical block demand from the network alone, but easily enough to supply blocks that have otherwise gone missing. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Hi Wladimir, If the motivation of the SPV wallet is to radically extend functionality, as in my case, then the index is specific to the added features and the subset of the blockchain that is of interest for the wallet. As you also point out, adding huge generic purpose indices to core would rather discourage people using full nodes due to excess requirements. I believe nothing would add more to the core’s popularity as a trusted background node to SPV than full validation at lowest possible memory, disk and CPU footprint. Serving headers should be default but storing and serving full blocks configurable to ranges, so people can tailor to their bandwith and space available. Tamas Blummer Bits of Proof On 09.04.2014, at 21:25, Wladimir laa...@gmail.com wrote: Adding a RPC call for a address - utxo query wouldn't be a big deal. It has been requested before for other purposes as well, all the better if it helps for interaction with Electrum. Spent history would be involve a much larger index, and it's not likely that will end up in bitcoin Wladimir signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I tend to agree with slush here - counting the IPs in addr broadcasts often gives a number like 100,000 vs just 10,000 for actually reachable nodes (or less). It seems like optimising the NAT tunneling code would help. Starting by adding more diagnostic stuff to the GUI. STUN support may also help. The main constraint with home devices is not IMHO their actual power but rather that a lot of people no longer keep computers switched on all the time. If you don't do that then spv with bundled Core can't help your security because the spv wallet would always be syncing from the p2p network for performance reasons. On 9 Apr 2014 22:13, slush sl...@centrum.cz wrote: I believe there're plenty bitcoind instances running, but they don't have configured port forwarding properly.There's uPNP support in bitcoind, but it works only on simple setups. Maybe there're some not yet considered way how to expose these *existing* instances to Internet, to strenghten the network. Maybe just self-test indicating the node is not reachable from outside (together with short howto like in some torrent clients). These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically? Maybe there're other ideas how to improve current situation without needs of reworking the architecture. Marek On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell gmaxw...@gmail.comwrote: On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier justusranv...@gmail.com wrote: Anyone reading the archives of the list will see about triple the number of people independently confirming the resource usage problem than they will see denying it, so I'm not particularly worried. The list has open membership, there is no particular qualification or background required to post here. Optimal use of an information source requires critical reading and understanding the limitations of the medium. Counting comments is usually not a great way to assess technical considerations on an open public forum. Doubly so because those comments were not actually talking about the same thing I am talking about. Existing implementations are inefficient in many known ways (and, no doubt, some unknown ones). This list is about developing protocol and implementations including improving their efficiency. When talking about incentives the costs you need to consider are the costs of the best realistic option. As far as I know there is no doubt from anyone technically experienced that under the current network rules full nodes can be operated with vastly less resources than current implementations use, it's just a question of the relatively modest implementation improvements. When you argue that Bitcoin doesn't have the right incentives (and thus something??) I retort that the actual resource _requirements_ are for the protocol very low. I gave specific example numbers to enable correction or clarification if I've said something wrong or controversial. Pointing out that existing implementations are not that currently as efficient as the underlying requirements and that some large number of users do not like the efficiency of existing implementations doesn't tell me anything I disagree with or didn't already know. Whats being discussed around here contributes to prioritizing improvements over the existing implementations. I hope this clarifies something. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Thu, Apr 10, 2014 at 8:38 AM, Mike Hearn m...@plan99.net wrote: I tend to agree with slush here - counting the IPs in addr broadcasts often gives a number like 100,000 vs just 10,000 for actually reachable nodes (or less). It seems like optimising the NAT tunneling code would help. Starting by adding more diagnostic stuff to the GUI. STUN support may also help. The main constraint with home devices is not IMHO their actual power but rather that a lot of people no longer keep computers switched on all the time. If you don't do that then spv with bundled Core can't help your security because the spv wallet would always be syncing from the p2p network for performance reasons. I agree that there is a fundamental incompatibility in usage between wallets and nodes. Wallets need to be online as little as possible, nodes need to online as much as possible. However, a full node background process could also be running if the wallet is not open itself. Ffor example - by running as a system service. Bitcoin Core's own wallet is also moving to SPV, so this means a general solution is needed to get people to run a node when the wallet is not running. Maybe the node shouldn't be controlled from the wallet at all, it could be a 'node control' user interface on its own (this is what -disablewallet does currently). In this case, there is no need for packaging it with a wallet The only drawback would be that initially, people wouldn't know why or when to install this, hence my suggestion to pack it with wallets... Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
It's an optimisation problem. Home environments are much more hostile than servers are due to things like virus scanners, wildly varying memory pressure as apps are started and shut down, highly asymmetrical upstream versus downstream bandwidth, complicated nat setups, people who only use laptops (which I think is most people these days) and so on. So I think the right way to go is to optimise the things that hurt server node operators like large memory and disk usage, and this will automatically make it more pleasant to run on the desktop as well. If at some point all the low hanging fruit for the server side is gone then improving things on the desktop would be the next place to go. But we have to be realistic. Desktop tower machines that are always on are dying and will not be coming back. Not a single person I know uses them anymore, they have been wiped out in favour of laptops. This is why, given the tiny size of the bitcoin core development team, I do not think it makes sense to spend precious coding hours chasing this goal. On 10 Apr 2014 08:51, Wladimir laa...@gmail.com wrote: On Thu, Apr 10, 2014 at 8:38 AM, Mike Hearn m...@plan99.net wrote: I tend to agree with slush here - counting the IPs in addr broadcasts often gives a number like 100,000 vs just 10,000 for actually reachable nodes (or less). It seems like optimising the NAT tunneling code would help. Starting by adding more diagnostic stuff to the GUI. STUN support may also help. The main constraint with home devices is not IMHO their actual power but rather that a lot of people no longer keep computers switched on all the time. If you don't do that then spv with bundled Core can't help your security because the spv wallet would always be syncing from the p2p network for performance reasons. I agree that there is a fundamental incompatibility in usage between wallets and nodes. Wallets need to be online as little as possible, nodes need to online as much as possible. However, a full node background process could also be running if the wallet is not open itself. Ffor example - by running as a system service. Bitcoin Core's own wallet is also moving to SPV, so this means a general solution is needed to get people to run a node when the wallet is not running. Maybe the node shouldn't be controlled from the wallet at all, it could be a 'node control' user interface on its own (this is what -disablewallet does currently). In this case, there is no need for packaging it with a wallet The only drawback would be that initially, people wouldn't know why or when to install this, hence my suggestion to pack it with wallets... Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
You ask why people would install this ? I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network. What about serving archive blocks to peers paying for it ? Tamas Blummer http://bitsofproof.com On 10.04.2014, at 08:50, Wladimir laa...@gmail.com wrote: The only drawback would be that initially, people wouldn't know why or when to install this, hence my suggestion to pack it with wallets... Wladimir signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network. It's not a new idea, obviously, but there are some practical consequences: 1) To pay a node for serving, you have to have bitcoins. To get bitcoins, you need to sync with the network via a node. Catch 22. 2) If some nodes choose to charge and others choose to not charge, a smart wallet will always use the free nodes. In the absence of any global load balancing algorithms, this would lead to the free nodes getting overloaded and collapsing whilst the for-pay nodes remain silent. 3) The only payment channel implementations today are bitcoinj's (Java) and one written by Jeff in Javascript. There are no C++ implementations. And as Matt and I can attest to, doing a real, solid, fully debugged implementation that's integrated into a real app is a lot of work. I still think the lowest hanging fruit is basic, boring optimisations rather than architectural rethinks. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 But we have to be realistic. Desktop tower machines that are always on are dying and will not be coming back. Not a single person I know uses them anymore, they have been wiped out in favour of laptops. This is why, given the tiny size of the bitcoin core development team, I do not think it makes sense to spend precious coding hours chasing this goal. Your social group is weird. Nearly every coworker at my previous job had a tower computer at work and at home. Equally in my nontechnical social group lots of people, a significant minority if not majority, have Apple and PC desktops hooked up to large monitors at home for media production and games. Those who don't more often than not have laptops used as desktops, sitting in one place 95% of the time and left on. People have found it most efficient to work at a static desk for centuries - that's not going away. Of course we're seeing desktop usage and sales falling, but that's only because previously the mobile usage was forced into suboptimal options by technical realities. The trend will bottom out a long way from zero. Besides, even if just 1% of bitcoin users had a machine they left on that could usefully contribute to the network it would still vastly outweigh the much smaller percentage who would run nodes on expensive hosted capacity out of the goodness of their hearts. If we educated users about the privacy advantages of full nodes and gave them software that automatically contributed back within defined limits we'd have tens of thousands more useful nodes in the exact same way that user friendly filesharing software has lead to millions of users contributing bandwidth to filesharing networks. Similarly take advantage of the fault tolerance inherent in what we're doing and ensure that our software can shrug off nodes with a few % of downtime - certainly possible. Of course, this doesn't fit in the business plans of those who might want to run full nodes to data mine and deanonymize users for marketing, tax collection, and law enforcement - one of the few profitable things to do with a full node - but screw those people. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRmVPMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYb9B/0QWgQusFrQoPfYIzig vpeo9sQgHAfA3gPYdqDLTtlaTgt8w3oP+/N46+Pi3lEphhCusXstzAFxi6c0XXsk s96l9MqrUCZu55oEj1wZP0LJQx1uyUdevFv9bmocG5+94jBvGIoy3JZ3iQg+nNjL uE9FpTnL43VOZ0WI9g6pXIE6XX6CxTx56tcxla4LTTypA1DijXa9MyYGOfYxXdPJ w7jiRKl6Ijb3twP8+pX07GSIlL9yP7bESydnwyzwEo/RxAJxPmUpLuxluQ5DKTNY G9TtwCpT+c6g5nXOxkI31XRcDuzhT+2kEhiDA6Neu2YNGrnyQx2WL6XuZNhi8nKB IOMm =rwGV -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I know the idea is not new. Just bringing it up to emphasize that if we don’t use it how could we expect other networks using it. Machine to machine micro payments could become the killer application for Bitcoin. 1) There is no catch 22 as there are plenty of ways getting bitcoin without bootstrapping a full node. 2) let markets work out and not speculate what would happen. 3) Serving archive bolcks does not have to be part of core but could be a distinct service written in a language of your choice using new protocol. As mentioned earlier I am for a stripped down core that does nothing else than consensus and stores nothing else needed for that task and offering SPV api to the wallets. Tamas Blummer http://bitsofproof.com On 10.04.2014, at 11:17, Mike Hearn m...@plan99.net wrote: I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network. It's not a new idea, obviously, but there are some practical consequences: 1) To pay a node for serving, you have to have bitcoins. To get bitcoins, you need to sync with the network via a node. Catch 22. 2) If some nodes choose to charge and others choose to not charge, a smart wallet will always use the free nodes. In the absence of any global load balancing algorithms, this would lead to the free nodes getting overloaded and collapsing whilst the for-pay nodes remain silent. 3) The only payment channel implementations today are bitcoinj's (Java) and one written by Jeff in Javascript. There are no C++ implementations. And as Matt and I can attest to, doing a real, solid, fully debugged implementation that's integrated into a real app is a lot of work. I still think the lowest hanging fruit is basic, boring optimisations rather than architectural rethinks. signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 05:17:28 GMT-04:00, Mike Hearn m...@plan99.net wrote: I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network. It's not a new idea, obviously, but there are some practical consequences: You're both missing a more important issue: a core security assumption of bitcoin is that information is so easy to spread that censorship of it becomes impractical. If we're at the point where nodes are charging for their data we've failed that assumption. More concretely, if my business is charging for block chain data and I can make a profit doing so via micro payments I have perverse incentives to drive away my competitors. If I give a peer a whole block they can sell access to that information in turn. Why would I make it easy for them if I don't have too? Anyway, much of this discussion seems to stem from the misconception that contributing back to the network is a binary all or nothing thing - it's not. Over a year ago I myself was lamenting how I and the other bitcoin-wizards working on scalability had quickly solved every scaling problem *but* how to make it possible to scale up and keep mining decentralised. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRmi9MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhT/BB/98DudTV85hmruk0WRS xVt7gGK6DJ2Isc7iJE09i9wSQc/PbHu7rZ2MYPreOdVzRmlHYhOV2ShnZaZJ7A9g qB8pwy1wVrZgbrFeXXOLegNxGF2Xzc8OtL1E0bkNtTUUkuPIvT3UV4xn/Z+UZToR XImXpfakfJvyRH80cbMNu4xG/t7Ym4K63nEpCCdsEKNm5a1vHTNRTNfGYMC1wrSV XI3boZk7BQjqFDZADnonUU9zQ1WOmpdaVBYm+Xtgc+HXl3QODLcwGCY9hIRvaxek L+IYX9yTVbgngDGy70BYG4ekWxvMtNzRU+9HG5DgA1/Er9r1KIaf/98xiFR7RVB4 Yfia =34aD -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
1) There is no catch 22 as there are plenty of ways getting bitcoin without bootstrapping a full node. I think I maybe wasn't clear. To spend coins you need transaction data. Today, the dominant model is that people get that data by scanning the block chain. If you can obtain the transaction data without doing that then, either: 1) Someone is doing chain scanning for free. See my point about why pay if you can get it for free. 2) You got your tx data direct from the person you who sent you the funds, perhaps via the payment protocol. This would resolve the catch 22 by allowing you to spend bitcoins without actually having talked to the P2P network first, but we're a loong way from this world. And that's it. I don't think there are any other ways to get the tx data you need. Either someone gives it to you in the act of spending, or someone else gives it away for free, undermining the charge-for-the-p2p-network model. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Thanks, Peter and you convinced me. I run away with a thought. It’d be great to find a spot to deploy payment channels, but I agree this is not it. Tamas Blummer http://bitsofproof.com On 10.04.2014, at 12:40, Mike Hearn m...@plan99.net wrote: 1) There is no catch 22 as there are plenty of ways getting bitcoin without bootstrapping a full node. I think I maybe wasn't clear. To spend coins you need transaction data. Today, the dominant model is that people get that data by scanning the block chain. If you can obtain the transaction data without doing that then, either: 1) Someone is doing chain scanning for free. See my point about why pay if you can get it for free. 2) You got your tx data direct from the person you who sent you the funds, perhaps via the payment protocol. This would resolve the catch 22 by allowing you to spend bitcoins without actually having talked to the P2P network first, but we're a loong way from this world. And that's it. I don't think there are any other ways to get the tx data you need. Either someone gives it to you in the act of spending, or someone else gives it away for free, undermining the charge-for-the-p2p-network model. signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Thu, Apr 10, 2014 at 8:04 AM, Tamas Blummer ta...@bitsofproof.comwrote: Serving headers should be default but storing and serving full blocks configurable to ranges, so people can tailor to their bandwith and space available. I do agree that it is important. This does require changes to the P2P protocol, as currently there is no way for a node to signal that they store only part of the block chain. Also, clients will have to be modified to take this into account. Right now they are under the assumption that every full node can send them every (previous) block. What would this involve? Do you know of any previous work towards this? Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
What would this involve? Do you know of any previous work towards this? Chain pruning is a fairly complicated project, partly because it spans codebases. For instance if you try and implement it *just* by changing Bitcoin Core, you will break all the SPV clients based on bitcoinj (i.e. all of them). Big changes to the P2P network like this require upgrading both codebases simultaneously. I think things like this may be why Gavin is now just chief scientist instead of Core maintainer - in future, the changes people need will span projects and require fairly significant planning. From a technical perspective, it means extending addr broadcasts so nodes broadcast how much of the chain they have, and teaching both Core and bitcoinj how to search for nodes that have enough of the chain for them to use. Currently bitcoinj still doesn't use addr broadcasts at all, there's an incomplete patch available but it was never finished or merged. So that has to be fixed first. And that probably implies improving Bitcoin Core so the results of getaddr are more usable, ideally as high quality as what the DNS seeds provide, because if lots of bad addresses are returned this will slow down initial connect time, which is an important performance metric. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 06:44:32 GMT-04:00, Tamas Blummer ta...@bitsofproof.com wrote: Thanks, Peter and you convinced me. I run away with a thought. It’d be great to find a spot to deploy payment channels, but I agree this is not it. No problem! I'm sure we'll see payment channels implemented sooner or later the form of hub and spoke payment networks. The idea there is you have one or more centralised hubs who in turn have payment channels setup to and from payors and payees. So long as the person you want to pay is connected to the same hub as you are, or in more advanced versions connected via a ripple style chain, you can push payment to the hub and get proof they did the same for the recipient. Your loss is always limited to the incremental payment amount and payment is essentially instant. Of course, it's got some disadvantages compared to standard bitcoin transactions - its less decentralised - but when compared to other forms of off-chain payment in most situations its a strict improvement, and having the capability available is always a strict improvement. Like fidelity bonded banks the trust required in the hubs is low enough that with some minor effort applied to anti-DoS you could probably get away with using even hubs run by anonymous actors, making the centralisation less important. (hubs are essentially interchangeable) Unlike pure fidelity bonded banks the effort required to build this is relatively minor! You can even combine it with chaum tokens for anonymity. You'll want to hold the tokens for some amount of time to thwart timing analysis, leaving you somewhat vulnerable to theft, but in that case fidelity bonded banking principles can be applied. Other than that case the idea is probably made obsolete by micropayment hubs. Regulatory issues will be interesting... If you wind up with a few central payment hubs, without chaum tokens, those hubs learn all details about every transaction made. Obviously if a big actor like BitPay implemented this there would be a lot of pressure on them to make those records available to law enforcement and tax authorities, not to mention marketing and other data mining. Equally I suspect that if an alternative more decentralised version was implemented there would be strong government pressure for those approved hubs to not interoperate with the decentralised hubs, and equally for merchants to not accept payment from the decentralised hubs. But all the same, if widely implemented this reduces pressure to raise the block size enormously, keeping the underlying system decentralised. So the net effect is probably positive regardless. Oh yeah, credit goes to Mike Hearn for the payment channels, and if I'm correct, for the hub concept as well. Amir: You should think about adding the above to dark wallet. It'd be good if the protocols are implemented in an open and decentralised fashion first, prior to vendor lock in. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRoIlMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhW26B/9A9OtYjoSHo620XZzF VqfnnVFCPr3DpD/HuT3JYhF1kkL2vTt5wkRIHmHmfJ29Sduj8St7EFiLOyUg2mvt q9heZgzCnqxLJm9zMiiQnb3Y/plvhTLfaONnHI+OPSfrABL6DA04nEe8OBPuFfv/ NowJ74DP/65YBq3EqbqG0dJExKm1BhdrEpWNq0v5YoCVuEYkWgFHL8SdRHnfFyxA XTkP8avzlG82r98k55IrV0O/6VQNHjE0+xF4EHjEYBacy6OwlpEYeLrqx/VDAQ5R RufXeAltNZI0tzLQ8nY0aMBH3YFxF0+14/sbmOAtmnD6EW49gEcV9MnSJc5ct4m7 Szq5 =aC39 -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 07:32:44 GMT-04:00, Pieter Wuille pieter.wui...@gmail.com wrote: There were earlier discussions. The two ideas were either using one or a few service bits to indicate availability of blocks, or to extend addr messages with some flags to indicate this information. I wonder whether we can't have a hybrid: bits to indicate general degree of availability of blocks (none, only recent, everything), but indicate actual availability only upon actually connecting (through a version extension, or - preferably - a separate message). Reason is that the actual blocks available are likely to change frequently (if you keep the last week of blocks, a 3-day old addr entry will have quite outdated information), and not that important to actual peer selection - only to drive the decision which blocks to ask after connection. Why not just put an expiration date on that information and delay deletion until the expiration is reached? Also, its worth noting how the node bit solution you proposed can be done as a gradual upgrade path for SPV client. From the perspective of nodes that don't know about it they just see the pruned nodes as SPV nodes without any chain data at all. The only issue would be if large numbers of uses turned off their full nodes, but that's a possibility regardless. Done with partial UTXO set mode this may even result in an eventual increase in the number of full nodes. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRoPZMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhetrCACA02EJQ0VpcYvafuNc 7pvqMVeirJRu3Uv7Wy8rcl9jW5irM5fmNdznARtv2vwpEZN7MU0wp3ZY1FYOCv2f PvWC7DBCSBs2BuyGkvPuwnXEppTrYmWFT3qjg+99lF1IlOV4yWFacja2RGDuJkea fYUkODosHJjFVcXi5aMkBPQ5sOFdlUVbC94YV4d4PDSmF2fHLGG8uEfEweYb6Pv+ gj1CsfuAWf8DWzygDeL8x/wOG9HeqYqEbjxyOb9hxlp1ByUof+4WJtz3QfGsR2Xt fvkmgS8vkUxSIZorMdypj7oLBOnfDW1bEK5He2SlqPdYi5FEQusZ/jMMX3Fw74GV fJKt =Wyv8 -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Oh yeah, credit goes to Mike Hearn for the payment channels, and if I'm correct, for the hub concept as well. Actually, the design is from Satoshi and Matt did most of the implementation work last year during a Google internship. Though I ended up doing a lot of work on it too. We actually got pretty far: there was Android UI for it and a couple of apps we coded up. I wish we could have pushed it over the finishing line and got real world usage. Hopefully we can return to it someday soon. I think the hub/spoke concept was invented by goldsmiths in 16th century Italy, as they started handing pieces of paper across their benches, or *bancos* in Italian :-) -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Thu, Apr 10, 2014 at 4:32 AM, Pieter Wuille pieter.wui...@gmail.com wrote: There were earlier discussions. On this list. The two ideas were either using one or a few service bits to indicate availability of blocks, or to extend addr messages with some flags to indicate this information. I wonder whether we can't have a hybrid: bits to indicate general degree of availability of blocks (none, only recent, everything), but indicate actual availability only upon actually connecting (through a version extension, or - preferably - a separate message). Reason is that the actual blocks available are likely to change frequently (if you keep the last week of blocks, a 3-day old addr entry will have quite outdated information), and not that important to actual peer selection - only to drive the decision which blocks to ask after connection. I think you actually do need the kept ranges to be circulated, otherwise you might need to hunt for a very long time to find the right nodes with the blocks you need. Alternatively, you give up and don't hunt and pick some node that has them all and we get poor load distribution. I'd rather be in a case where the nodes that have the full history are only hit as a last resort. WRT the changing values, I think that is pretty uniquely related to the most recent blocks, and so instead I think that should be handled symbolically (e.g. the hybrid approach... a flag for the I keep the most recent 2000 blocks, I say 2000 because thats about where the request offset historgrams flattened out) or as a single offset range I keep the last N=200, and the flag or the offset would be in addition to whatever additional range was signaled. The latter could be infrequently changing. Signaling _more_ and more current range data on connect seems fine to me, I just don't think it replaces something that gets relayed. Based on the safety against reorgs and the block request access patterns we observed I'm pretty sure we'd want any node serving blocks at all to be at least the last N (for some number between 144 and 2000 or so). Based on the request patterns if just the recent blocks use up all the space you're willing to spend, then I think thats probably still the optimal contribution... (Just be glad I'm not suggesting coding the entire blockchain with an error correcting code so that it doesn't matter which subset you're holding) -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 07:45:16 GMT-04:00, Mike Hearn m...@plan99.net wrote: Oh yeah, credit goes to Mike Hearn for the payment channels, and if I'm correct, for the hub concept as well. Actually, the design is from Satoshi and Matt did most of the implementation work last year during a Google internship. Ah right, of course. Along those lines we should credit Jeremy Spilman (?) for figuring out how to get rid of the dependency on nSequence, makimg the protocol trust-free. I do recall it having an issue with malleability, semi-fixed with the P2SH trick. Be good to clear that up for good for Pieter's proposed malleability patch. Though I ended up doing a lot of work on it too. We actually got pretty far: there was Android UI for it and a couple of apps we coded up. I wish we could have pushed it over the finishing line and got real world usage. Hopefully we can return to it someday soon. I think the hub/spoke concept was invented by goldsmiths in 16th century Italy, as they started handing pieces of paper across their benches, or *bancos* in Italian :-) ...and only took another five hundred years for math to catch up and make it trust free, modulo miner centralisation! -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRoYCMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhdNMB/9zFBT3nFSkasHkMVW8 01pE/VYrclH5BsxANakOqfb57Rprc4xue9H4AuppqAlIo/i/gYtOyoDy9y8oQDmC YRiM6K5+bqUH8uC/Rjz1AASBrnb3zAasqCJlklheb5nP3+EoLpKNJ04Brk8rnlqp CMO5GktE43r9buYL9MOMXHanB523wRmNV8JjpEF+y5tPYe9YW7rsdmpjX8F8sRga PE1MMNy43lctoej4tR0iSBfK2ZNudsz7PdzW9+4Gvpc4NDMHp5O4JwhX/vPbSyyC +Or1BvPKe58zVSi5kBi7AJEeFXasfuKpwBwT9r2CGmjvI62ESmkj5M9eqcb4i8Yy i2Zr =E4NH -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 07:50:55 GMT-04:00, Gregory Maxwell gmaxw...@gmail.com wrote: (Just be glad I'm not suggesting coding the entire blockchain with an error correcting code so that it doesn't matter which subset you're holding) I forgot to ask last night: if you do that, can you add new blocks to the chain with the encoding incrementally? -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRoZ+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYudCAC7ImifMnLIFHv1UifV zRxtDkx7UxIf9dncDAcrTIyKEDhoouh0TmoZl3HKQ3KUEETAVKsMzqXLgqVe6Ezr ny1bm0pQlkBCZFRwuZvmB27Y3mwC8PD6rT9ywtWzFjWd8PEg6/UaM547nQPw7ir0 27S3XMfE/BMiQWfWnWc/nqpbmJjd8x/dM3oiTG9SVZ7iNxotxAqfnW2X5tkhJb0q dAV08wpu6aZ5hTyLpvDxXDFjEG119HJeLkT9QVIrg+GBG55PYORqE4gQr6uhrF4L fGZS2EIlbk+kAiv0EjglQfxWM7KSRegplSASiKEOuX80tqLIsEugNh1em8qvG401 NOAS =CWql -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Error correction is an interesting suggestion. If there was 1 nodes and each stored 0.1% of the blocks, at random, then the odds of a block not being stored is 45 in a million. Blocks are stored on average 10 times, so there is already reasonable redundancy. With 1 million blocks, 45 would be lost in that case, even though most are stored multiple times. With error correction codes, the chances of blocks going missing is much lower. For example, if there was 32 out of 34 Reed-Solomon-like system, then 2 blocks out of 34 could be lost without any actual data loss for the network. As a back of the envelop check, the odds of 2 missing blocks landing within 34 of another is 68/100. That means that the odds of 2 missing blocks falling in the same correction section is 45 * 34 / 100 = 0.153%. Even in that case, the missing blocks could be reconstructed, as long as you know that they are missing. The error correction code has taken it from being a near certainty that some blocks would be lost to less than 0.153%. A simple error correction system would just take 32 blocks in sequence and then compute 2 extra blocks. The extra blocks would have to be the same length as the longest block in the 32 being corrected. The shorter blocks would be padded with zeroes so everything is the same size. For each byte position in the blocks you compute the polynomial that goes through byte (x, data(x)), for x = 0 to 31. This could be a finite field, or just mod 257. You can then compute the value for x=32 and x = 33. Those are the values for the 2 extra blocks. If mod 257 is used, then only the 2 extra blocks have to deal with symbols from 0 to 256. If you have 32 of the 34 blocks, you can compute the polynomial and thus generate the 32 actual blocks. This could be achieved by a soft fork by having a commitment every 32 blocks in the coinbase. It makes the header chain much longer though. Longer sections are more efficient, but need more calculations to recover everything. You could also do interleaving to handle the case where entire sections are missing. On Thu, Apr 10, 2014 at 12:54 PM, Peter Todd p...@petertodd.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10 April 2014 07:50:55 GMT-04:00, Gregory Maxwell gmaxw...@gmail.com wrote: (Just be glad I'm not suggesting coding the entire blockchain with an error correcting code so that it doesn't matter which subset you're holding) I forgot to ask last night: if you do that, can you add new blocks to the chain with the encoding incrementally? -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRoZ+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYudCAC7ImifMnLIFHv1UifV zRxtDkx7UxIf9dncDAcrTIyKEDhoouh0TmoZl3HKQ3KUEETAVKsMzqXLgqVe6Ezr ny1bm0pQlkBCZFRwuZvmB27Y3mwC8PD6rT9ywtWzFjWd8PEg6/UaM547nQPw7ir0 27S3XMfE/BMiQWfWnWc/nqpbmJjd8x/dM3oiTG9SVZ7iNxotxAqfnW2X5tkhJb0q dAV08wpu6aZ5hTyLpvDxXDFjEG119HJeLkT9QVIrg+GBG55PYORqE4gQr6uhrF4L fGZS2EIlbk+kAiv0EjglQfxWM7KSRegplSASiKEOuX80tqLIsEugNh1em8qvG401 NOAS =CWql -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
YES Such a bitcoind is what I called border router in a previous mail. Yes, SPV wallets are getting ahead of features, so people will use them also because on size just does not fit all, but all want to ensure being on the same trunk of the chain. Therefore serious user of Bitcoin run a bitcoind as a border router and connect SPV wallets with higher functionality to that trusted node(s). This is what I think the core should focus on: Being a lightweight superfast consensus building border router and nothing more. No wallet, no GUI, no RPC calls, no Payment protocol and the rest. Regards, Tamas Blummer http://bitsofproof.com On 09.04.2014, at 17:29, Wladimir laa...@gmail.com wrote: Hello, This is primarily aimed at developers of SPV wallets. The recently reported decrease in number of full nodes could have several reasons, one of them that less people are running Bitcoin Core for the wallet because the other wallets are getting ahead in both features and useability. It's great to see innovation in wallets, but it's worrying that the number of full nodes decreases. It may be that lots of people would support the network by running a full node, but don't want to go through the trouble of installing bitcoin core separately (and get confused because it's a wallet, too). Hence I'd like to explore the idea of adding an option to popular SPV wallets, to spin a bitcoind process in the background. This could be pretty much transparent to the user - it would sync in the background, the wallet could show statistics about the node, but is not dependent on it. In exchange the user would get increased (full node level) security, as the SPV wallet would have a local trusted node. Does this sound like a good idea? Is there any way that Bitcoin Core can help to accomedate this 'embedded' usage? Specific Interfaces, special builds - maybe add a walletless bitcoind build to gitian - bindings, dlls, etc? Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
How would this affect the user in terms of disk storage? They're going to get hammered on space constraints aren't they? If it's not required how likely are users to enable this? On Wed, Apr 9, 2014 at 11:29 AM, Wladimir laa...@gmail.com wrote: Hello, This is primarily aimed at developers of SPV wallets. The recently reported decrease in number of full nodes could have several reasons, one of them that less people are running Bitcoin Core for the wallet because the other wallets are getting ahead in both features and useability. It's great to see innovation in wallets, but it's worrying that the number of full nodes decreases. It may be that lots of people would support the network by running a full node, but don't want to go through the trouble of installing bitcoin core separately (and get confused because it's a wallet, too). Hence I'd like to explore the idea of adding an option to popular SPV wallets, to spin a bitcoind process in the background. This could be pretty much transparent to the user - it would sync in the background, the wallet could show statistics about the node, but is not dependent on it. In exchange the user would get increased (full node level) security, as the SPV wallet would have a local trusted node. Does this sound like a good idea? Is there any way that Bitcoin Core can help to accomedate this 'embedded' usage? Specific Interfaces, special builds - maybe add a walletless bitcoind build to gitian - bindings, dlls, etc? Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 8:42 AM, Brian Hoffman brianchoff...@gmail.com wrote: How would this affect the user in terms of disk storage? They're going to get hammered on space constraints aren't they? If it's not required how likely are users to enable this? If Bitcoin core activates pruning a full node can be supported in— say— 4GBytes or so. (That gives enough space to store the utxo about 350MB now, and a couple gigs for blocks to serve out). I'd imagine getting information from SPV wallet developers how much disk usage agility they think is required is part of what Wladimir is looking for. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 8:41 AM, Natanael natanae...@gmail.com wrote: This could probably be done fairly easily by bundling Stratum (it's not just for pools!) and allowing SPV wallets to ask Bitcoind to start it (if you don't use it, there's no need to waste the resources), and then connect to it. The point of using Stratum is that it already is being used by Electrum, Sadly today Electrum requires more than a full node, it requires a number of large additional indexes over what a full node has and pruning is precluded. I don't think that increasing the resource utilization of the node is a good way to go there for the purposes expressed here. (not that electrum couldn't be used here, but not unmodified without the resource usage increasing route) and that it might be an easier way to support SPV clients than creating a new API in bitcoind for it since Stratum itself already relies on bitcoind to provide it's services. Bitcoin's own P2P protocol is already the API for a ordinary SPV client. So I don't believe any new API would be require, except perhaps for some process management stuff (which also isn't provided for Electrum). -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Le 09/04/2014 17:54, Gregory Maxwell a écrit : Sadly today Electrum requires more than a full node, it requires a number of large additional indexes over what a full node has and pruning is precluded. I don't think that increasing the resource utilization of the node is a good way to go there for the purposes expressed here. (not that electrum couldn't be used here, but not unmodified without the resource usage increasing route) Electrum uses two large indexes: address - utxo (patricia tree, aka ultimate blockchain compression, see thread started by Alan Reiner in the bitcointalk forum) address - spent history The first index is not going to grow larger than what bitcoind already needs to store, because bitcoind will always need to store utxos. The second index threatens to become large. However, Electrum servers do not keep the full histories, they prune older entries. Without adapting Electrum clients, it would even be possible to keep only one bit per address (to know whether that address has been used or not), and that information is only used to restore wallets from seed, not during normal operations. If the first index (patricia tree) was implemented in bitcoind, that would obviously be a big relief for electrum servers. and that it might be an easier way to support SPV clients than creating a new API in bitcoind for it since Stratum itself already relies on bitcoind to provide it's services. Bitcoin's own P2P protocol is already the API for a ordinary SPV client. So I don't believe any new API would be require, except perhaps for some process management stuff (which also isn't provided for Electrum). -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I am glad that SPV wallets are discussed outside the scope of mobile devices! Yes, SPV is a sufficient API to a trusted node to build sophisticated features not offered by the core. SPV clients of the border router will build their own archive and indices based on their interest of the chain therefore the border router core does not need to store (and process) anything not needed for consensus, its memory or disk footprint would be as low as an optimal storage of UTXO. Regards, Tamás Blummer http://bitsofproof.com On 09.04.2014, at 17:57, Gregory Maxwell gmaxw...@gmail.com wrote: On Wed, Apr 9, 2014 at 8:42 AM, Brian Hoffman brianchoff...@gmail.com wrote: How would this affect the user in terms of disk storage? They're going to get hammered on space constraints aren't they? If it's not required how likely are users to enable this? If Bitcoin core activates pruning a full node can be supported in— say— 4GBytes or so. (That gives enough space to store the utxo about 350MB now, and a couple gigs for blocks to serve out). I'd imagine getting information from SPV wallet developers how much disk usage agility they think is required is part of what Wladimir is looking for. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On 04/09/2014 09:09 AM, Tamas Blummer wrote: Yes, SPV is a sufficient API to a trusted node to build sophisticated features not offered by the core. SPV clients of the border router will build their own archive and indices based on their interest of the chain therefore the border router core does not need to store (and process) anything not needed for consensus, its memory or disk footprint would be as low as an optimal storage of UTXO. Storing zero full blocks does nothing to aid the network. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
1) It's more private. Bloom filters gives away quite accurate statistical information about what coins you own to whom ever you happen to be connected too. An attacker can easily use this to deanonymize you even if you don't reuse addresses; Tor does not help much against this attack. There is also an option to download everything, but do only a very basic surface validation (without keeping track of UTXOs). You do not need a full node for that. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 7:33 PM, Alex Mizrahi alex.mizr...@gmail.com wrote: 1) It's more private. Bloom filters gives away quite accurate statistical information about what coins you own to whom ever you happen to be connected too. An attacker can easily use this to deanonymize you even if you don't reuse addresses; Tor does not help much against this attack. There is also an option to download everything, but do only a very basic surface validation (without keeping track of UTXOs). You do not need a full node for that. You may not *need* a full node, but the point of this (which I clearly explained in my opening post) would be to support the network. Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 9 April 2014 12:27:13 GMT-04:00, Tamas Blummer ta...@bitsofproof.com wrote: A border router that is not able to serve blocks is still protecting consensus rules, that SPVs do not. If the network would only consist of SPV nodes only then e.g. a majority coalition of miner could increase their reward at will. Archives need a different solution. Any collective group that has a majority of hashing power will have no major issues running enough nodes that follow their rules to make SPV insecure anyway. There's no good reason not to have SPV security nodes distribute block chain data, particularly block headers. It helps provide redundancy in the network topology and helps provide more resources for full nodes to sync up faster. For instance in a network with a large number of partial UTXO set nodes if those nodes are forwarding block data to each other they can get enough data to become fully fledged full nodes without putting all the load on the existing full nodes. This is a good thing. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRYdYMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhdwRB/46MAw7OHwnVkLHOD0g Y4X6p1/QHgRisJIgpG2Y4nGVeAjOFleQWe4PWS4Wwdr4u0BDGPmJompiR3A99CaL MUPnxJhtdiUsomn6kI704f5pkrqslQGLzejWFb7/9WuQtvGm8RwnzIs7uAqKasni KJMn3jmqFIUcCEy9dePUdmMySCQj8qSxjGDdwZnwf8BZSdSLqd8dYiN0jQi/aA1I 2hWWyyDK9V9yQ8peAg+1dfg754Tc76lj3mEQOD39Wu3Klb9mBF3+wQCW2tJYEj2E stzeOdFZsUNUIOKFb6mo0IoUipPOvrAkfm91ais+FIwlCf+k5KcwmAUXpV45rLHm egCr =2vMf -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Block header has to be available in SPV and also in an UTXO only storing core node, so why not serve it if bandwith allows. Serving any additional information like known peer adresses or known full blocks is certainly beneficial and should be offered if at hand. Regards, Tamas Blummer http://bitsofproof.com On 09.04.2014, at 19:46, Peter Todd p...@petertodd.org wrote: Signed PGP part On 9 April 2014 12:27:13 GMT-04:00, Tamas Blummer ta...@bitsofproof.com wrote: A border router that is not able to serve blocks is still protecting consensus rules, that SPVs do not. If the network would only consist of SPV nodes only then e.g. a majority coalition of miner could increase their reward at will. Archives need a different solution. Any collective group that has a majority of hashing power will have no major issues running enough nodes that follow their rules to make SPV insecure anyway. There's no good reason not to have SPV security nodes distribute block chain data, particularly block headers. It helps provide redundancy in the network topology and helps provide more resources for full nodes to sync up faster. For instance in a network with a large number of partial UTXO set nodes if those nodes are forwarding block data to each other they can get enough data to become fully fledged full nodes without putting all the load on the existing full nodes. This is a good thing. signature.asc Description: Message signed with OpenPGP using GPGMail -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
The right way to start with this, if anyone cares, is to add instrumentation to existing SPV wallet apps to report back to home base how long they are running for, how much disk space / RAM they have, and possibly what kind of hardware. I *strongly* suspect that the vast majority of SPV wallets are not left running permanently, and run on laptops where battery life is at a premium. These people will never want to run full nodes. Sorry. I don't think it will ever make sense to run full nodes on consumer hardware again. Our time is much better spent on optimising so it's cheaper for full node operators to run them on cheap virtualised servers. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 9 April 2014 13:50:03 GMT-04:00, Tamas Blummer ta...@bitsofproof.com wrote: Block header has to be available in SPV and also in an UTXO only storing core node, so why not serve it if bandwith allows. Serving any additional information like known peer adresses or known full blocks is certainly beneficial and should be offered if at hand. Big security advantages too. For instance if an attacker hacks, say, 10℅ of hashing power the next step for them to attack SPV clients is to try to Sybil attack them so they won't find out about the longer chain. The fewer providers of block chain data there are out there the easier that attack is - just simultaneously DoS a bunch of nodes, perhaps by a low-bandwidth exploit like the bloom io or division by zero DoS attacks. This is much harder to pull off if every SPV client is passing around block headers. Similarly by passing around full blocks the attacker has a harder time knocking other miners off the network. Regardless of whether or not a miner's peers are fully validating chain data they still have the data they need to mine the next block and thus extend the longest correct chain. -BEGIN PGP SIGNATURE- Version: APG v1.1.1 iQFQBAEBCgA6BQJTRYu+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8 cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwheuQCADUyClLOLP1xpG1000l uzcfPTZuIXTpzOAmYHKs/MSb6mph/Shsu0/94eW7npQNSVeZC8wQQZ1oFQ9j1GJc SKViYJfn5yMdNvMkaWazhC0r3jxxF0AI7oy2KlnSjasfczfOQuYICJadTCwvUHrb GrKVDbgsKNzZYYKn86vF4hsLwtJN4moeqX85TYN1DC7//7hgNywA73Xt2/gdwfqe LOsD4nS7mUQObQd6TcLwXDDNEGTrdS572jdYH5sykwZjPH+wqwcm2WKTnIULsJR0 OwGUi505AKJJnLcEmZ/kGbCmKB+xJ5kExjlExtcUPrJlc+xqubhnnGCMjBiGCXSY kYCK =HXRZ -END PGP SIGNATURE- -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 8:00 PM, Mike Hearn m...@plan99.net wrote: The right way to start with this, if anyone cares, is to add instrumentation to existing SPV wallet apps to report back to home base how long they are running for, how much disk space / RAM they have, and possibly what kind of hardware. I *strongly* suspect that the vast majority of SPV wallets are not left running permanently, and run on laptops where battery life is at a premium. These people will never want to run full nodes. Bitcoins stands or falls with people running full nodes. If no one wants to volunteer resources to support the network anymore, we'll have failed. Sorry. I don't think it will ever make sense to run full nodes on consumer hardware again. Our time is much better spent on optimising so it's cheaper for full node operators to run them on cheap virtualised servers. Most consumer hardware is much more powerful than 'cheap virtualized servers'. More memory, disks are cheap, and at least in the Netherlands home bandwidth is much cheaper than server bandwidth. Also: any optimization that helps running on cheap servers will also help running it on consumer hardware. It's not the one or the other. Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On 4/9/2014 11:29 AM, Wladimir wrote: Hello, This is primarily aimed at developers of SPV wallets. The recently reported decrease in number of full nodes could have several reasons, one of them that less people are running Bitcoin Core for the wallet because the other wallets are getting ahead in both features and useability. It's great to see innovation in wallets, but it's worrying that the number of full nodes decreases. It may be that lots of people would support the network by running a full node, but don't want to go through the trouble of installing bitcoin core separately (and get confused because it's a wallet, too). Hence I'd like to explore the idea of adding an option to popular SPV wallets, to spin a bitcoind process in the background. This could be pretty much transparent to the user - it would sync in the background, the wallet could show statistics about the node, but is not dependent on it. In exchange the user would get increased (full node level) security, as the SPV wallet would have a local trusted node. Does this sound like a good idea? Is there any way that Bitcoin Core can help to accomedate this 'embedded' usage? Specific Interfaces, special builds - maybe add a walletless bitcoind build to gitian - bindings, dlls, etc? Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development I personally like the ida. Are you talking about a flag that could toggle this in the background mode or recoding for in the background use? -- Kevin -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/09/2014 06:19 PM, Wladimir wrote: If no one wants to volunteer resources to support the network anymore, we'll have failed. If the security of the network depends on a broken incentive model, then fix the design of the network so that economics works for you instead of against you. - -- Support online privacy by using email encryption whenever possible. Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTRZLiAAoJECoisBQbQ4v0RRMIAKL84ju4b9XAls9sOxQGrLMr xifQDhThCQKC4/qkOmGU8zseIdKRgXEq4auMF9/0BlgoSxsBcKynlRH2fFtYY7ol sdjCy/5dk+MBu3K1GCvNn/cuGkpIJJxrom/9riSiaPtivE5ncl7cyW5hFqf2MzRd dF6q937ocgBd8d2VuOQleQ9N5gue1+du77soxfp8oY7dXNdwk5ZrngeYxz1umsjQ lBAI/3JYkKVhU5Rte3deJcMHe6xA+neIzvcfWbOZYrkdwWE+jLSKnDkDkCOXJnuP vOI8CAteaFctV2mfgXX2CmNDWVeFsyCwoQwbnCmuE/uiM5y235PcTrsyr6U+Zzs= =oQPK -END PGP SIGNATURE- 0x1B438BF4.asc Description: application/pgp-keys -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 8:35 PM, Justus Ranvier justusranv...@gmail.comwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/09/2014 06:19 PM, Wladimir wrote: If no one wants to volunteer resources to support the network anymore, we'll have failed. If the security of the network depends on a broken incentive model, then fix the design of the network so that economics works for you instead of against you. My solution would be quick to implement and could help increase the number of nodes in the short term. It will also smooth the way towards splitting off the wallet from Bitcoin Core, by giving people a choice what wallet they use with their full node. That doesn't preclude looking for longer-term solutions that change the incentive structure, but that is much more difficult and risky. I don't believe it's a matter of 'fixing that instead' in a few hours, though I nevertheless look forward to your pull request. Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 11:35 AM, Justus Ranvier justusranv...@gmail.com wrote: If the security of the network depends on a broken incentive model, then fix the design of the network so that economics works for you instead of against you. Who says anything about a broken incentive model. You've made past claims about resource requirements that I think made no sense and then failed to defend them when they were challenge. With suitable software improvements running a full node could be done in as little as a few gigabytes in disk space (e.g. cost 25-50 cents), and as 50-100kbit/sec bandwidth in and out ongoing, and a moderate amount of ram. Power costs are already just a few cents per month. By far the greatest cost is the figuring out and setting up part, which bundling could fix. The exact resources could be tunable to what the users are willing and able to contribute. If improved marginal security and privacy in addition to supporting the network is not enough incentive to overcome costs like these then Bitcoin is already doomed. I think that fundamental costs aren't an issue at all, just implementation warts and education are. Part of asking the question is feeling out which improvements need to happen first, and what the prospects of getting the bundling going once those improvements are made. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/09/2014 06:50 PM, Gregory Maxwell wrote: Who says anything about a broken incentive model. You've made past claims about resource requirements that I think made no sense and then failed to defend them when they were challenge. Anyone reading the archives of the list will see about triple the number of people independently confirming the resource usage problem than they will see denying it, so I'm not particularly worried. - -- Support online privacy by using email encryption whenever possible. Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTRZhsAAoJECoisBQbQ4v0iD8H/A1RZyLJN6f+zt5x78CFgRdp zqZua3NBwUN2L9mI/PU1NADtxoXgkq49XusHRc+bjLu17VxGMUOsmB+AeK3VU4sN BVC1qIcWCa+OPMkgnMFrdydz4OGxX5xiJoCNsqveZIEYc8nQoCDfsn/fwqL50/Ct NfmPzh/cW7uUZ+h2bBgduGD3fzpQBlnnswAbHHX3FbBh9MvcaCfROeXXqUCedWj9 H2qVCNWTkIFJksJkqyF3f+KfedOTIyUwflx/0niBvzXcP9w4oK+WpBApHACWxiSA H3KHYNF0s2/fs+mIkabPLRX1PIUk15ln29FapoDvlHW9jzDE92x6JFnMppI4rEs= =ILks -END PGP SIGNATURE- 0x1B438BF4.asc Description: application/pgp-keys -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 6:09 PM, Thomas Voegtlin thoma...@gmx.de wrote: Le 09/04/2014 17:54, Gregory Maxwell a écrit : Sadly today Electrum requires more than a full node, it requires a number of large additional indexes over what a full node has and pruning is precluded. I don't think that increasing the resource utilization of the node is a good way to go there for the purposes expressed here. (not that electrum couldn't be used here, but not unmodified without the resource usage increasing route) Electrum uses two large indexes: address - utxo (patricia tree, aka ultimate blockchain compression, see thread started by Alan Reiner in the bitcointalk forum) Thanks for the explanation. Adding a RPC call for a address - utxo query wouldn't be a big deal. It has been requested before for other purposes as well, all the better if it helps for interaction with Electrum. Spent history would be involve a much larger index, and it's not likely that will end up in bitcoin Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I believe there're plenty bitcoind instances running, but they don't have configured port forwarding properly.There's uPNP support in bitcoind, but it works only on simple setups. Maybe there're some not yet considered way how to expose these *existing* instances to Internet, to strenghten the network. Maybe just self-test indicating the node is not reachable from outside (together with short howto like in some torrent clients). These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically? Maybe there're other ideas how to improve current situation without needs of reworking the architecture. Marek On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell gmaxw...@gmail.com wrote: On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier justusranv...@gmail.com wrote: Anyone reading the archives of the list will see about triple the number of people independently confirming the resource usage problem than they will see denying it, so I'm not particularly worried. The list has open membership, there is no particular qualification or background required to post here. Optimal use of an information source requires critical reading and understanding the limitations of the medium. Counting comments is usually not a great way to assess technical considerations on an open public forum. Doubly so because those comments were not actually talking about the same thing I am talking about. Existing implementations are inefficient in many known ways (and, no doubt, some unknown ones). This list is about developing protocol and implementations including improving their efficiency. When talking about incentives the costs you need to consider are the costs of the best realistic option. As far as I know there is no doubt from anyone technically experienced that under the current network rules full nodes can be operated with vastly less resources than current implementations use, it's just a question of the relatively modest implementation improvements. When you argue that Bitcoin doesn't have the right incentives (and thus something??) I retort that the actual resource _requirements_ are for the protocol very low. I gave specific example numbers to enable correction or clarification if I've said something wrong or controversial. Pointing out that existing implementations are not that currently as efficient as the underlying requirements and that some large number of users do not like the efficiency of existing implementations doesn't tell me anything I disagree with or didn't already know. Whats being discussed around here contributes to prioritizing improvements over the existing implementations. I hope this clarifies something. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
Another idea: Integrate torrent download of bootstrap.dat into bitcoind. Normal user (especially a beginner) won't learn how to download bootstrap separately and import it into bitcoind; he simply give up the synchronization once he realize it takes too much time. From my experience downloading the bootstrap significantly improves catching the blockchain, which may attract some more users to run bitcoind. Not sure about C++, but simple torrent client in python is like 30 lines of code (using libtorrent). Marek On Wed, Apr 9, 2014 at 10:12 PM, slush sl...@centrum.cz wrote: I believe there're plenty bitcoind instances running, but they don't have configured port forwarding properly.There's uPNP support in bitcoind, but it works only on simple setups. Maybe there're some not yet considered way how to expose these *existing* instances to Internet, to strenghten the network. Maybe just self-test indicating the node is not reachable from outside (together with short howto like in some torrent clients). These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically? Maybe there're other ideas how to improve current situation without needs of reworking the architecture. Marek On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell gmaxw...@gmail.comwrote: On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier justusranv...@gmail.com wrote: Anyone reading the archives of the list will see about triple the number of people independently confirming the resource usage problem than they will see denying it, so I'm not particularly worried. The list has open membership, there is no particular qualification or background required to post here. Optimal use of an information source requires critical reading and understanding the limitations of the medium. Counting comments is usually not a great way to assess technical considerations on an open public forum. Doubly so because those comments were not actually talking about the same thing I am talking about. Existing implementations are inefficient in many known ways (and, no doubt, some unknown ones). This list is about developing protocol and implementations including improving their efficiency. When talking about incentives the costs you need to consider are the costs of the best realistic option. As far as I know there is no doubt from anyone technically experienced that under the current network rules full nodes can be operated with vastly less resources than current implementations use, it's just a question of the relatively modest implementation improvements. When you argue that Bitcoin doesn't have the right incentives (and thus something??) I retort that the actual resource _requirements_ are for the protocol very low. I gave specific example numbers to enable correction or clarification if I've said something wrong or controversial. Pointing out that existing implementations are not that currently as efficient as the underlying requirements and that some large number of users do not like the efficiency of existing implementations doesn't tell me anything I disagree with or didn't already know. Whats being discussed around here contributes to prioritizing improvements over the existing implementations. I hope this clarifies something. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 10:12 PM, slush sl...@centrum.cz wrote: Maybe there're other ideas how to improve current situation without needs of reworking the architecture. Nothing I've proposed here would require larger changes to the architecture then were already planned. After SPV lands we are going to split off the wallet, and that will need an interface to an bitcoind to allow 'running with full node'. If that can be generalized to be useful for other (SPV) clients as well, that would be useful, hence I asked for input. It of course doesn't preclude also looking for other solutions. Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 10:31 PM, slush sl...@centrum.cz wrote: Another idea: Integrate torrent download of bootstrap.dat into bitcoind. Normal user (especially a beginner) won't learn how to download bootstrap separately and import it into bitcoind; he simply give up the synchronization once he realize it takes too much time. From my experience downloading the bootstrap significantly improves catching the blockchain, which may attract some more users to run bitcoind. Not sure about C++, but simple torrent client in python is like 30 lines of code (using libtorrent). Parallel block download would be better for that. No need to involve bittorrent. But please let's not derail this thread, this is not about other solutions for faster block download or such, let's keep it focused. Wladimir -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Apr 9, 2014, at 8:12 PM, slush sl...@centrum.cz wrote: These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically? Teredo is available by default on Microsoft systems and it's actually very common to see Teredo addresses as peers in torrents - this should work for bitcoin too, though I'm not sure if an app needs to set special flags to gain access to it.. there are probably some security settings around it. In the US, ATT/CenturyLink provide IPv6 by way of 6RD for DSL customers, Comcast has native IPv6 on residential (but not business) cable modems, and of course those who want to can always set up with a tunnel broker like Hurricane Electric - they even let you use your own IPv6 addresses. IPv6 is great, but having an application running its own tunnels would not be a good way to leverage it. Probably what's keeping a lot of them from being reachable is that most people just plug their CPE into a NAT router (without IPv6). Teredo can help here though. Putting an IPv6 checkbox in the settings with a link to an explanation might help with education and result in smarter operators who can make their nodes reachable. A built in checker would be even better - something like a checklist showing red/green for different aspects of reachability (v4/teredo/v6). -Laszlo -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
I've advocated for this in the past, and reasonable counter-arguments I was presented with are: (1) bittorrent is horribly insecure - it would be easy to DoS the initial block download if that were the goal, and (2) there's a reasonable pathway to doing this all in-protocol, so there's no reason to introduce external dependencies. On 04/09/2014 01:31 PM, slush wrote: Another idea: Integrate torrent download of bootstrap.dat into bitcoind. Normal user (especially a beginner) won't learn how to download bootstrap separately and import it into bitcoind; he simply give up the synchronization once he realize it takes too much time. From my experience downloading the bootstrap significantly improves catching the blockchain, which may attract some more users to run bitcoind. Not sure about C++, but simple torrent client in python is like 30 lines of code (using libtorrent). Marek On Wed, Apr 9, 2014 at 10:12 PM, slush sl...@centrum.cz mailto:sl...@centrum.cz wrote: I believe there're plenty bitcoind instances running, but they don't have configured port forwarding properly.There's uPNP support in bitcoind, but it works only on simple setups. Maybe there're some not yet considered way how to expose these *existing* instances to Internet, to strenghten the network. Maybe just self-test indicating the node is not reachable from outside (together with short howto like in some torrent clients). These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically? Maybe there're other ideas how to improve current situation without needs of reworking the architecture. Marek On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell gmaxw...@gmail.com mailto:gmaxw...@gmail.com wrote: On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier justusranv...@gmail.com mailto:justusranv...@gmail.com wrote: Anyone reading the archives of the list will see about triple the number of people independently confirming the resource usage problem than they will see denying it, so I'm not particularly worried. The list has open membership, there is no particular qualification or background required to post here. Optimal use of an information source requires critical reading and understanding the limitations of the medium. Counting comments is usually not a great way to assess technical considerations on an open public forum. Doubly so because those comments were not actually talking about the same thing I am talking about. Existing implementations are inefficient in many known ways (and, no doubt, some unknown ones). This list is about developing protocol and implementations including improving their efficiency. When talking about incentives the costs you need to consider are the costs of the best realistic option. As far as I know there is no doubt from anyone technically experienced that under the current network rules full nodes can be operated with vastly less resources than current implementations use, it's just a question of the relatively modest implementation improvements. When you argue that Bitcoin doesn't have the right incentives (and thus something??) I retort that the actual resource _requirements_ are for the protocol very low. I gave specific example numbers to enable correction or clarification if I've said something wrong or controversial. Pointing out that existing implementations are not that currently as efficient as the underlying requirements and that some large number of users do not like the efficiency of existing implementations doesn't tell me anything I disagree with or didn't already know. Whats being discussed around here contributes to prioritizing improvements over the existing implementations. I hope this clarifies something. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net mailto:Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets
On Wed, Apr 9, 2014 at 1:36 PM, Mark Friedenbach m...@monetize.io wrote: (2) there's a reasonable pathway to doing this all in-protocol, so there's no reason to introduce external dependencies. Not just a speculative pathway. Pieter implemented headers first: https://github.com/sipa/bitcoin/tree/headersfirst and it was everything we hoped it would be— it easily can saturate residential broadband, produces less load hot-spotting, copes with unreliable peers, etc. It's now just slowly making its way into Bitcoin core. -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
