Re: [Bitcoin-development] Service bits for pruned nodes
We would only end up with few copies of the historic data if users could choose what parts of the blockchain to store. Simply store chunks randomly, according to users available space, and give priority to the N most recent chunks to have more replicas in the network. You don't need bittorrent specifically for a DHT, if publicity is a problem. There are many DHT proposals and implementations, and i bet one of them should be more suitable to the bitcoin network than bittorrent's. On Sun, Apr 28, 2013 at 9:29 AM, Mike Hearn mike@... wrote: I'd imagined that nodes would be able to pick their own ranges to keep rather than have fixed chosen intervals. Everything or two weeks is rather X most recent is special for two reasons: It meshes well with actual demand, and the data is required for reorganization. So whatever we do for historic data, N most recent should be treated specially. But I also agree that its important that everything be splittable into ranges because otherwise when having to choose between serving historic data and— say— 40 GB storage, a great many are going to choose not to serve historic data... and so nodes may be willing to contribute 4-39 GB storage to the network there will be no good way for them to do so and we may end up with too few copies of the historic data available. As can be seen in the graph, once you get past the most recent 4000 blocks the probability is fairly uniform... so N most recent is not a good way to divide load for the older blocks. But simple ranges— perhaps quantized to groups of 100 or 1000 blocks or something— would work fine. This doesn't have to come in the first cut, however— and it needs new addr messages in any case. -- AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Thu, May 16, 2013 at 7:26 AM, Ricardo Filipe ricardojdfil...@gmail.com wrote: We would only end up with few copies of the historic data if users could choose what parts of the blockchain to store. Simply store chunks randomly, according to users available space, and give priority to the N most recent chunks to have more replicas in the network. You don't need bittorrent specifically for a DHT, if publicity is a problem. There are many DHT proposals and implementations, and i bet one of them should be more suitable to the bitcoin network than bittorrent's. That's just about the worst thing you could do for bitcoin. DoS one part of the DHT, you DoS the entire blockchain by breaking the chain. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
You are welcome to optimise P2P addr broadcasts or develop better bootstrap mechanisms. On Sun, May 5, 2013 at 3:12 PM, John Dillon john.dillon...@googlemail.comwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Sorry I should have used the word bootstrapping there rather than discovery. But again I think that shows my point clearly. Centralized methods like DNS should be used for as little as possible, just simple initial bootstrapping, and focus the development efforts towards the non-centralized peer discovery mechanisms. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQEcBAEBCAAGBQJRhlpyAAoJEEWCsU4mNhiP+NwH/3RY5vBpSYkwKgTmdKHRc/gw BJCSV/1MEDECgBTxaRYSzYZyargjsdG50KaIaCq8M1+8DWkBEkH8JFif7UYMlZGM WROMP6UjAnP1fJ3B2JChdMgRv1HdXJQDQVcO8UnSJschhX8lZZiUySbaqIPuRuV/ lI7/JkUZvmnms4+HGiaqwfbPO0k6ytJNKxORrk4TzFnThh4dy9WytElc8JHZOFaQ ly159X5JuEwh8DLOoUtPhaR6tJaJbJLBEt+QJiGnSktPsJCE8p9+4HQ0kMCQr3Ha 05EHTZEw+TqEPaA7vFLgA/9tWjK9s1Y6sqLOAYiLp/0wSKzCkBO0C5LWFHsJ/XQ= =aCgi -END PGP SIGNATURE- -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Mon, May 06, 2013 at 10:19:35AM +0200, Mike Hearn wrote: You are welcome to optimise P2P addr broadcasts or develop better bootstrap mechanisms. I think John's actually has a point here. If we're judging the quality of a protocol change by how compatible it is with DNS seeding, then we're clearly not using DNS seeding as seeding anymore (=getting an entry point into the P2P network), but as a mechanism for choosing (all) peers. Eventually, I think it makes sense to move to a system where you get seeds from a DNS (or other mechanism), connect to one or a few of the results, do a getaddr, fill your peer IP database with it, and disconnect from the DNS seeded peer. This probably means we need to look at ways to optimize current peer exchange, but that's certainly welcome in any case. -- Pieter -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I think you too should ask yourself why you are putting so much effort into optimizing a centralized service, the DNS seeds, rather than putting effort into optimizing the P2P peer discovery instead. DNS seeds are a necessary evil, one that shouldn't be promoted with additional features beyond simply obtaining your initial set of peers. After all Peter, just like you have implemented alternate block header distribution over twitter, in the future we should have many different means of peer discovery. Right now we have DNS seeds, a fixed list, and IRC discovery that does not work because the servers it was pointed too no longer exist. Not a good place to be. Some random ideas: search engines - search for bitcoin seed address or something and try IP's found (twitter is similar) ipv4 scanning - not exactly friendly, but the density of bitcoin nodes is probably getting to the point where a brute force search is feasible anycast peers - would work best with UDP probably, who has the resources to set this up? It is probably not worth the effort implementing the above immediately, but it is worth the effort to ensure that we don't make the DNS seed system so complex and sophisticated that we depend on it. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQEcBAEBCAAGBQJRhU44AAoJEEWCsU4mNhiPtssH/1yb/FZRRaZpr3CwkoaOhbhu pxfRNWgOEOL/mlWKTVgp2812qEnY9DySpJ5DJMjx7/GhSvOtnteza5ts4+pbuWhd l6E1R9zAYxX+VOiBxcBtoZNEXDcS+CjMumuBH5S1v+L5jEntOWS9G8DKasjD2WAQ DzX8YbOuzIOqasEbr5Hpr9Vfl7ZtW/+q/sPhQ1q3a7n7MaaIZrZicisJw3z7T7+0 T0yK8vUdYfstTjs0zLzfI5PW9+TG5T0kvj0kXSCjnK723Mfl7SXp6UZx6yebBi6q tcTVOPo4hfBWk8XryZxaSNCkDYY6kryy5cb2V+BojVfqLWVKgR3pdZqXqnEKNLo= =0XFF -END PGP SIGNATURE- -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Sat, May 4, 2013 at 2:07 PM, John Dillon john.dillon...@googlemail.com wrote: After all Peter, just like you have implemented alternate block header distribution over twitter, in the future we should have many different means of peer discovery. Right now we have DNS seeds, a fixed list, and IRC discovery that does not work because the servers it was pointed too no longer exist. Not a good place to be. Let's not confuse bootstrapping with overall peer discovery. Peer exchange between P2P nodes is the primary and best method of obtaining free peers. Obviously you need to bootstrap into that, though. DNS seed and fixed list are those bootstrap methods (IRC code was deleted), but are only used to limp along until you can contact a real P2P node, at which point peer discovery truly begins. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
(generic comment on the discussion that spawned off: ideas about how to allow additional protocols for block exchange are certainly interesting, and in the long term we should certainly consider that. For now I'd like to keep this about the more immediate way forward with making the P2P protocol not break in the presence of pruning nodes) On Sun, Apr 28, 2013 at 6:57 PM, Mike Hearn m...@plan99.net wrote: That's true. It can be perhaps be represented as I keep the last N blocks and then most likely for any given node the policy doesn't change all that fast, so if you know the best chain height you can calculate which nodes have what. Yes, I like that better than broadcasting the exact height starting at which you serve (though I would put that information immediately in the version announcement). I don't think we can rely on the addr broadcasting mechanism for fast information exchange anyway. One more problem with this: DNS seeds cannot convey this information (neither do they currently convey service bits, but at least those can be indexed separately, and served explicitly through asking for a specific subdomain or so). So to summarize: * Add a field to addr messages (after protocol number increase) that maintains number of top blocks served)? * Add a field to version message to announce the actual first block served? * Add service bits to separately enable relaying/verifying node and serves (part of) the historic chain? My original reason for suggesting this was different, I think better compatibility with DNS seeds may be a good reason for this. You could ask the seed first for a subset that at least serves some part of the historic chain, until you hit a node that has enough, and once caught up, ask for nodes that relay. Disconnecting in case something is requested that isn't served seems like an acceptable behaviour, yes. A specific message indicating data is pruned may be more flexible, but more complex to handle too. Well, old nodes would ignore it and new nodes wouldn't need it? I'm sure there will be cases where a new node connects based on outdated information. I'm just stating that I agree with the generic policy of if a node requests something it should have known the peer doesn't serve, it is fair to be disconnected. The reason for splitting them is that I think over time these may be handled by different implementations. You could have stupid storage/bandwidth nodes that just keep the blockchain around, and others that validate it. Even if that doesn't happen implementation-wise, I think these are sufficiently independent functions to start thinking about them as such. Maybe so, with a last N blocks in addr messages though such nodes could just set their advertised history to zero and not have to deal with serving blocks to nodes. If you have a node that serves the chain but doesn't validate it, how does it know what the best chain is? Just whatever the hardest is? Maybe it validates, maybe it doesn't. What matters is that it doesn't guarantee relaying fresh blocks and transactions. Maybe it does validate, maybe it just stores any blocks, and uses a validating node to know what to announce as best chain, or it uses an SPV mechanism to determine that. Or it only validates and relays blocks, but not transactions. My point is that serving historic data and relaying fresh data are separate responsibilities, and there's no need to require them to be combined. -- Pieter -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
Yes, I like that better than broadcasting the exact height starting at which you serve (though I would put that information immediately in the version announcement). I don't think we can rely on the addr broadcasting mechanism for fast information exchange anyway. One more problem with this: DNS seeds cannot convey this information (neither do they currently convey service bits, but at least those can be indexed separately, and served explicitly through asking for a specific subdomain or so). That's true, but we can extend the DNS seeding protocol a little bit - you could query current-chain-height.dnsseed.whatever.com and the DNS server then only returns nodes it knows matches your requirement. This might complicate existing seeds a bit, and it's a bit of a hack, but protocol-wise it's still possible. Of course if you want to add more dimensions it gets uglier fast. -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Fri, May 03, 2013 at 04:06:29PM +0200, Mike Hearn wrote: That's true, but we can extend the DNS seeding protocol a little bit - you could query current-chain-height.dnsseed.whatever.com and the DNS server then only returns nodes it knows matches your requirement. If you're going to take a step like that, the current-chain-height should be rounded off, perhaps to some number of bits, or you'll allow DNS caching to be defeated. Make clients check for the largest rounded off value first, and then drill down if required. Some complexity involved... This might complicate existing seeds a bit, and it's a bit of a hack, but protocol-wise it's still possible. Of course if you want to add more dimensions it gets uglier fast. Maybe I should make my blockheaders-over-dns thing production worthy first so we can see how many ISP's come at us with pitchforks? :P -- 'peter'[:-1]@petertodd.org 00142de0244ee8fac516e7c0a29da1eafc0d43f2da8b6388b387 signature.asc Description: Digital signature -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
If you're going to take a step like that, the current-chain-height should be rounded off, perhaps to some number of bits, or you'll allow DNS caching to be defeated. Don't the seeds already set small times? I'm not sure we want these responses to be cacheable, otherwise there's a risk of a wall of traffic suddenly showing up at one set of nodes if a large ISP caches a response. (yes yes, I know, SPV node should be remembering addr broadcasts and such). -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Fri, May 03, 2013 at 05:02:26PM +0200, Mike Hearn wrote: If you're going to take a step like that, the current-chain-height should be rounded off, perhaps to some number of bits, or you'll allow DNS caching to be defeated. Don't the seeds already set small times? I'm not sure we want these responses to be cacheable, otherwise there's a risk of a wall of traffic suddenly showing up at one set of nodes if a large ISP caches a response. (yes yes, I know, SPV node should be remembering addr broadcasts and such). Hmm, on second thought you're probably right for the standard case where it's really P2P. On the other hand it kinda limits us in the future if seeds have high-bandwidth nodes they can just point clients too, but maybe just assuming the DNS seed might need high bandwidth as well is acceptable. I dunno, given how badly behaved a lot of ISP dns servers are re: caching, maybe we're better off keeping it simple. -- 'peter'[:-1]@petertodd.org 013bfdf35da40a40c35ccd75e09652ae541d94d26130a695f757 signature.asc Description: Digital signature -- Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Sun, Apr 28, 2013 at 11:51 AM, Pieter Wuille pieter.wui...@gmail.com wrote: Hello all, I think it is time to move forward with pruning nodes, i.e. nodes that fully validate and relay blocks and transactions, but which do not keep (all) historic blocks around, and thus cannot be queried for these. The biggest roadblock is making sure new and old nodes that start up are able to find nodes to synchronize from. To help them find peers, I would like to propose adding two extra service bits to the P2P protocol: * NODE_VALIDATE: relay and validate blocks and transactions, but is only guaranteed to answer getdata requests for (recently) relayed blocks and transactions, and mempool transactions. * NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without guarantee for relaying/validating new blocks and transactions. * NODE_NETWORK (which existed before) will imply NODE_VALIDATE and guarantee availability of all historic blocks. In general, I support this, as anybody on IRC knows. Though it does seem to open the question about snapshotting. Personally, it seems important to enable running a fully validating node, that may bootstrap from a UTXO snapshot + all blocks since that snapshot. NODE_BLOCKS_2016, in particular, is too short. For users, I've seen plenty of use cases in the field where you start a network sync after a 2-week period. Set a regular interval for creating a UTXO snapshot, say 3 months (6*2016 blocks), and serve all blocks after that snapshot. For older nodes, they would contact an archive node or torrent for 3 month blocks, and then download normally = 3 month blocks (if the archive node didn't serve up to present day). Where are we on nailing down a stable, hash-able UTXO serialization? -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
Sounds like this part of Bitcoin (block sharing) would definitely benefit from having a REST (HTTP) API. REST-based web APIs are a common feature of most online services these days. Makes writing other client services so much easier. Plus you get the benefit of the HTTP ecosystem for free (HTTP caches, etc). - Brenton Camac On Apr 30, 2013, at 1:04 PM, Jeff Garzik jgar...@exmulti.com wrote: On Tue, Apr 30, 2013 at 12:14 PM, Rebroad (sourceforge) rebroad+sourceforge@gmail.com wrote: As part of a roadmap for block downloading, I think this may be a good time to look into providing an HTTP/HTTPS protocol for block downloading - this would also allow web proxies to cache blocks and thus make it more accessible, as well as cater for resumeable downloads. Speaking generally, I've always been a supporter of finding new and creative ways to store and transmit blocks. The more diversity, the less likely bitcoin can be shut down worldwide. HTTP is fine, but you run into many issues with large files. You would need a very well defined HTTP-retrievable layout, with proper HTTP headers along the entire path, if you want web caches to function properly. You need HTTP byte range support, HTTP 1.1 keep-alives, and other features for resuming large, interrupted downloads. The format currently used by bitcoind would be just fine -- blocks/blk.dat for raw data, size-limited well below 1GB. Just need to add a small metadata download, and serve the raw block files. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On 4/28/2013 8:55 PM, Peter Todd wrote: On Mon, Apr 29, 2013 at 03:48:18AM +, John Dillon wrote: We can build this stuff incrementally I'll agree. It won't be the case that one in a thousand nodes serve up the part of the chain you need overnight. So many I am over engineering the solution with BitTorrent. I think that pretty much sums it up. With the block-range served in the anounce message you just need to find an annoucement with the right range, and at worst connect to a few more node to get what you need. One of the technologies that can be borrowed from Bittorrent (besides downloading from multiple peers at once) is analysis by clients of the part distribution, which allows a client to download and share the least-propagated parts first to maintain high availability of the whole file, even when not one individual currently has downloaded the complete file (the seed has left the swarm). Unlike Bittorrent, a partial-blockchain swarm client needs to make informed decisions about how much to download, such as rules like until it sees at least 20 complete blockchain-equivalents in the swarm, until it has 10% of the blockchain itself, work backwards, all blocks from the hash tree required to verify my payments or other minimums that might all be criteria. Bittorrent only considers directly connected peers' piecemaps when making decisions of what to download. Bitcoin, however, already has a protocol to allow peer discovery beyond the connected nodes; this could be extended to communicate what parts the peer is hosting. Careful thought into attack vectors would need to be paid in design, so that only a majority of outbound-connected peers's advertisement are able to inform consensus about part or peer availability, messages able to remove a peer or part availability from other's lists are confirmed independently without such removal verification generating DDOS traffic amplification, lying clients can be marked as discovered by the majority, etc. Such thought doesn't have to be paid if directly implementing Bittorrent, but it has the burden of centralized trackers or expensive DHT, and it also doesn't have any logic informing it besides don't quit until I get the whole file. -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
I'd imagined that nodes would be able to pick their own ranges to keep rather than have fixed chosen intervals. Everything or two weeks is rather restrictive - presumably node operators are constrained by physical disk space, which means the quantity of blocks they would want to keep can vary with sizes of blocks, cost of storage, etc. Adding new fields to the addr message and relaying those fields to newer nodes means every node could advertise the height at which it pruned. I know it means a longer time before the data is available everywhere vs service bits, but it seems like most nodes won't be pruning right away anyway. There's plenty of time for upgrades. If an old node connected to a new node and getdata-d blocks that had been pruned, immediate disconnection should make the old node go find a different one. It means the combination of old node+not run for a long time might take a while before it can find a node that has what it wants, but that doesn't seem like a big deal. What is the use case for NODE_VALIDATE? Nodes that throw away blocks almost immediately? Why would a node do that? On Sun, Apr 28, 2013 at 5:51 PM, Pieter Wuille pieter.wui...@gmail.comwrote: Hello all, I think it is time to move forward with pruning nodes, i.e. nodes that fully validate and relay blocks and transactions, but which do not keep (all) historic blocks around, and thus cannot be queried for these. The biggest roadblock is making sure new and old nodes that start up are able to find nodes to synchronize from. To help them find peers, I would like to propose adding two extra service bits to the P2P protocol: * NODE_VALIDATE: relay and validate blocks and transactions, but is only guaranteed to answer getdata requests for (recently) relayed blocks and transactions, and mempool transactions. * NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without guarantee for relaying/validating new blocks and transactions. * NODE_NETWORK (which existed before) will imply NODE_VALIDATE and guarantee availability of all historic blocks. The idea is to separate the different responsibilities of network nodes into separate bits, so they can - at some point - be implemented independently. Perhaps we want more than just one degree (2016 blocks), maybe also 144 or 21, but those can be added later if necessary. I monitored the frequency of block depths requested from my public node, and got this frequency distribution: http://bitcoin.sipa.be/depth-small.png so it seems 2016 nicely matches the set of frequently-requested blocks (indicating that few nodes are offline for more than 2 weeks consecutively. I'll write a BIP to formalize this, but wanted to get an idea of how much support there is for a change like this. Cheers, -- Pieter -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Sun, Apr 28, 2013 at 6:29 PM, Mike Hearn m...@plan99.net wrote: I'd imagined that nodes would be able to pick their own ranges to keep rather than have fixed chosen intervals. Everything or two weeks is rather restrictive - presumably node operators are constrained by physical disk space, which means the quantity of blocks they would want to keep can vary with sizes of blocks, cost of storage, etc. Sure, that's why eventually several levels may be useful. Adding new fields to the addr message and relaying those fields to newer nodes means every node could advertise the height at which it pruned. I know it means a longer time before the data is available everywhere vs service bits, but it seems like most nodes won't be pruning right away anyway. There's plenty of time for upgrades. That's a more flexible model, indeed. I'm not sure how important speed of propagation will be though - it may be very slow, given that there are 10s of IPs circulating, and only a few are relayed in one go between nodes. Even then, I'd like to see the relay/validation responsibility split off from the serve historic data one, and have separate service bits for those. If an old node connected to a new node and getdata-d blocks that had been pruned, immediate disconnection should make the old node go find a different one. It means the combination of old node+not run for a long time might take a while before it can find a node that has what it wants, but that doesn't seem like a big deal. Disconnecting in case something is requested that isn't served seems like an acceptable behaviour, yes. A specific message indicating data is pruned may be more flexible, but more complex to handle too. What is the use case for NODE_VALIDATE? Nodes that throw away blocks almost immediately? Why would a node do that? NODE_VALIDATE doesn't say anything about which blocks are available, it just means it relays and validates (and thus is not an SPV node). It can be combined with NODE_BLOCKS_2016 if those blocks are also served. The reason for splitting them is that I think over time these may be handled by different implementations. You could have stupid storage/bandwidth nodes that just keep the blockchain around, and others that validate it. Even if that doesn't happen implementation-wise, I think these are sufficiently independent functions to start thinking about them as such. -- Pieter -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Sun, Apr 28, 2013 at 9:29 AM, Mike Hearn m...@plan99.net wrote: I'd imagined that nodes would be able to pick their own ranges to keep rather than have fixed chosen intervals. Everything or two weeks is rather X most recent is special for two reasons: It meshes well with actual demand, and the data is required for reorganization. So whatever we do for historic data, N most recent should be treated specially. But I also agree that its important that everything be splittable into ranges because otherwise when having to choose between serving historic data and— say— 40 GB storage, a great many are going to choose not to serve historic data... and so nodes may be willing to contribute 4-39 GB storage to the network there will be no good way for them to do so and we may end up with too few copies of the historic data available. As can be seen in the graph, once you get past the most recent 4000 blocks the probability is fairly uniform... so N most recent is not a good way to divide load for the older blocks. But simple ranges— perhaps quantized to groups of 100 or 1000 blocks or something— would work fine. This doesn't have to come in the first cut, however— and it needs new addr messages in any case. -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Sun, Apr 28, 2013 at 7:57 PM, John Dillon john.dillon...@googlemail.com wrote: Have we considered just leaving that problem to a different protocol such as BitTorrent? Offering up a few GB of storage capacity is a nice idea but it means we would soon have to add structure to the network to allow nodes to find each other to actually get that data. BitTorrent already has that issue thought through carefully with it's DHT support. I think this is not a great idea on a couple levels— Least importantly, our own experience with tracker-less torrents on the bootstrap files that they don't work very well in practice— and thats without someone trying to DOS attack it. More importantly, I think it's very important that the process of offering up more storage not take any more steps. The software could have user overridable defaults based on free disk space to make contributing painless. This isn't possible if it takes extra software, requires opening additional ports.. etc. Also means that someone would have to be constantly creating new torrents, there would be issues with people only seeding the old ones, etc. It's also the case that bittorrent is blocked on many networks and is confused with illicit copying. We would have the same problems with that that we had with IRC being confused with botnets. We already have to worry about nodes finding each other just for basic operation. The only addition this requires is being able to advertise what parts of the chain they have. What are the logistics of either integrating a DHT capable BitTorrent client, or just calling out to some library? We could still use the Bitcoin network to bootstrap the BitTorrent DHT. Using Bitcoin to bootstrap the Bittorrent DHT would probably make it more reliable, but then again it might cause commercial services that are in the business of poisoning the bittorrent DHT to target the Bitcoin network. Integration also brings up the question of network exposed attack surface. Seems like it would be more work than just adding the ability to add ranges to address messages. I think we already want to revise the address message format in order to have signed flags and to support I2P peers. -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
While I like the idea of a client using a DHT blockchain or UTXO list, I don't think that the reference client is the place for it. But it would make for a very interesting experimental project! On 29 April 2013 13:36, Gregory Maxwell gmaxw...@gmail.com wrote: On Sun, Apr 28, 2013 at 7:57 PM, John Dillon john.dillon...@googlemail.com wrote: Have we considered just leaving that problem to a different protocol such as BitTorrent? Offering up a few GB of storage capacity is a nice idea but it means we would soon have to add structure to the network to allow nodes to find each other to actually get that data. BitTorrent already has that issue thought through carefully with it's DHT support. I think this is not a great idea on a couple levels— Least importantly, our own experience with tracker-less torrents on the bootstrap files that they don't work very well in practice— and thats without someone trying to DOS attack it. More importantly, I think it's very important that the process of offering up more storage not take any more steps. The software could have user overridable defaults based on free disk space to make contributing painless. This isn't possible if it takes extra software, requires opening additional ports.. etc. Also means that someone would have to be constantly creating new torrents, there would be issues with people only seeding the old ones, etc. It's also the case that bittorrent is blocked on many networks and is confused with illicit copying. We would have the same problems with that that we had with IRC being confused with botnets. We already have to worry about nodes finding each other just for basic operation. The only addition this requires is being able to advertise what parts of the chain they have. What are the logistics of either integrating a DHT capable BitTorrent client, or just calling out to some library? We could still use the Bitcoin network to bootstrap the BitTorrent DHT. Using Bitcoin to bootstrap the Bittorrent DHT would probably make it more reliable, but then again it might cause commercial services that are in the business of poisoning the bittorrent DHT to target the Bitcoin network. Integration also brings up the question of network exposed attack surface. Seems like it would be more work than just adding the ability to add ranges to address messages. I think we already want to revise the address message format in order to have signed flags and to support I2P peers. -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mon, Apr 29, 2013 at 3:36 AM, Gregory Maxwell gmaxw...@gmail.com wrote: On Sun, Apr 28, 2013 at 7:57 PM, John Dillon john.dillon...@googlemail.com wrote: Have we considered just leaving that problem to a different protocol such as BitTorrent? Offering up a few GB of storage capacity is a nice idea but it means we would soon have to add structure to the network to allow nodes to find each other to actually get that data. BitTorrent already has that issue thought through carefully with it's DHT support. I think this is not a great idea on a couple levels— Least importantly, our own experience with tracker-less torrents on the bootstrap files that they don't work very well in practice— and thats without someone trying to DOS attack it. Unfortunate. What makes them not work out? DHT torrents seem pretty popular. More importantly, I think it's very important that the process of offering up more storage not take any more steps. The software could have user overridable defaults based on free disk space to make contributing painless. This isn't possible if it takes extra software, requires opening additional ports.. etc. Also means that someone would have to be constantly creating new torrents, there would be issues with people only seeding the old ones, etc. Now don't get me wrong, I'm not proposing we do this if it requires additional steps or other software. I only mean if it is possible in an easy way to integrate the BitTorrent technology into Bitcoin in an automatic fashion. Yes part of that may have to be finding a way to re-use the existing port for instance. We already have to worry about nodes finding each other just for basic operation. The only addition this requires is being able to advertise what parts of the chain they have. Sure I guess my concern is more how do you find the specific part of the chian you need without some structure to the network? Although I guess it may be enough to just add that structure or depend on just walking the nodes advertising themselves until you find what you want. We can build this stuff incrementally I'll agree. It won't be the case that one in a thousand nodes serve up the part of the chain you need overnight. So many I am over engineering the solution with BitTorrent. Using Bitcoin to bootstrap the Bittorrent DHT would probably make it more reliable, but then again it might cause commercial services that are in the business of poisoning the bittorrent DHT to target the Bitcoin network. Good point. Sadly one that may apply to the Tor network too in the future. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQEcBAEBCAAGBQJRfe1LAAoJEEWCsU4mNhiPuDgIAM1zz+ohlHgz37RgToQhInRc 1tv4Fnb6uGWyb4+U6UpK24LlXMFvOJsLm2czgbBc1Iz4z4wvb1m5IGw0ubJuV4mT GPUJhM4sNqfeKZlSWRw4Gia6Vk1jTkue+uVYvZn2vBS4SS6vYhYCC3zXIITyb2mp 7CVjcM84bTHKxIaMW1rIgmVJmfslsFdeNOp/cDVvkNl9+WvzWPeJ32BkT522p+pT AcPVFMsEJirYrXYi8HwdtGSeiG+mv0IemTAObJNPRrpw3x04ja6qecqzM51AkQ4t hPems5ShXM9FyDKFQNmtoC6ULpbd3CBBjsiQj0pp55epy6UC0eiUIXP8L9v0giM= =AOj8 -END PGP SIGNATURE- -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Service bits for pruned nodes
On Mon, Apr 29, 2013 at 03:48:18AM +, John Dillon wrote: We can build this stuff incrementally I'll agree. It won't be the case that one in a thousand nodes serve up the part of the chain you need overnight. So many I am over engineering the solution with BitTorrent. I think that pretty much sums it up. With the block-range served in the anounce message you just need to find an annoucement with the right range, and at worst connect to a few more node to get what you need. It will be a long time before the bandwidth used for finding a node with the part of the chain that you need is a significant fraction of the load required for downloading the data itself. Remember that BitTorrent's DHT is a system giving you access to tens of petabytes worth of data. The Bitcoin blockchain on the other hand simply can't grow more than 57GiB per year. It's a cute idea though. Also, while we're talking about the initial download: http://blockchainbymail.com Lots of options out there. -- 'peter'[:-1]@petertodd.org signature.asc Description: Digital signature -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
