Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Tuesday 30 April 2013 21:11:47 Jeff Garzik wrote: Hardly. The storage format is bitcoin protocol wire format, plus a tiny header. It is supported in multiple applications already, and is the most efficient storage format for bitcoin protocol blocks. Most efficient for what purpose? There is more that one might do than just duplicate bitcoind exactly. I can well imagine storing bitcoin blocks parsed and separated out into database fields. Wouldn't it be better to add support for more bitcoin-protocol-oriented HTTP requests? Then any client can supply the same interface, rather than being forced to create blk.dat on the fly? You don't have to create anything on the fly, if you store blocks in their native P2P wire protocol format. If. What if I'm writing a client and don't want to store them the way bitcoind has? This is a whole new client interface. It's fun to dream this up, but it is far outside the scope of an efficient HTTP protocol that downloads blocks. Except the alternative is no schema at all -- essentially it's just give access to a file on disk. Well, that hardly needs discussion at all, and it hardly needs the involvement of bitcoind, apache could do it right now. Your proposal is closer to a full P2P rewrite over HTTP (or a proxy thereof). I don't think it's a rewrite. The wire protocol is only a small part of what bitcoind does. Adding another thread listening for HTTP requests at the same time as on 8333 for stadnard format. Anyway -- I've obviously misunderstood what the idea behind a HTTP protocol was, and it's not like I was volunteering to do any of the work ;-) Andy -- Dr Andy Parkins andypark...@gmail.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Wednesday 01 May 2013 15:26:57 Jeff Garzik wrote: A generalized HTTP REST query protocol would be a nice addition... it is just off-topic for this thread. On IRC yesterday, we discussed an HTTP query interface like you suggested. It was agreed that it was a nice interface, and might be a nice addition to bitcoind. That is a separate topic for a separate email thread, though. As an example, see the pull request I wrote for an HTTP REST interface that downloads an encrypted wallet backup: https://github.com/bitcoin/bitcoin/pull/1982 Fair enough. I'm usually behind the state-of-the-art when I suggest things here :-) I should just trust you guys have already planned everything I might think of. Andy -- Dr Andy Parkins andypark...@gmail.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Fwd: Service bits for pruned nodes
As part of a roadmap for block downloading, I think this may be a good time to look into providing an HTTP/HTTPS protocol for block downloading - this would also allow web proxies to cache blocks and thus make it more accessible, as well as cater for resumeable downloads. -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Tue, Apr 30, 2013 at 12:14 PM, Rebroad (sourceforge) rebroad+sourceforge@gmail.com wrote: As part of a roadmap for block downloading, I think this may be a good time to look into providing an HTTP/HTTPS protocol for block downloading - this would also allow web proxies to cache blocks and thus make it more accessible, as well as cater for resumeable downloads. Speaking generally, I've always been a supporter of finding new and creative ways to store and transmit blocks. The more diversity, the less likely bitcoin can be shut down worldwide. HTTP is fine, but you run into many issues with large files. You would need a very well defined HTTP-retrievable layout, with proper HTTP headers along the entire path, if you want web caches to function properly. You need HTTP byte range support, HTTP 1.1 keep-alives, and other features for resuming large, interrupted downloads. The format currently used by bitcoind would be just fine -- blocks/blk.dat for raw data, size-limited well below 1GB. Just need to add a small metadata download, and serve the raw block files. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Tuesday 30 April 2013 19:04:59 Jeff Garzik wrote: The format currently used by bitcoind would be just fine -- blocks/blk.dat for raw data, size-limited well below 1GB. Just need to add a small metadata download, and serve the raw block files. That doesn't seem very generic. It's tied far too much to the current storage format of bitcoind. Wouldn't it be better to add support for more bitcoin-protocol-oriented HTTP requests? Then any client can supply the same interface, rather than being forced to create blk.dat on the fly? http://bitcoind.example.com/block/BBB http://bitcoind.example.com/tx/ http://bitcoind.example.com/block/oftx/TTT http://bitcoind.example.com/peers http://bitcoind.example.com/peer/nnn Essentially: block explorer's raw mode but in every bitcoind. The hardest operation for light clients is finding out the block that contains a particular transaction -- something that bitcoind already knows. I'd like to see support for HTTP POST/PUT of signed transactions and block announcements too. Andy -- Dr Andy Parkins andypark...@gmail.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Tue, Apr 30, 2013 at 3:27 PM, Andy Parkins andypark...@gmail.com wrote: On Tuesday 30 April 2013 19:04:59 Jeff Garzik wrote: The format currently used by bitcoind would be just fine -- blocks/blk.dat for raw data, size-limited well below 1GB. Just need to add a small metadata download, and serve the raw block files. That doesn't seem very generic. It's tied far too much to the current storage format of bitcoind. Hardly. The storage format is bitcoin protocol wire format, plus a tiny header. It is supported in multiple applications already, and is the most efficient storage format for bitcoin protocol blocks. Wouldn't it be better to add support for more bitcoin-protocol-oriented HTTP requests? Then any client can supply the same interface, rather than being forced to create blk.dat on the fly? You don't have to create anything on the fly, if you store blocks in their native P2P wire protocol format. http://bitcoind.example.com/block/BBB http://bitcoind.example.com/tx/ http://bitcoind.example.com/block/oftx/TTT http://bitcoind.example.com/peers http://bitcoind.example.com/peer/nnn Essentially: block explorer's raw mode but in every bitcoind. The hardest operation for light clients is finding out the block that contains a particular transaction -- something that bitcoind already knows. This is a whole new client interface. It's fun to dream this up, but it is far outside the scope of an efficient HTTP protocol that downloads blocks. Your proposal is closer to a full P2P rewrite over HTTP (or a proxy thereof). -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
And then the problem of what domain name to use - ideally a single name would be used so caches had the maximum chance to reuse content. To keep the network distributed perhaps the existing DNS seed mechanism could be used - a few names, each serving a random bitcoind's address. Put :8333 after the name, and enhance bitcoind to respond to HTTP and p2p caching would be used! Simon On Tue 30 Apr 2013 12:27:10 PM PDT, Andy Parkins wrote: On Tuesday 30 April 2013 19:04:59 Jeff Garzik wrote: The format currently used by bitcoind would be just fine -- blocks/blk.dat for raw data, size-limited well below 1GB. Just need to add a small metadata download, and serve the raw block files. That doesn't seem very generic. It's tied far too much to the current storage format of bitcoind. Wouldn't it be better to add support for more bitcoin-protocol-oriented HTTP requests? Then any client can supply the same interface, rather than being forced to create blk.dat on the fly? http://bitcoind.example.com/block/BBB http://bitcoind.example.com/tx/ http://bitcoind.example.com/block/oftx/TTT http://bitcoind.example.com/peers http://bitcoind.example.com/peer/nnn Essentially: block explorer's raw mode but in every bitcoind. The hardest operation for light clients is finding out the block that contains a particular transaction -- something that bitcoind already knows. I'd like to see support for HTTP POST/PUT of signed transactions and block announcements too. Andy -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
