[Bitcoin-development] BIP: Custom Services
Hey everyone, I was working on some custom protocol extensions for Bitcoin that I wanted to experiment with and I noticed that in order to enable nodes to announce these services the only mechanism the protocol currently provides is to use one of the 64 bits of the services field. This is obviously a resource that will run out quickly if we all just help ourselves, so I set out to come up with a standardized way to announce custom protocol extensions, without using up NODE_* flags. Please kindly review my solution: https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Services Thanks to Amir Taaki, Mike Hearn and Pieter Wuille who provided invaluable feedback while writing the draft. Note: Earlier drafts of this BIP contained a description of a mechanism for peer exchange for these custom services. However, since that part of the BIP was (1) just a recommendation and (2) rather complex, Amir and I agreed to split it off into a separate BIP [1] that will be refined some more and submitted later. Cheers, Stefan [1] https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Service_Discovery -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP: Custom Services
I think it's pretty reasonable, although people will want to use node flags to get into the addr broadcasts anyway. That said, I suspect (based on previous discussions) that there would be quite some pushback against putting extra functionality into the core Bitcoin network. Most likely people will re-use the code with different peer discovery seeds and bootstrap similar but unrelated P2P networks for doing new applications. For instance, what if we want to do the language translation app I've talked about a few times before? You need a way to floodfill broadcast invalid transactions to interested parties. The pubsub mechanism in the Bitcoin protocol was an interesting way to do that, but I think it got removed. To broadcast to interested nodes now, you'd have to find them via addr broadcasts and then connect directly. And if you're going to do that, you may as well just form an entirely independent network. More elaboration of the use cases might therefore be useful. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP: Custom Services
On Mon, Aug 13, 2012 at 3:41 AM, Stefan Thomas m...@justmoon.de wrote: I was working on some custom protocol extensions for Bitcoin that I wanted to experiment with and I noticed that in order to enable nodes to announce these services the only mechanism the protocol currently provides is to use one of the 64 bits of the services field. This is obviously a resource that will run out quickly if we all just help ourselves, so I set out to come up with a standardized way to announce custom protocol extensions, without using up NODE_* flags. Please kindly review my solution: https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Services heh, this is not a new idea. I even implemented a pull request for service discovery myself, which simply consisted of querying the list of supported commands: https://github.com/bitcoin/bitcoin/pull/1471 On IRC, I proposed several alternatives including modifying 'version' (which you did) and a new getcaps (get capabilities) command to be added in protocol_version X. gmaxwell seems continually unenthused, and made a valid point about service advertisement: these capabilities are not advertised with CAddress, so how does one usefully discover and make use of them? What are real world use cases, that cannot be solved with nService bits? My only response is a weak one: inevitability. It seems likely that -somebody- will implement their own P2P commands for their own client subset, even if only a simple use 'getstatus' with strSubVer matching /FooClient/ Therefore, if it is inevitable, we might as well make some basic rules about how to extended your P2P command set. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP: Custom Services
On Mon, Aug 13, 2012 at 10:24 AM, Jeff Garzik jgar...@exmulti.com wrote: My only response is a weak one: inevitability. It seems likely that -somebody- will implement their own P2P commands for their own client subset, even if only a simple use 'getstatus' with strSubVer matching /FooClient/ Therefore, if it is inevitable, we might as well make some basic rules about how to extended your P2P command set. I'm not opposed to that logic. But for cases where an introduction mechanism will be needed... it would be awfully good to have one, and I do think that there is harm in making people think that simple services negotiation will actually work for their needs for cases where a separate p2p network is needed. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Protocol changes for SPV clients: mempool, getdata commands
A small change to the protocol is proposed for satoshi client v0.7 (upcoming release): Add 'mempool' P2P command, and extend 'getdata' behavior https://github.com/bitcoin/bitcoin/pull/1641 The genesis of this command is to permit SPV clients to access the memory pool, but secondary uses include diagnostics and miner download (so that miners won't miss lucrative TX's due to node restart). getdata behavior is extended to include any memory pool transaction, relaxing a previous getdata restriction. Didn't feel this warranted a BIP, but can write one if people really want one. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP: Custom Services
More elaboration of the use cases might therefore be useful. I'm implementing a DHT, rather than storing the whole blockchain index locally, a future version of BitcoinJS will store only a user defined percentage (anywhere from 0-100%.) Any misses will be resolved by querying the network. Thanks to ultraprune, we no longer need a full index for verification. And for all the other use cases like lightweight server queries and block explorer queries a bit of latency is acceptable in exchange for scalability and decentralization. This feature will give people to option to run anywhere from a large BitcoinJS node (100% index) to a small one (20% index) to a lightweight one (0% index.) All of them are equally queryable, so if you're just trying out BitcoinJS you won't have to download the block chain just to run the block explorer example. Only when your block explorer's traffic grows will you need to contribute some query services back to the network in order not to get rate-limited. these capabilities are not advertised with CAddress, so how does one usefully discover and make use of them? Pieter brought up this very point when he reviewed an earlier draft. This prompted the creation of the second BIP I mentioned: https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Service_Discovery The basic principle is quite simple - prefix the standard addr message with a service-specific message to mark off service support. It's easy to implement and very efficient (without compression it's 1.288 bits per node, with compression it's quite possibly more efficient than the services field.) Because this stuff is a bit more complex and because it requires no changes to the Bitcoin protocol, Amir and I chose to split it off into a separate BIP and I want to work on it a little more. But if you are wondering how peer exchange might work for these custom services, please do take a look at it. you may as well just form an entirely independent network When I designed the DHT, I did just that. Later I was working on a concept for a decentralized pool and I noticed I was solving the same problems again. And with all three services running a node might be maintaining three separate TCP connections to the exact same peer. So then I considered making the DHT network extensible, so that the decentralized pool protocol could live in there. But, well if I'm doing that, why not just make the regular Bitcoin protocol extensible and let both extensions live in there. For a custom service you DO need the following: - service-specific DNS seeds - service announcement (BIP: CS) - service-specific messages (BIP: CS) - service-specific peer exchange (BIP: CSD) And those four things cover a lot of what Bitcoin does. But once you're thinking about n custom services it starts to look easier to add semantics for some nodes support some things in one network rather than instantiating n networks. On the opposite end of the spectrum there are very simple services. Consider a WebSocket transport. Some nodes might offer Bitcoin-over-WebSocket, for example to implement an SPV client in the browser. But they don't connect to each other via WebSocket, since they prefer plain TCP. So in this case you need peer exchange, but not much else. You could create a new P2P network for the sole purpose of exchanging peers, but again it seems much easier if there were ways to do this on the Bitcoin network. One final point: A major focus of this BIP is to make it easy to canonicalize custom services if we choose to do so. The idea is that custom services get to prove themselves in the wild - those that work well may be added to the standard protocol. That's a good reason to 1) encourage custom services to live in-band and 2) recommend compatibility with Bitcoin's standard mechanisms (12 byte command names, 1 bit service announcement, 1 bit peer exchange etc.) On 8/13/2012 3:15 PM, Mike Hearn wrote: I think it's pretty reasonable, although people will want to use node flags to get into the addr broadcasts anyway. That said, I suspect (based on previous discussions) that there would be quite some pushback against putting extra functionality into the core Bitcoin network. Most likely people will re-use the code with different peer discovery seeds and bootstrap similar but unrelated P2P networks for doing new applications. For instance, what if we want to do the language translation app I've talked about a few times before? You need a way to floodfill broadcast invalid transactions to interested parties. The pubsub mechanism in the Bitcoin protocol was an interesting way to do that, but I think it got removed. To broadcast to interested nodes now, you'd have to find them via addr broadcasts and then connect directly. And if you're going to do that, you may as well just form an entirely independent network. More elaboration of the use cases might therefore be useful.
