Re: [Bitcoin-development] BIP 35: add mempool message
I think MSG_TX is fine. Simply sending an inv to the other node at startup would work, but it's better to request it explicitly as it will let the connecting peer configure a bloom filter before requesting mempool contents. It's already too heavy for mobile clients to download the entire mempool contents at startup so I probably wouldn't implement/activate support for this on the bitcoinj side until bloom filtering is done, and then this BIP would have to be updated to reflect that the response from mempool is filtered. On Thu, Aug 16, 2012 at 11:05 PM, Jeff Garzik jgar...@exmulti.com wrote: On MSG_MEMTX: The current version has a much higher Just Works value. On empty inv: It is generally better to do something unconditionally, than have a response generated only under certain conditions. And Alan is correct to note that unknown messages are ignored (intentionally, for expansion). However, unconditionally returning a response has little to do with feature probing/discovery. It is simply a clear, deterministic indication that processing is complete, for each invocation. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Fri, Aug 17, 2012 at 9:40 AM, Pieter Wuille pieter.wui...@gmail.com wrote: On Thu, Aug 16, 2012 at 05:05:58PM -0400, Jeff Garzik wrote: On MSG_MEMTX: The current version has a much higher Just Works value. On empty inv: It is generally better to do something unconditionally, than have a response generated only under certain conditions. And Alan is correct to note that unknown messages are ignored (intentionally, for expansion). However, unconditionally returning a response has little to do with feature probing/discovery. It is simply a clear, deterministic indication that processing is complete, for each invocation. I disagree. Returning an empty inv is a very strange way of replying empty mempool. Bitcoin P2P is not a request-response protocol, and inv messages are sent where there are inventory items to send. The reaction to a request (for example getblocks) can be nothing, or one or more inv messages if necessary. Special casing an empty inv to mean empty mempool is trying to hack a request-response system on top of the asynchronous system. OK, just updated 'mempool' branch to not return inv if mempool is empty. If there is need for confirming the transmission of the mempool is complete, the proposal to use a MSG_MEMTX sounds good to me. No client will ever receive such an inv without requesting the mempool, and implementing handling MSG_MEMTX is trivial. MSG_MEMTX is not a good idea for this use case. Just sent a ping(nonce). Bitcoin P2P processes requests in-order, and responds accordingly. The remote end may insert asynchronous messages into the response stream, certainly, but responses to queries are processed and returned in-order. A 'getdata' response is fully sent before a 'ping' response is sent, etc. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 1:40 PM, Amir Taaki zgen...@yahoo.com wrote: The format for mempool packet is missing. I'm guessing that it is an empty message, right? Yes, it is an empty message. BIP updated. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille pieter.wui...@gmail.com wrote: I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. Yes. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? A simple guarantee of 1:1 correspondence between request and response. The bitcoin protocol sometimes simply elides a response when the response would be empty, and this makes it difficult to know whether a request is timing out or already processed. Sending a ping(nonce) after each P2P command is another way of achieving same :) This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? An nServices bit would be a better fit for this optional service, but nServices bits seemed like a scarce resource, so I elected to be conservative. Absent the scarce-resource concern, I'd vote for an nServices bit. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. 1) Upon receipt of a mempool message, the node will respond with an inv message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? 3) Feature discovery is enabled by checking two version message attributes: a) Protocol version = 60002 b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Pieter -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
My thoughts: The extension is simple. It's only really useful for the use-cases listed if the majority of nodes implement it. As I view the proposal, it is perfectly simple and uncomplicated. If it's implemented, then I suggest to just increment version and make it part of the protocol. On the flipside it is another notch in complicating an already diffuse protocol, but it seems a rather benign offense in that regard compared to other changes (past and future). - Original Message - From: Pieter Wuille pieter.wui...@gmail.com To: Jeff Garzik jgar...@exmulti.com Cc: Bitcoin Development bitcoin-development@lists.sourceforge.net Sent: Thursday, August 16, 2012 6:56 PM Subject: Re: [Bitcoin-development] BIP 35: add mempool message On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. 1) Upon receipt of a mempool message, the node will respond with an inv message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? 3) Feature discovery is enabled by checking two version message attributes: a) Protocol version = 60002 b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Pieter -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? BitcoinJS will implement it, it's a useful feature and there is no reason not to support it. Two comments from my end: - This is just a thought, but I wouldn't mind using a new inv_type for this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast and relay txs are stored in a very fast local cache whereas the general mempool is stored in a slower data structure. By being able to distinguish incoming getdata requests I can save a few milliseconds by querying the right storage right away. Might also help with things like telling apart broadcast/relayed transactions from the response to a mempool request for purposes like DoS scoring etc. Not a big deal by any means, but I also don't see a downside to it. inv_types are not a scarce resource, we have four billion of them available. For now clients would just treat MSG_TX and MSG_MEMTX interchangeably. - If a node doesn't have anything in it's mempool it sends back an empty inv message. This is either ambiguous (if other things also send empty inv messages in the future) or arbitrary (why should an empty inv be associated with a mempool request of all things.) Instead why not respond with an inv message that contains a single element of type MSG_MEMTX and hash 0. That would a very direct way to indicate that this response is associated with a mempool request. I'm not married to either suggestion, just trying to add my perspective. One thing you notice when reimplementing Bitcoin is that Bitcoin's protocol leaves out a lot of information not for space reasons, but because the reference client's implementation doesn't happen to need it. Sometimes however this locks other clients into doing things the same way. If we can make the protocol a bit richer, especially if this doesn't cost any extra bytes, then we should consider it as it might help some implementation down the road make a neat optimization. On 8/16/2012 7:56 PM, Pieter Wuille wrote: On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. 1) Upon receipt of a mempool message, the node will respond with an inv message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? 3) Feature discovery is enabled by checking two version message attributes: a) Protocol version = 60002 b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 2:04 PM, Jeff Garzik jgar...@exmulti.com wrote: On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille pieter.wui...@gmail.com wrote: I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. Yes. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? A simple guarantee of 1:1 correspondence between request and response. The bitcoin protocol sometimes simply elides a response when the response would be empty, and this makes it difficult to know whether a request is timing out or already processed. Sending a ping(nonce) after each P2P command is another way of achieving same :) Is there a problem with sending unrecognized messages to nodes? If we create a new message type specifically asking for memory pool transactions, and we broadcast it to all nodes that we are connected to, and none of them respond, then either there are no tx in their memory pools, or they don't recognize the message and ignore it. Either way, you're not going to get any extra information out of them. If you really care, a simple ping can identify whether they're still connected and should've responded (as Jeff said). As long as the older node won't cut you off for sending one unrecognized request, it seems that you can get by fine without requiring that bit. I guess it depends on the utility of definitively identifying whether a node supports the functionality. I personally don't feel like it's critical, especially considering that this is most useful only during the transient period when it's not normal for nodes to support it yet. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
MSG_MEMTX solves the issue of not knowing whether a given inv is in response to a mempool command or not. I don't buy the argument that always sending a response inv makes things easier because code should always be able to handle misbehaviour from the remote node (ommiting the inv). However I would argue that it is good to have it, as it makes designing flows of logic much easier (first send this, wait for response, do this, ...). - Original Message - From: Stefan Thomas m...@justmoon.de To: bitcoin-development@lists.sourceforge.net Cc: Sent: Thursday, August 16, 2012 8:21 PM Subject: Re: [Bitcoin-development] BIP 35: add mempool message This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? BitcoinJS will implement it, it's a useful feature and there is no reason not to support it. Two comments from my end: - This is just a thought, but I wouldn't mind using a new inv_type for this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast and relay txs are stored in a very fast local cache whereas the general mempool is stored in a slower data structure. By being able to distinguish incoming getdata requests I can save a few milliseconds by querying the right storage right away. Might also help with things like telling apart broadcast/relayed transactions from the response to a mempool request for purposes like DoS scoring etc. Not a big deal by any means, but I also don't see a downside to it. inv_types are not a scarce resource, we have four billion of them available. For now clients would just treat MSG_TX and MSG_MEMTX interchangeably. - If a node doesn't have anything in it's mempool it sends back an empty inv message. This is either ambiguous (if other things also send empty inv messages in the future) or arbitrary (why should an empty inv be associated with a mempool request of all things.) Instead why not respond with an inv message that contains a single element of type MSG_MEMTX and hash 0. That would a very direct way to indicate that this response is associated with a mempool request. I'm not married to either suggestion, just trying to add my perspective. One thing you notice when reimplementing Bitcoin is that Bitcoin's protocol leaves out a lot of information not for space reasons, but because the reference client's implementation doesn't happen to need it. Sometimes however this locks other clients into doing things the same way. If we can make the protocol a bit richer, especially if this doesn't cost any extra bytes, then we should consider it as it might help some implementation down the road make a neat optimization. On 8/16/2012 7:56 PM, Pieter Wuille wrote: On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. 1) Upon receipt of a mempool message, the node will respond with an inv message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An inv message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? 3) Feature discovery is enabled by checking two version message attributes: a) Protocol version = 60002 b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On MSG_MEMTX: The current version has a much higher Just Works value. On empty inv: It is generally better to do something unconditionally, than have a response generated only under certain conditions. And Alan is correct to note that unknown messages are ignored (intentionally, for expansion). However, unconditionally returning a response has little to do with feature probing/discovery. It is simply a clear, deterministic indication that processing is complete, for each invocation. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
