Re: [Bitcoin-development] BIP 35: add mempool message
On MSG_MEMTX: The current version has a much higher Just Works value. On empty "inv": It is generally better to do something unconditionally, than have a response generated only under certain conditions. And Alan is correct to note that unknown messages are ignored (intentionally, for expansion). However, unconditionally returning a response has little to do with feature probing/discovery. It is simply a clear, deterministic indication that processing is complete, for each invocation. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
MSG_MEMTX solves the issue of not knowing whether a given inv is in response to a "mempool" command or not. I don't buy the argument that always sending a response "inv" makes things easier because code should always be able to handle misbehaviour from the remote node (ommiting the "inv"). However I would argue that it is good to have it, as it makes designing flows of logic much easier (first send this, wait for response, do this, ...). - Original Message - From: Stefan Thomas To: bitcoin-development@lists.sourceforge.net Cc: Sent: Thursday, August 16, 2012 8:21 PM Subject: Re: [Bitcoin-development] BIP 35: add mempool message > This seems safe, although it forces other full implementations that want to > expose protocol version 60002 (or later) to also implement this. What do they > think about this? BitcoinJS will implement it, it's a useful feature and there is no reason not to support it. Two comments from my end: - This is just a thought, but I wouldn't mind using a new inv_type for this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast and relay txs are stored in a very fast local cache whereas the general mempool is stored in a slower data structure. By being able to distinguish incoming getdata requests I can save a few milliseconds by querying the right storage right away. Might also help with things like telling apart broadcast/relayed transactions from the response to a mempool request for purposes like DoS scoring etc. Not a big deal by any means, but I also don't see a downside to it. inv_types are not a scarce resource, we have four billion of them available. For now clients would just treat MSG_TX and MSG_MEMTX interchangeably. - If a node doesn't have anything in it's mempool it sends back an empty inv message. This is either ambiguous (if other things also send empty inv messages in the future) or arbitrary (why should an empty inv be associated with a mempool request of all things.) Instead why not respond with an inv message that contains a single element of type MSG_MEMTX and hash 0. That would a very direct way to indicate that this response is associated with a mempool request. I'm not married to either suggestion, just trying to add my perspective. One thing you notice when reimplementing Bitcoin is that Bitcoin's protocol leaves out a lot of information not for space reasons, but because the reference client's implementation doesn't happen to need it. Sometimes however this locks other clients into doing things the same way. If we can make the protocol a bit richer, especially if this doesn't cost any extra bytes, then we should consider it as it might help some implementation down the road make a neat optimization. On 8/16/2012 7:56 PM, Pieter Wuille wrote: > On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: >> Consensus was we should do a BIP for all P2P changes, so here it is... >> feedback requested. >> >> https://en.bitcoin.it/wiki/BIP_0035 > I like the idea of being able to query the memory pool of a node; the > implementation is straightforward, which is good. Maybe effectively using the > command can be added? I suppose it is interesting in general for nodes to > get a memory pool refill at startup anyway. > >> 1) Upon receipt of a "mempool" message, the node will respond >> with an "inv" message containing MSG_TX hashes of all the >> transactions in the node's transaction memory pool. >> >> An "inv" message is always returned, even if empty. > I'm not sure about this last. What is it good for? inv packets can always be > sent, even not in response to others, so it is not that this gives you an > acknowledgement the mempool is updated? > >> 3) Feature discovery is enabled by checking two "version" message attributes: >> >> a) Protocol version >= 60002 >> b) NODE_NETWORK bit set in nServices > This seems safe, although it forces other full implementations that want to > expose protocol version 60002 (or later) to also implement this. What do they > think about this? > > I would like to suggest to allocate an extra service bit for this. We still > have 63 left, and this is a well-defined and useful extra service that was > not yet provided by any earlier node. Doing that would also mean that > mempool-providing survices may be discovered before connecting to them, as > the service bits are carried around in addr messages. Any opinions about that? > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 2:04 PM, Jeff Garzik wrote: > On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille > wrote: > > I suppose it is interesting in general for nodes to > > get a memory pool refill at startup anyway. > > Yes. > > >>An "inv" message is always returned, even if empty. > > > > I'm not sure about this last. What is it good for? inv packets can > always be > > sent, even not in response to others, so it is not that this gives you an > > acknowledgement the mempool is updated? > > A simple guarantee of 1:1 correspondence between request and response. > The bitcoin protocol sometimes simply elides a response when the > response would be empty, and this makes it difficult to know whether a > request is timing out or already processed. > > Sending a ping(nonce) after each P2P command is another way of achieving > same :) > > Is there a problem with sending unrecognized messages to nodes? If we create a new message type specifically asking for memory pool transactions, and we broadcast it to all nodes that we are connected to, and none of them respond, then either there are no tx in their memory pools, or they don't recognize the message and ignore it. Either way, you're not going to get any extra information out of them. If you really care, a simple ping can identify whether they're still connected and should've responded (as Jeff said). As long as the older node won't cut you off for sending one unrecognized request, it seems that you can get by fine without requiring that bit. I guess it depends on the utility of definitively identifying whether a node supports the functionality. I personally don't feel like it's critical, especially considering that this is most useful only during the transient period when it's not normal for nodes to support it yet. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
> This seems safe, although it forces other full implementations that want to > expose protocol version 60002 (or later) to also implement this. What do they > think about this? BitcoinJS will implement it, it's a useful feature and there is no reason not to support it. Two comments from my end: - This is just a thought, but I wouldn't mind using a new inv_type for this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast and relay txs are stored in a very fast local cache whereas the general mempool is stored in a slower data structure. By being able to distinguish incoming getdata requests I can save a few milliseconds by querying the right storage right away. Might also help with things like telling apart broadcast/relayed transactions from the response to a mempool request for purposes like DoS scoring etc. Not a big deal by any means, but I also don't see a downside to it. inv_types are not a scarce resource, we have four billion of them available. For now clients would just treat MSG_TX and MSG_MEMTX interchangeably. - If a node doesn't have anything in it's mempool it sends back an empty inv message. This is either ambiguous (if other things also send empty inv messages in the future) or arbitrary (why should an empty inv be associated with a mempool request of all things.) Instead why not respond with an inv message that contains a single element of type MSG_MEMTX and hash 0. That would a very direct way to indicate that this response is associated with a mempool request. I'm not married to either suggestion, just trying to add my perspective. One thing you notice when reimplementing Bitcoin is that Bitcoin's protocol leaves out a lot of information not for space reasons, but because the reference client's implementation doesn't happen to need it. Sometimes however this locks other clients into doing things the same way. If we can make the protocol a bit richer, especially if this doesn't cost any extra bytes, then we should consider it as it might help some implementation down the road make a neat optimization. On 8/16/2012 7:56 PM, Pieter Wuille wrote: > On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: >> Consensus was we should do a BIP for all P2P changes, so here it is... >> feedback requested. >> >> https://en.bitcoin.it/wiki/BIP_0035 > I like the idea of being able to query the memory pool of a node; the > implementation is straightforward, which is good. Maybe effectively using the > command can be added? I suppose it is interesting in general for nodes to > get a memory pool refill at startup anyway. > >> 1) Upon receipt of a "mempool" message, the node will respond >>with an "inv" message containing MSG_TX hashes of all the >>transactions in the node's transaction memory pool. >> >>An "inv" message is always returned, even if empty. > I'm not sure about this last. What is it good for? inv packets can always be > sent, even not in response to others, so it is not that this gives you an > acknowledgement the mempool is updated? > >> 3) Feature discovery is enabled by checking two "version" message attributes: >> >>a) Protocol version >= 60002 >>b) NODE_NETWORK bit set in nServices > This seems safe, although it forces other full implementations that want to > expose protocol version 60002 (or later) to also implement this. What do they > think about this? > > I would like to suggest to allocate an extra service bit for this. We still > have 63 left, and this is a well-defined and useful extra service that was > not yet provided by any earlier node. Doing that would also mean that > mempool-providing survices may be discovered before connecting to them, as > the service bits are carried around in addr messages. Any opinions about that? > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
My thoughts: The extension is simple. It's only really useful for the use-cases listed if the majority of nodes implement it. As I view the proposal, it is perfectly simple and uncomplicated. If it's implemented, then I suggest to just increment version and make it part of the protocol. On the flipside it is another notch in complicating an already diffuse protocol, but it seems a rather benign offense in that regard compared to other changes (past and future). - Original Message - From: Pieter Wuille To: Jeff Garzik Cc: Bitcoin Development Sent: Thursday, August 16, 2012 6:56 PM Subject: Re: [Bitcoin-development] BIP 35: add mempool message On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: > Consensus was we should do a BIP for all P2P changes, so here it is... > feedback requested. > > https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. > 1) Upon receipt of a "mempool" message, the node will respond > with an "inv" message containing MSG_TX hashes of all the > transactions in the node's transaction memory pool. > > An "inv" message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? > 3) Feature discovery is enabled by checking two "version" message attributes: > > a) Protocol version >= 60002 > b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Pieter -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote: > Consensus was we should do a BIP for all P2P changes, so here it is... > feedback requested. > > https://en.bitcoin.it/wiki/BIP_0035 I like the idea of being able to query the memory pool of a node; the implementation is straightforward, which is good. Maybe effectively using the command can be added? I suppose it is interesting in general for nodes to get a memory pool refill at startup anyway. > 1) Upon receipt of a "mempool" message, the node will respond >with an "inv" message containing MSG_TX hashes of all the >transactions in the node's transaction memory pool. > >An "inv" message is always returned, even if empty. I'm not sure about this last. What is it good for? inv packets can always be sent, even not in response to others, so it is not that this gives you an acknowledgement the mempool is updated? > 3) Feature discovery is enabled by checking two "version" message attributes: > >a) Protocol version >= 60002 >b) NODE_NETWORK bit set in nServices This seems safe, although it forces other full implementations that want to expose protocol version 60002 (or later) to also implement this. What do they think about this? I would like to suggest to allocate an extra service bit for this. We still have 63 left, and this is a well-defined and useful extra service that was not yet provided by any earlier node. Doing that would also mean that mempool-providing survices may be discovered before connecting to them, as the service bits are carried around in addr messages. Any opinions about that? -- Pieter -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille wrote: > I suppose it is interesting in general for nodes to > get a memory pool refill at startup anyway. Yes. >>An "inv" message is always returned, even if empty. > > I'm not sure about this last. What is it good for? inv packets can always be > sent, even not in response to others, so it is not that this gives you an > acknowledgement the mempool is updated? A simple guarantee of 1:1 correspondence between request and response. The bitcoin protocol sometimes simply elides a response when the response would be empty, and this makes it difficult to know whether a request is timing out or already processed. Sending a ping(nonce) after each P2P command is another way of achieving same :) > This seems safe, although it forces other full implementations that want to > expose protocol version 60002 (or later) to also implement this. What do they > think about this? > > I would like to suggest to allocate an extra service bit for this. We still > have 63 left, and this is a well-defined and useful extra service that was > not yet provided by any earlier node. Doing that would also mean that > mempool-providing survices may be discovered before connecting to them, as > the service bits are carried around in addr messages. Any opinions about that? An nServices bit would be a better fit for this optional service, but nServices bits seemed like a scarce resource, so I elected to be conservative. Absent the scarce-resource concern, I'd vote for an nServices bit. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
On Thu, Aug 16, 2012 at 1:40 PM, Amir Taaki wrote: > The format for "mempool" packet is missing. I'm guessing that it is an empty > message, right? Yes, it is an empty message. BIP updated. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 35: add mempool message
The format for "mempool" packet is missing. I'm guessing that it is an empty message, right? Might be good to add that. - Original Message - From: Jeff Garzik To: Bitcoin Development Cc: Sent: Thursday, August 16, 2012 6:32 PM Subject: [Bitcoin-development] BIP 35: add mempool message Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 Abstract --- Make a network node's transaction memory pool accessible via a new "mempool" message. Extend the existing "getdata" message behavior to permit accessing the transaction memory pool. Motivation --- Several use cases make it desireable to expore a network node's transaction memory pool: * SPV clients, wishing to obtain zero-confirmation transactions sent or received. * Miners, downloading existing network transactions after a restart. * Remote network diagnostics. Specification --- 1) Upon receipt of a "mempool" message, the node will respond with an "inv" message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An "inv" message is always returned, even if empty. 2) The typical node behavior in response to an "inv" is "getdata". However, the reference Satoshi implementation ignores requests for transaction hashes outside that which is recently relayed. To support "mempool", an implementation must extend its "getdata" message support to querying the memory pool. 3) Feature discovery is enabled by checking two "version" message attributes: a) Protocol version >= 60002 b) NODE_NETWORK bit set in nServices Backwards compatibility --- Older clients remain 100% compatible and interoperable after this change. Implementation --- See https://github.com/bitcoin/bitcoin/pull/1641 -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] BIP 35: add mempool message
Consensus was we should do a BIP for all P2P changes, so here it is... feedback requested. https://en.bitcoin.it/wiki/BIP_0035 Abstract --- Make a network node's transaction memory pool accessible via a new "mempool" message. Extend the existing "getdata" message behavior to permit accessing the transaction memory pool. Motivation --- Several use cases make it desireable to expore a network node's transaction memory pool: * SPV clients, wishing to obtain zero-confirmation transactions sent or received. * Miners, downloading existing network transactions after a restart. * Remote network diagnostics. Specification --- 1) Upon receipt of a "mempool" message, the node will respond with an "inv" message containing MSG_TX hashes of all the transactions in the node's transaction memory pool. An "inv" message is always returned, even if empty. 2) The typical node behavior in response to an "inv" is "getdata". However, the reference Satoshi implementation ignores requests for transaction hashes outside that which is recently relayed. To support "mempool", an implementation must extend its "getdata" message support to querying the memory pool. 3) Feature discovery is enabled by checking two "version" message attributes: a) Protocol version >= 60002 b) NODE_NETWORK bit set in nServices Backwards compatibility --- Older clients remain 100% compatible and interoperable after this change. Implementation --- See https://github.com/bitcoin/bitcoin/pull/1641 -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
