Re: [bitcoin-dev] Removing BIP-125 Rule #5 Pinning with the Always-Replaceable Invariant

[Suhas Daftuar via bitcoin-dev]

Hello bitcoin-dev,

The description in the OP is completely broken for the simple reason that
Bitcoin transactions can have multiple inputs, and so a single transaction
can conflict with multiple in-mempool transactions. The proposal would
limit the number of descendants of each in-mempool transaction to
MAX_REPLACEMENT_CANDIDATES (note that this is duplicative of the existing
Bitcoin Core descendant limits), but a transaction that has, say, 500
inputs might arrive and conflict with 500 unrelated in-mempool
transactions. This could result in 12,500 evictions -- far more than the
100 that was intended.

Also, note that those 500 transactions might instead have 24 ancestors each
(again, using the default chain limits in Bitcoin Core) -- this would
result in 12,000 transactions whose state would be updated as a result of
evicting those 500 transactions. Hopefully this gives some perspective on
why we have a limit.


On Mon, Nov 7, 2022 at 3:17 PM Peter Todd via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> tl;dr: We can remove the problem of Rule #5 pinning by ensuring that all
> transactions in the mempool are always replaceable.
>
>
> Currently Bitcoin Core implements rule #5 of BIP-125:
>
> The number of original transactions to be replaced and their descendant
> transactions which will be evicted from the mempool must not exceed a
> total
> of 100 transactions.
>
> As of Bitcoin Core v24.0rc3, this rule is implemented via the
> MAX_REPLACEMENT_CANDIDATES limit in GetEntriesForConflicts:
>
> // Rule #5: don't consider replacing more than
> MAX_REPLACEMENT_CANDIDATES
> // entries from the mempool. This potentially overestimates the number
> of actual
> // descendants (i.e. if multiple conflicts share a descendant, it will
> be counted multiple
> // times), but we just want to be conservative to avoid doing too much
> work.
> if (nConflictingCount > MAX_REPLACEMENT_CANDIDATES) {
> return strprintf("rejecting replacement %s; too many potential
> replacements (%d > %d)\n",
>  txid.ToString(),
>  nConflictingCount,
>  MAX_REPLACEMENT_CANDIDATES);
> }
>
> There is no justification for this rule beyond avoiding "too much work";
> the
> exact number was picked at random when the BIP was written to provide an
> initial conservative limit, and is not justified by benchmarks or memory
> usage
> calculations. It may in fact be the case that this rule can be removed
> entirely
> as the overall mempool size limits naturally limit the maximum number of
> possible replacements.
>
>
> But for the sake of argument, let's suppose some kind of max replacement
> limit
> is required. Can we avoid rule #5 pinning? The answer is yes, we can, with
> the
> following invariant:
>
> No transaction may be accepted into the mempool that would cause
> another
> transaction to be unable to be replaced due to Rule #5.
>
> We'll call this the Replaceability Invariant. Implementing this rule is
> simple:
> for each transaction maintain an integer, nReplacementCandidates. When a
> non-conflicting transaction is considered for acceptance to the mempool,
> verify
> that nReplacementCandidates + 1 < MAX_REPLACEMENT_CANDIDATES for each
> unconfirmed parent. When a transaction is accepted, increment
> nReplacementCandidates by 1 for each unconfirmed parent.
>
> When a *conflicting* transaction is considered for acceptance, we do _not_
> need
> to verify anything: we've already guaranteed that every transaction in the
> mempool is replaceable! The only thing we may need to do is to decrement
> nReplacementCandidates by however many children we have replaced, for all
> unconfirmed parents.
>
> When a block is mined, note how the nReplacementCandidates of all
> unconfirmed
> transactions remains unchanged because a confirmed transaction can't spend
> an
> unconfirmed txout.
>
> The only special case to handle is a reorg that changes a transaction from
> confirmed to unconfirmed. Setting nReplacementCandidates to
> MAX_REPLACEMENT_CANDIDATES would be fine in that very rare case.
>
>
> Note that like the existing MAX_REPLACEMENT_CANDIDATES check, the
> Replaceability Invariant overestimates the number of transactions to be
> replaced in the event that multiple children are spent by the same
> transaction.
> That's ok: diamond tx graphs are even more unusual than unconfirmed
> children,
> and there's no reason to bend over backwards to accomodate them.
>
> Prior art: this proposed rule is similar in spirit to the package limits
> aready
> implemented in Bitcoin Core.
>
> --
> https://petertodd.org 'peter'[:-1]@petertodd.org
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org

Re: [bitcoin-dev] On mempool policy consistency

[Suhas Daftuar via bitcoin-dev]

AJ,

Thanks for the thoughtful post. I think your observations about how we view
mempool policy in the Bitcoin Core project, and how that seems to be
changing in the discussions around `-mempoolfullrbf`, are on-point and
provide a helpful baseline for considering future policy changes.

For a long time I viewed fullrbf as an eventuality and I considered myself
to be philosophically supportive of the idea.  However, after giving this
issue some thought in the past few weeks, I am reversing my thinking on
this.  Concretely, I will argue that we should continue to maintain a relay
policy where replacements are rejected for transactions that don't opt-in
to RBF (as described in BIP 125), and moreover, that we should remove the
`-mempoolfullrbf` flag from Bitcoin Core’s latest release candidate and not
plan to release software with that flag, unless (or until) circumstances
change on the network, which I'll discuss below.

This is, of course, a nuanced topic, and among the considerations is a
philosophy of how to think about the relay policy and configuration options
that we make available in Bitcoin Core (a consideration that is perhaps
unique to that project, but I think relevant for this mailing list).

I'll start with some technical issues regarding the benefits of enabling
fullrbf on the network.  In the current BIP 125 regime, every time a
transaction is created, a choice is made whether to subject the transaction
to BIP 125’s RBF rules or not (based on the sequence values of the
inputs).  So given that users can already opt-in to RBF, the benefit of a
“fullrbf” network policy would be if, somehow, RBF users were still denied
the benefits of RBF due to the existence of other transactions that don’t
opt-in.

Along those lines, Antoine Riard brought up[1] a DoS vector that is
available to someone who wants to interfere with multi-party funded
transactions, and suggested that fullrbf would eliminate the problem.
After exploring that question again in this thread (thanks to Greg Sanders
for clarifying this to me), I understand that the issue is around ensuring
that a multiparty (coinjoin-type) protocol is able to make eventual
progress, by having a candidate multiparty transaction either eventually
confirm or become conflicted with something that has been confirmed, in
which case the double-spend information could be used to start a new
coinjoin round with fewer participants.  The concern Antoine and Greg have
brought up is that non-rbf transactions can persist in the mempool
~indefinitely (at a low feerate and not subject to replacement) and
interfere with progress being made in a coinjoin protocol.

However, it seems to me that similar problems exist for such a protocol
even in a fullrbf world, as we understand that term today.  I mentioned the
ability for rbf “pinning” to interfere with relay of the multiparty
transaction (even if the conflicting transaction signals for RBF – a set of
large but low feerate conflicting transactions can persist in the mempool
and make it difficult for the coinjoin transaction from confirming, at
least without attaching a very large fee); and as Greg mentioned in a
followup, the BIP 125 rule to only permit 100 transactions to be removed
from the mempool at a time during a replacement can also be used to pin a
coinjoin protocol in the same way as a non-rbf transaction today.  It seems
to me that what these multiparty protocols actually need is some sort of
"maximal rbf" network policy: a way to guarantee that a transaction which
should be desirable for a miner to mine would always get to a miner and
considered for inclusion in a block, no matter what the state of node’s
mempools on the network.

While that sounds like a reasonable thing to want on its face (and worth
working on), it's not how opt-in RBF works today, nor is it how transaction
relay has ever conceptually worked.  We have not, thus far, been able to
come up with a total ordering on transaction desirability.  Moreover, due
to all the DoS issues that exist with transaction relay, there are plenty
of seemingly legitimate ways to construct transactions that would not relay
well on the network.  Relay has only ever been a best-efforts concept,
where we carve out a small subset of the entire transaction universe for
which we try to optimize propagation.  The idea behind this approach is
that if every use case we can come up with has some way to achieve its
goals using transactions that should (eventually) be able to relay, then
users wouldn’t have much demand for transactions that would deviate from
the supported policies, and we therefore shouldn’t need to worry too much
about incentive compatibility concerns when it comes to transaction types
that wouldn’t relay at all, even if they are high feerate.  (And when those
situations arise where the standard transactions do not accommodate some
needed use case, developers typically work to define a policy that is
compatible with our anti-DoS goals to support such use 

Re: [bitcoin-dev] On mempool policy consistency

[Suhas Daftuar via bitcoin-dev]

I have more to say on this broader topic, but since you've brought up this
particular example I think it's worth commenting:

On Thu, Oct 27, 2022 at 1:23 PM Anthony Towns via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Is that true? Antoine claims [1] that opt-in RBF isn't enough to avoid
> a DoS issue when utxos are jointly funded by untrusting partners, and,
> aiui, that's the main motivation for addressing this now.
>
> [1]
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-May/003033.html
>
> The scenario he describes is: A, B, C create a tx:
>
>   inputs: A1, B1, C1 [opts in to RBF]
>   fees: normal
>   outputs:
> [lightning channel, DLC, etc, who knows]
>
> they all analyse the tx, and agree it looks great; however just before
> publishing it, A spams the network with an alternative tx, double
> spending her input:
>
>   inputs: A1 [does not opt in to RBF]
>   fees: low
>   outputs: A
>
> If A gets the timing right, that's bad for B and C because they've
> populated their mempool with the 1st transaction, while everyone else
> sees the 2nd one instead; and neither tx will replace the other. B and
> C can't know that they should just cancel their transaction, eg:
>
>   inputs: B1, C1 [opts in to RBF]
>   fees: 50% above normal
>   outputs:
> [smaller channel, refund, whatever]
>
> and might instead waste time trying to fee bump the tx to get it mined,
> or similar.
>
> What should folks wanting to do coinjoins/dualfunding/dlcs/etc do to
> solve that problem if they have only opt-in RBF available?
>

I think this is not a real example of a DoS vector that is available
because we support non-rbf signaling transactions. Even in a world where
all transactions are replaceable, person A could double-spend their input
in a way that is annoying for B and C.  For instance, the double-spend
could be low-feerate and large, and effectively pin any attempt to replace
it.  Or it could be higher feerate and confirm and B/C have to start all
over.  Or, A could stall things in the signing phase and B/C have to figure
out when to give up on the channel with A.

So I find this example to be unconvincing.  Are there any other examples
where having a non-replacement policy for some transactions causes problems
for protocols people are trying to build?

Thanks,
Suhas
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Packaged Transaction Relay

[Suhas Daftuar via bitcoin-dev]

(Apologies for the double-post -- I'm resending this message to the list
with much of the quoted text trimmed, because my first message was placed
in the moderation queue for being too large)

Hi,

Thanks for sharing your thoughts on packaged transaction relay.

The sole objective, as expressed in the OP proposal, is to:

"Propagate transactions that are incentive-compatible to mine, even if they
> don't meet minimum feerate alone."


I actually do think there are additional goals we should include in any
protocol change involving transaction relay, such as ensuring that we
minimize bandwidth waste as much as possible (as I mentioned in a previous
message in this thread).

While I understand your proposal seeks to improve on an idea of static
packages in favor of dynamic package construction based on knowledge a node
should have of its peers, I think the main drawback of your proposal is
that it doesn't take into account the complexities of what a peer's
"minimum feerate" might mean.  The consequence of this is that it's not
generally possible for a node to accurately guess whether a given
transaction should be sent in a package to a given peer, or not, and so in
addition to any "packaged transaction relay" mechanism that is implemented
by a transaction announcer, we'd still need to add protocol support for a
receiving peer to retrieve a package as well.

First of all, a node's feerate is a dynamic value.  BIP 133 allows for
nodes to send feefilter messages at any time during the lifetime of a peer
connection.  If we were to compare the feerate of ancestors of a relayed
transaction to the feerate in place at a peer as indicated by feefilter
messages, and use that determine whether those ancestors would have been
successfully relayed or not, then doing so accurately would seem to require
caching relay success for each transaction, for each peer, at the time such
transaction is relayed (or perhaps caching feefilter values that are
received from a peer?).  This seems likely to be undesirable, and, at any
rate, is more complex than merely comparing a pair of feerates.

But even more fundamental than feerates being dynamic is that feerate
itself is not a well-defined concept when applied to arbitrary transaction
(sub-)graphs, and this is at the crux of the difficulty in coming up with
proposals that would meet the objective of ensuring that transactions which
are incentive-compatible to mine all get relayed successfully across the
network.  Here are a couple examples that illustrate this:

- Let A be a low feerate transaction (below any peer's minimum feerate).
Let B and C be descendants of A (but are otherwise unrelated).  Suppose
these transactions are relayed to a node in the order A, B, C.  In the
algorithm you proposed, I believe the determination for whether C should be
announced to a given peer as a package (A, C) or as a singleton would
either (1) depend on whether the package (A, B) was sufficient to meet the
peer's feerate, or (2) waste bandwidth by engaging in packaged relay
whenever A was already successfully relayed as part of a package.  Both of
these approaches seem undesirable.

- Let A be a high fee, but low feerate transaction.  Suppose B is a
transaction that conflicts with A, has a high feerate, but lower total
fee.  In this situation, two different nodes that learned of these two
transactions in opposite order [(A, B) vs (B, A)] might be expected to have
differing mempools -- this at least would be the case in the BIP 125
algorithm (which requires that both feerate and total fee must increase
when replacing an existing transaction), and at any rate it's not obvious
from the information given which would be more optimal to include in a
block, as that depends on factors that go beyond just these two
transactions.  Suppose further that a new transaction C is relayed on the
network, which is a child of B and very high feerate, such that B + C have
higher fee and higher feerate than A, when taken together.  In this case
we'd want our relay protocol to ensure that even nodes which saw A first
should still have a chance to consider transaction C, but the packaging
design you propose (which would compare transaction B's feerate to the
peer's, and conclude packaging is unnecessary because B's feerate might
already exceed the peer's feerate) would not facilitate this.

To summarize, the point I'd make from these examples is that we should not
expect that "feerate" (whatever that means) alone will be a sufficient
predictor of what is in our peer's mempools.  So while there may be some
situations where a transaction relayer might be able to usefully package up
a transaction with its dependencies (perhaps in narrowly defined
situations), there will also always be situations where this isn't
possible, and what I conclude from that is that it should be helpful to add
to the protocol some way for the recipient of a transaction to request the
dependencies directly.

Taken together, I roughly understand 

Re: [bitcoin-dev] Package Relay Proposal

[Suhas Daftuar via bitcoin-dev]

Hi,

Thanks again for your work on this!

One question I have is about potential bandwidth waste in the case of nodes
running with different policy rules.  Here's my understanding of a scenario
I think could happen:

1) Transaction A is both low-fee and non-standard to some nodes on the
network.
2) Whenever a transaction T that spends A is relayed, new nodes will send
INV(PKGINFO1, T) to all package-relay peers.
3) Nodes on the network that have implemented package relay, but do not
accept A, will send getdata(PKGINFO1, T) and learn all of T's unconfirmed
parents (~32 bytes * number of parents(T)).
4) Such nodes will reject T.  But because of transaction malleability, and
to avoid being blinded to a transaction unnecessarily, these nodes will
likely still send getdata(PKGINFO1, T) to every node that announces T, in
case someone has a transaction that includes an alternate set of parent
transactions that would pass policy checks.

Is that understanding correct?  I think a good design goal would be to not
waste bandwidth in non-adversarial situations.  In this case, there would
be bandwidth waste from downloading duplicate data from all your peers,
just because the announcement doesn't commit to the set of parent wtxids
that we'd get from the peer (and so we are unable to determine that all our
peers would be telling us the same thing, just based on the announcement).

Some ways to mitigate this might be to: (a) include a hash (maybe even just
a 20-byte hash -- is that enough security?) of the package wtxids (in some
canonical ordering) along with the wtxid of the child in the initial
announcement; (b) limit the use of v1 packages to transactions with very
few parents (I don't know if this is reasonable for the use cases we have
in mind).

Another point I wanted to bring up is about the rules around v1 package
validation generally, and the use of a blockhash in transaction relay
specifically.  My first observation is that it won't always be the case
that a v1 package relay node will be able to validate that a set of package
transactions is fully sorted topologically, because there may be
(non-parent) ancestors that are missing from the package and the best a
peer can validate is topology within the package -- this means that a peer
can validly (under this BIP) relay transaction packages out of the true
topological sort (if all ancestors were included).

This makes me wonder how useful this topological rule is.  I suppose there
is some value in preventing completely broken implementations from staying
connected and so there is no harm in having the rule, but perhaps it would
be helpful to add that nodes SHOULD order transactions based on topological
sort in the complete transaction graph, so that if missing-from-package
ancestors are already known by a peer (which is the expected case when
using v1 package relay on transactions that have more than one generation
of unconfirmed ancestor) then the remaining transactions are already
properly ordered, and this is helpful even if unenforceable in general.

The other observation I wanted to make was that having transaction relay
gated on whether two nodes agree on chain tip seems like an overly
restrictive criteria.  I think an important design principle is that we
want to minimize disruption from network splits -- if there are competing
blocks found in a small window of time, it's likely that the utxo set is
not materially different on the two chains (assuming miners are selecting
from roughly the same sets of transactions when this happens, which is
typical).  Having transaction relay bifurcate on the two network halves
would seem to exacerbate the difference between the two sides of the split
-- users ought to be agnostic about how benign splits are resolved and
would likely want their transactions to relay across the whole network.

Additionally, use of a chain tip might impose a larger burden than is
necessary on software that would seek to participate in transaction relay
without implementing headers sync/validation.  I don't know what software
exists on the network, but I imagine there are a lot of scripts out there
for transaction submission to the public p2p network, and in thinking
about modifying such a script to utilize package relay it seems like an
unnecessary added burden to first learn a node's tip before trying to relay
a transaction.

Could you explain again what the benefit of including the blockhash is?  It
seems like it is just so that a node could prioritize transaction relay
from peers with the same chain tip to maximize the likelihood of
transaction acceptance, but in the common case this seems like a pretty
negligible concern, and in the case of a chain fork that persists for many
minutes it seems better to me that we not partition the network into
package-relay regimes and just risk a little extra bandwidth in one
direction or the other.  If we solve the problem I brought up at the
beginning (of de-duplicating package data across peers 

Re: [bitcoin-dev] Proposal for new "disabletx" p2p message

[Suhas Daftuar via bitcoin-dev]

On Thu, Jan 14, 2021 at 1:46 AM Anthony Towns  wrote:

> > ==Specification==
> > # A new disabletx message is added, [...]
> > # A node that has sent or received a disabletx message to/from a peer
> MUST NOT send any of these messages to the peer:
> > ## inv messages for transactions
> > [...]
> > # It is RECOMMENDED that a node that has sent or received a disabletx
> message to/from a peer not send any of these messages to the peer:
> > ## addr/getaddr
> > ## addrv2 (BIP 155)
>
> In particular, I think combining these doesn't make sense for:
>
>  * nodes running with -blocksonly (that want to stay up to date with
>the blockchain, but don't care about txes)
> - not sending disabletx reduces their potential connectivity, if
>   nodes are willing to accept more disabletx peers due to the lower
>   resource usage
> - sending disabletx means they can't maintain their addr db and find
>   other nodes to connect to
>

This is quite implementation specific, but the Bitcoin Core implementation
of -blocksonly already differs from what you would get if all outbound
connections were treated as block-relay-only peers; I initially tried to
make the -blocksonly setting behave the same as block-relay-only
connections and never relay transactions, but it turned out that Bitcoin
Core currently allows users running in -blocksonly mode to sometimes relay
transactions themselves (there's even a test for it [1]).

This isn't compatible with the design of disabletx, so I don't think it
could be used.  I think that is okay, as I don't think we need to make
allowances right now to encourage the use of more -blocksonly nodes on the
network by prioritizing their existing outbound connections[2]. In the case
of block-relay-only connections, the next step I plan to propose in the
Bitcoin Core implementation is an increase in the number of inbound slots
to accommodate additional disabletx peers, to facilitate an eventual
increase in the number of outbound block-relay-only connections that
Bitcoin Core would initiate by default (and reduce potential concern about
overwhelming the network's capacity).

 * non-listening nodes running with -connect to one/more preselected peers
> - they can't send disabletx generally because they want txes
> - they don't need addr information (since they only make connections
>   to some known peers), and don't have many peers to relay addresses
>   on to, so are essentially blackholes, so would like to disable
>   addr relay for much the same reasons
>

While I think it's a valid use-case if someone wants to run nodes in this
way, I don't see why it has direct bearing on this discussion -- adding
protocol support for this use case is not incompatible with my disabletx
proposal; indeed, once we have worked out an addr relay protocol that
supports a peer telling another to not relay addresses, I would plan to use
it along with disabletx.

-

There was some discussion about this proposal on IRC recently as well [3]
and this issue of mentioning addr-relay in a BIP about tx-relay seems to
have drawn some critique.  I think if there were some other use case that
other developers have which would use disabletx (say, because the current
fRelay solution also is inadequate for those use cases today), yet would
want addr relay on these links, then that would be a good reason to drop
mention of it from the BIP and defer discussion of addr relay entirely to
some new future proposal.  However, I'm not yet aware of such a use case,
as the -blocksonly one mentioned above does not actually fit (and nor do I
think that a -blocksonly mode where transactions are never sent but addr
messages are exchanged is necessary beyond how we already do that today, by
just sending fRelay=false).

Moreover, because there is no specification/BIP that governs how address
relay should work on the network, I think that if we just dropped mention
of addr-relay from the BIP, that software ought to just disable addr-relay
to nodes sending disabletx, anyway!  Addr relay is currently purely local
policy, so I think it's reasonable to conclude that if the only known use
case of a peer sending disabletx is for a behavior that also would drop
addr messages, then optimizing software for that use case is logical and I
would propose that myself for how Bitcoin Core behaves.  It seems polite to
mention this in the BIP as well, so that other software implementors have
the same understanding that I have for how this should work, until we have
an address relay protocol extension that governs it explicitly.

-

Why not simply add a "disableaddr" type of message now?  I am not opposed
to someone championing an addr relay protocol, but I personally think it is
premature.  I'm not aware of any research or shared understanding of what
the goals of address relay are or ought to be, particularly as they pertain
to how addresses for obscure networks ought to propagate to nodes that may
or may not support those 

[bitcoin-dev] Proposal for new "disabletx" p2p message

[Suhas Daftuar via bitcoin-dev]

Hi,

I'm proposing the addition of a new, optional p2p message to allow peers to
communicate that they do not want to send or receive (loose) transactions
for the lifetime of a connection.

The goal of this message is to help facilitate connections on the network
over which only block-related data (blocks/headers/compact blocks/etc) are
relayed, to create low-resource connections that help protect against
partition attacks on the network.  In particular, by adding a network
message that communicates that transactions will not be relayed for the
life of the connection, we ease the implementation of software that could
have increased inbound connection limits for such peers, which in turn will
make it easier to add additional persistent block-relay-only connections on
the network -- strengthening network security for little additional
bandwidth.

Software has been deployed for over a year now which makes such
connections, using the BIP37/BIP60 "fRelay" field in the version message to
signal that transactions should not be sent initially.  However, BIP37
allows for transaction relay to be enabled later in the connection's
lifetime, complicating software that would try to distinguish inbound peers
that will never relay transactions from those that might.

This proposal would add a single new p2p message, "disabletx", which (if
used at all) must be sent between version and verack.  I propose that this
message is valid for peers advertising protocol version 70017 or higher.
Software is free to implement this BIP or ignore this message and remain
compatible with software that does implement it.

Full text of the proposed BIP is below.

Thanks,
Suhas

---


  BIP: XXX
  Layer: Peer Services
  Title: Disable transaction relay message
  Author: Suhas Daftuar 
  Comments-Summary: No comments yet.
  Comments-URI:
  Status: Draft
  Type: Standards Track
  Created: 2020-09-03
  License: BSD-2-Clause


==Abstract==

This BIP describes a change to the p2p protocol to allow a node to tell a peer
that a connection will not be used for transaction relay, to support
block-relay-only connections that are currently in use on the network.

==Motivation==

For nearly the past year, software has been deployed[1] which initiates
connections on the Bitcoin network and sets the transaction relay field
(introduced by BIP 37 and also defined in BIP 60) to false, to prevent
transaction relay from occurring on the connection. Additionally, addr messages
received from the peer are ignored by this software.

The purpose of these connections is two-fold: by making additional
low-bandwidth connections on which blocks can propagate, the robustness of a
node to network partitioning attacks is strengthened.  Additionally, by not
relaying transactions and ignoring received addresses, the ability of an
adversary to learn the complete network graph (or a subgraph) is reduced[2],
which in turn increases the cost or difficulty to an attacker seeking to carry
out a network partitioning attack (when compared with having such knowledge).

The low-bandwidth / minimal-resource nature of these connections is currently
known only by the initiator of the connection; this is because the transaction
relay field in the version message is not a permanent setting for the lifetime
of the connection.  Consequently, a node receiving an inbound connection with
transaction relay disabled cannot distinguish between a peer that will never
enable transaction relay (as described in BIP 37) and one that will.  Moreover,
the node also cannot determine that the incoming connection will ignore relayed
addresses; with that knowledge a node would likely choose other peers to
receive announced addresses instead.

This proposal adds a new, optional message that a node can send a peer when
initiating a connection to that peer, to indicate that connection should not be
used for transaction-relay for the connection's lifetime. In addition, without
a current mechanism to negotiate whether addresses should be relayed on a
connection, this BIP suggests that address messages not be sent on links where
tx-relay has been disabled.

==Specification==

# A new disabletx message is added, which is defined as an empty
message where pchCommand == "disabletx".
# The protocol version of nodes implementing this BIP must be set to
70017 or higher.
# If a node sets the transaction relay field in the version message to
a peer to false, then the disabletx message MAY also be sent in
response to a version message from that peer if the peer's protocol
version is >= 70017. If sent, the disabletx message MUST be sent prior
to sending a verack.
# A node that has sent or received a disabletx message to/from a peer
MUST NOT send any of these messages to the peer:
## inv messages for transactions
## getdata messages for transactions
## getdata messages for merkleblock (BIP 37)
## filteradd/filterload/filterclear (BIP 37)
## mempool (BIP 35)
# It is 

Re: [bitcoin-dev] A Replacement for RBF and CPFP: Non-Destructive TXID Dependencies for Fee Sponsoring

[Suhas Daftuar via bitcoin-dev]

Hi,

I think the topic of how to improve transaction relay policy and fee
bumping is an important one that needs to be worked on, so I'm glad
this is a topic of discussion.  However I am pretty skeptical of this
consensus change proposal:

The Sponsor Vector TXIDs must also be in the block the transaction is
> validated in, with no restriction on order or on specifying a TXID more
> than once.


That means that if a transaction is confirmed in a block without its
sponsor, the sponsor is no longer valid.  This breaks a design principle
that has been discussed many times over the years, which is that once a
valid transaction is created, it should not become invalid later on unless
the inputs are double-spent.  This principle has some logical consequences
that we've come to accept, such as transaction chains being valid across
small reorgs in the absence of malicious (double-spend) behavior.

I think that this principle is a useful one and that there should be a high
bar for doing away with it.  And it seems to me that this proposal doesn't
clear that bar -- the fee bumping improvement that this proposal aims at is
really coming from the policy change, rather than the consensus change. But
if policy changes are the direction we're going to solve these problems, we
could instead just propose new policy rules for the existing types of
transaction chaining that we have, rather than couple them to a new
transaction type.

My understanding of the main benefit of this approach is that this allows
3rd parties to participate in fee bumping.  But that behavior strikes me as
also problematic, because it introduces the possibility of 3rd party
griefing, to the extent that sponsor transactions in any way limit chains
of transactions that would be otherwise permitted.  If Alice sends Bob some
coins, and Alice and Bob are both honest and cooperating, Mallory shouldn't
be able to interfere with their low-feerate transaction by (eg) pinning it
with a large transaction that "sponsors" it (ie a large transaction that is
just above the feerate of the parent, which prevents additional child
transactions and makes it more expensive to RBF).

This last issue of pinning could be improved in this proposal by requiring
that a sponsor transaction bring the effective feerate of its package up to
something which should be confirmed soon (rather than just being a higher
feerate than the tx it is sponsoring).  However, we could also carve out a
policy rule just like that today, without any consensus changes needed, to
help with pinning (which is probably a good idea!  I think this would be
useful work).  So I don't think that approaches in that direction would be
unique to this proposal.

We allow one Sponsor to replace another subject to normal replacement
> policies, they are treated as conflicts.


This policy rule of allowing sponsor transactions to RBF each other also
seems problematic; that means that if Alice is paying Bob in a transaction
that is also sponsoring some other transaction (perhaps from Alice to
someone else), then Mallory can cause the transaction going to Bob to
become invalid by RBF bumping it and sponsoring the parent transaction
herself?  Allowing 3rd parties to interfere with transactions between
others seems like a complex and undesirable design to introduce.

In summary: this proposal seems like a CPFP replacement, requiring many
policy rules along with a consensus change to be worked out to get right; I
think we could achieve largely the same effect by improving the current
policy rules to make CPFP work better without a consensus change.  And
while what is unique about this proposal is that it allows for 3rd parties
to attach themselves to the transaction graph of other parties, I think
that is a complex interaction to introduce and has negative side effects as
well.



On Mon, Sep 21, 2020 at 12:27 PM Jeremy via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Responses Inline:
>
> Would it make sense that, instead of sponsor vectors
>> pointing to txids, they point to input outpoints?  E.g.:
>>
>> 1. Alice and Bob open a channel with funding transaction 0123...cdef,
>>output 0.
>>
>> 2. After a bunch of state updates, Alice unilaterally broadcasts a
>>commitment transaction, which has a minimal fee.
>>
>> 3. Bob doesn't immediately care whether or not Alice tried to close the
>>channel in the latest state---he just wants the commitment
>>transaction confirmed so that he either gets his money directly or he
>>can send any necessary penalty transactions.  So Bob broadcasts a
>>sponsor transaction with a vector of 0123...cdef:0
>>
>> 4. Miners can include that sponsor transaction in any block that has a
>>transaction with an input of 0123...cdef:0.  Otherwise the sponsor
>>transaction is consensus invalid.
>>
>> (Note: alternatively, sponsor vectors could point to either txids OR
>> input outpoints.  This complicates the serialization of the vector but
>> seems 

Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup

[Suhas Daftuar via bitcoin-dev]

Hi all,

Thanks for the helpful discussion.

My primary motivation in starting this thread was to establish what the
expectations are for new feature deployment (particularly whether the
protocol version should continue to be bumped or not), and I think I have
that answer -- different from what I proposed when I started this thread,
but not in a way that I think meaningfully hinders future work.  So I'm
happy to leave it at that and withdraw my suggestion.

Cheers,
Suhas


On Sun, Aug 23, 2020 at 1:51 PM Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> > On Aug 21, 2020, at 15:16, Matt Corallo 
> wrote:
> >
> > Hmm, could that not be accomplished by simply building this into new
> messages? eg, send "betterprotocol", if you see a verack and no
> "betterprotocol" from your peer, send "worseprotocol" before you send a
> "verack".
> >
> > Matt
> >
> >> On 8/21/20 5:17 PM, Jeremy wrote:
> >> As for an example of where you'd want multi-round, you could imagine a
> scenario where you have a feature A which gets bugfixed by the introduction
> of feature B, and you don't want to expose that you support A unless you
> first negotiate B. Or if you can negotiate B you should never expose A, but
> for old nodes you'll still do it if B is unknown to them.
>
> This seems to imply a security benefit (I can’t discern any other
> rationale for this complexity). It should be clear that this is no more
> than trivially weak obfuscation and not worth complicating the protocol to
> achieve.
>
> >> An example of this would be (were it not already out without a feature
> negotiation existing) WTXID/TXID relay.
> >> The SYNC primitve simply codifies what order messages should be in and
> when you're done for a phase of negotiation offering something. It can be
> done without, but then you have to be more careful to broadcast in the
> correct order and it's not clear when/if you should wait for more time
> before responding.
> >> On Fri, Aug 21, 2020 at 2:08 PM Jeremy  jlru...@mit.edu>> wrote:
> >>Actually we already have service bits (which are sadly limited)
> which allow negotiation of non bilateral feature
> >>support, so this would supercede that.
> >>--
> >>@JeremyRubin <
> https://twitter.com/JeremyRubin>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup

[Suhas Daftuar via bitcoin-dev]

Hi Eric,

Thanks for your response.  If I understand correctly, you're suggesting
that in the future we do the same as what was done in BIP 339, of
accompanying new messages (which are optional) with a protocol version
bump, so that network clients are never reading unknown messages from a
peer (and can be free to disconnect a peer for sending an unknown message)?

I think that works fine, so if indeed there will be software that will
expect things to operate this way then I can withdraw the suggestion I've
made in this thread.  However I wanted to clarify that this is what you
suggest, because there is another downside to this approach (beyond the
sequential nature of sequence numbers that you mention) -- if a software
implementation misses a proposed new protocol upgrade, and thus fails to
parse (and ignore) some proposed new message, the result can be a network
split down the road as incompatible clients get slowly upgraded over time.

I think this coordination cost is something to be concerned about -- for
instance, the lack of response to my wtxid-relay proposal made me wonder if
other software would be implementing something to account for the new
message that proposal introduces (for clients with version >= 70016).  It's
reasonable for people to be busy and miss things like this, and I think
it's worth considering whether there's a safer way for us to deploy changes.

That said, I don't think this coordination cost is unbearable, so as long
as we have a process for making p2p protocol improvements I'm not too
worried about what mechanism we use.  So if this concern over coordination
of changes doesn't sway you, I think we can continue to just bump protocol
version at the same time as deploying new messages, as we have been doing,
and hope that we don't run into problems down the road.

If I have misunderstood how you think we should be making future protocol
changes, please let me know.

Thanks,
Suhas



On Sun, Aug 16, 2020 at 3:06 PM Eric Voskuil  wrote:

> A requirement to ignore unknown (invalid) messages is not only a protocol
> breaking change but poor protocol design. The purpose of version
> negotiation is to determine the set of valid messages. Changes to version
> negotiation itself are very problematic.
>
> The only limitation presented by versioning is that the system is
> sequential. As such, clients that do not wish to implement (or operators
> who do not wish to enable) them are faced with a problem when wanting to
> support later features. This is resolvable by making such features optional
> at the new protocol level. This allows each client to limit its
> communication to the negotiated protocol, and allows ignoring of known but
> unsupported/disabled features.
>
> Sorry I missed your earlier post. Been distracted for a while.
>
> e
>
>
> On Aug 14, 2020, at 12:28, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> 
> Hi,
>
> Back in February I posted a proposal for WTXID-based transaction relay[1]
> (now known as BIP 339), which included a proposal for feature negotiation
> to take place prior to the VERACK message being received by each side.  In
> my email to this list, I had asked for feedback as to whether that proposal
> was problematic, and didn't receive any responses.
>
> Since then, the implementation of BIP 339 has been merged into Bitcoin
> Core, though it has not yet been released.
>
> In thinking about the mechanism used there, I thought it would be helpful
> to codify in a BIP the idea that Bitcoin network clients should ignore
> unknown messages received before a VERACK.  A draft of my proposal is
> available here[2].
>
> I presume that software upgrading past protocol version 70016 was already
> planning to either implement BIP 339, or ignore the wtxidrelay message
> proposed in BIP 339 (if not, then this would create network split concerns
> in the future -- so I hope that someone would speak up if this were a
> problem).  When we propose future protocol upgrades that would benefit from
> feature negotiation at the time of connection, I think it would be nice to
> be able to use the same method as proposed in BIP 339, without even needing
> to bump the protocol version.  So having an understanding that this is the
> standard of how other network clients operate would be helpful.
>
> If, on the other hand, this is problematic for some reason, I look forward
> to hearing that as well, so that we can be careful about how we deploy
> future p2p changes to avoid disruption.
>
> Thanks,
> Suhas Daftuar
>
> [1]
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html
>
> [2]
> https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.me

[bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup

[Suhas Daftuar via bitcoin-dev]

Hi,

Back in February I posted a proposal for WTXID-based transaction relay[1]
(now known as BIP 339), which included a proposal for feature negotiation
to take place prior to the VERACK message being received by each side.  In
my email to this list, I had asked for feedback as to whether that proposal
was problematic, and didn't receive any responses.

Since then, the implementation of BIP 339 has been merged into Bitcoin
Core, though it has not yet been released.

In thinking about the mechanism used there, I thought it would be helpful
to codify in a BIP the idea that Bitcoin network clients should ignore
unknown messages received before a VERACK.  A draft of my proposal is
available here[2].

I presume that software upgrading past protocol version 70016 was already
planning to either implement BIP 339, or ignore the wtxidrelay message
proposed in BIP 339 (if not, then this would create network split concerns
in the future -- so I hope that someone would speak up if this were a
problem).  When we propose future protocol upgrades that would benefit from
feature negotiation at the time of connection, I think it would be nice to
be able to use the same method as proposed in BIP 339, without even needing
to bump the protocol version.  So having an understanding that this is the
standard of how other network clients operate would be helpful.

If, on the other hand, this is problematic for some reason, I look forward
to hearing that as well, so that we can be careful about how we deploy
future p2p changes to avoid disruption.

Thanks,
Suhas Daftuar

[1]
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html

[2]
https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] A proposal for WTXID-based transaction relay

[Suhas Daftuar via bitcoin-dev]

Hi all,

I've been working on a proposal to add support for relaying transactions
based on their wtxid, rather than just their txid.  The current draft is at
https://github.com/sdaftuar/bips/blob/2020-02-wtxid-relay/bip-wtxid-relay.mediawiki,
and for some background I'll paste the motivation section here:

Historically, the INV messages sent on the Bitcoin peer-to-peer network to
> announce transactions refer to transactions by their txid, which is a hash
> of the transaction that does not include the witness (see BIP 141). This
> has been the case even since Segregated Witness (BIP 141/143/144) has been
> adopted by the network.
>


> Not committing to the witness in transaction announcements creates
> inefficiencies: because a transaction's witness can be malleated without
> altering the txid, a node in receipt of a witness transaction that the node
> does not accept will generally still download that same transaction when
> announced by other peers. This is because the alternative -- of not
> downloading a given txid after rejecting a transaction with that txid --
> would allow a third party to interfere with transaction relay by malleating
> a transaction's witness and announcing the resulting invalid transaction to
> nodes, preventing relay of the valid version of the transaction as well.
>


> We can eliminate this concern by using the wtxid in place of the txid when
> announcing and fetching transactions.
>

One point specifically that I'm seeking feedback on is feature negotiation:
for efficiency, I think it makes sense for peers to negotiate at the
beginning of a connection whether they are going to use wtxid- or
txid-based, prior to announcing any transactions.  To achieve this, I
propose in the BIP to send a message between receiving a VERSION message
and prior to sending VERACK (to nodes advertising version at least 70016)
to announce support for this new feature; if both sides send it then they
each know to enable it on the link.  My thinking is that in general, it'd
be great to use messages sent between VERSION and VERACK to negotiate
features prior to fully initializing a peer connection (it's sort of a
natural way to extend what we might want to send in a VERSION message,
without breaking existing VERSION-message parsers).  However, I don't know
whether inserting a message before VERACK would break any assumptions of
other software on the network, or if this is a problematic paradigm for
some reason, so I'd welcome feedback here.

Thanks,
Suhas
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] [BIP Proposal] P2SH and Version 0 Segwit Script enforcement from genesis

[Suhas Daftuar via bitcoin-dev]

Hi,

I propose backdating the P2SH and Segwit version 0 script rules back to the
genesis block, as a way to simplify these consensus rules.  Here's the
abstract from a draft BIP I wrote up to explain this change:

The Pay to Script Hash (P2SH, BIP 16) script rules and the Version 0
Witness Program script rules (BIP 143/141) can be enforced from the genesis
block with only one historical exception. Doing so simplifies consensus
rules and allows protocol implementers to avoid writing and testing code
paths that are no longer relevant.

The full BIP draft can be found here:
https://github.com/sdaftuar/bips/blob/p2sh-v0segwit-from-genesis/bip-sdaftuar-p2sh-v0segwit-from-genesis.mediawiki

And the currently open pull request to Bitcoin Core which implements this
change can be found here: https://github.com/bitcoin/bitcoin/pull/11739
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] "Compressed" headers stream

[Suhas Daftuar via bitcoin-dev]

Hi,

First, thanks for resurrecting this, I agree this is worth pursuing.

On Mon, Dec 11, 2017 at 4:04 PM, Gregory Maxwell via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> Nbits _never_ needs to be sent even with other consensus rules because
> its more or less necessarily a strict function of the prior headers.
> This still holds in every clone of Bitcoin I'm aware of; sending it
> with the first header in a group probably makes sense so it can be
> checked independently.
>

I think it would be nice, though, to not require the consensus-correct
calculation of nBits in order to process p2p messages.  For instance, I
think there's a use for nBits at the p2p layer for calculating the work on
a chain, which can be used as an anti-DoS measure, even without verifying
that the difficulty adjustments are following the consensus rules.

Moreover I think it's a bit messy if the p2p layer depends on intricate
consensus rules in order to reconstruct a message -- either we'd need to
interact with existing consensus logic in a potentially new way, or we'd
reimplement the same logic in the p2p layer, neither of which is very
desirable imo.

But I think we should be able to get nearly all the benefit just by
including nBits in any messages where the value is ambiguous; ie we include
it with the first header in a message, and whenever it changes from the
previous header's nBits.

I would rather not change the serialization of existing messages,
> nodes are going to have to support speaking both messages for a long
> time, and I think we already want a different protocol flow for
> headers fetching in any case.
>

I agree with this.  Specifically the way I envisioned this working is that
we could introduce a new 'cmpctheaders'/'getcmpcthdrs' message pair for
syncing using this new message type, while leaving the existing
'headers'/'getheaders' messages unchanged.  So when communicating with
upgraded peers, we'd never use 'getheaders' messages, and we'd only use
'headers' messages for potentially announcing new blocks.

Of course, we'll have to support the existing protocol for a long time.
But one downside I've discovered from deploying BIP 130, which introduced
block announcements via headers messages, is that overloading a 'headers'
message to be either a block announcement or a response to a 'getheaders'
message results in p2p-handling logic which is more complicated than it
needs to be.  So splitting off the headers chain-sync functionality to a
new message pair seems like a nice side-effect benefit, in the long run.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] I do not support the BIP 148 UASF

[Suhas Daftuar via bitcoin-dev]

I also do not support the BIP 148 UASF, and I'd like to add to the points
that Greg has already raised in this thread.

BIP 148 would introduce a new consensus rule that softforks out non-segwit
signalling blocks in some time period.  I reject this consensus rule as
both arbitrary and needlessly disruptive.  Bitcoin's primary purpose is to
reach consensus on the state of a shared ledger, and even though I think
the Bitcoin network ought to adopt segwit, I don't think that concern
trumps the goal of not splitting the network.

Many BIP 148 advocates seem to start with the assumption that segwit
already has a lot of support, and suggest that BIP 148 does as well.
However I don't think it's fair or correct to separate the activation
proposal for segwit from the rest of the segwit proposal.  The deployment
parameters for segwit are consensus-critical; assuming that some other
deployment has consensus because it would result in the rest of the segwit
proposal activating is an unjustified leap.

Even if there were no feasible alternate segwit deployment method available
to us, I would hesitate to recommend that the network adopt a potentially
consensus-splitting approach, even though I firmly believe that the ideas
behind segwit are fundamentally good ones.  But fortunately that is not the
situation we are in; we have substantially less disruptive methods
available to us to activate it, even if the current BIP 9 deployment were
to fail -- such as another BIP 9 deployment in the future, or perhaps a BIP
149 deployment.

If we do pursue a "user-activated" deployment of segwit, I'd recommend that
we do so in a more careful way than BIP 148 or 149 currently suggest, which
as I understand would otherwise make very few changes to the current
implementation.  However, due to the BIP 9 activation assumption, the
Bitcoin Core 0.13.1 - 0.14.0 segwit implementation largely lumps together
the idea that miners would both enforce the rules and mine segwit blocks.
However, we can separate these concerns, as we started to do in the Bitcoin
Core 0.14.1 release, where mining segwit blocks is not required in order to
generally mine or signal for segwit in the software.  And we can go further
still: without too much work, we could make further improvements to
accommodate miners who, for whatever reason, don't want to upgrade their
systems, such as by improving block relay from pre-segwit peers [1], or
optimizing transaction selection for miners who are willing to enforce the
segwit rules but haven't upgraded their systems to mine segwit blocks [2].

If we would seek to activate a soft-fork with less clear miner signaling
(such as BIP 149), then I think such improvements are warranted to minimize
network disruption.  In general, we should not seek to censor hashpower on
the network unless we have a very important reason for doing so.  While the
issues here are nuanced, if I were to evaluate the BIP 148 soft-fork
proposal on the spectrum of "censorship attack on Bitcoin" to "benign
protocol upgrade", BIP 148 strikes me as closer to the former than the
latter.  There is simply no need here to orphan these non-signalling blocks
that could otherwise be used to secure the network.

To go further: I think BIP 148 is ill-conceived even for achieving its own
presumed goals -- the motivation for adding a consensus rule that applies
to the version bits on blocks is surely for the effect such bits have on
older software, such as Bitcoin Core releases 0.13.1 and later.  Yet in
trying to bring those implementations along as segwit-enforcing software,
BIP 148 would risk forking from such clients in the short term!  If one
really cared about maintaining consensus with older, segwit-enabled
software, it would make far more sense to seek segwit activation in a way
that didn't fork from them (such as BIP 149, or a new BIP 9 deployment
after this one times out).  And if one doesn't care about such consensus,
then it'd be far simpler to just set (e.g.) August 1 as the flag day
activation of segwit, and not play these contortionist games with block
version bits, which carry no useful or intrinsic meaning.  Either of these
two approaches should have the advantage of reduced fork risk, compared
with BIP 148.

Of course, everyone is free to run the software of their choosing.  I write
this to both generally convey my opposition to a careless proposal, which I
believe represents a way of thinking that is detrimental to Bitcoin's long
run success, and specifically explain why I oppose inclusion of this
proposal in the Bitcoin Core implementation [3].  The Bitcoin Core project
hasn't been, and shouldn't be, careless in deploying consensus changes.
Instead, I think the Bitcoin Core project ought to stand up for the best
practices that our community has learned about how to deploy such changes
(specifically for minimizing chain-split risk when deploying a soft fork!),
and I think we should all avoid adoption or encouragement of practices that
would depart from 

Re: [bitcoin-dev] TXMempool and dirty entries

[Suhas Daftuar via bitcoin-dev]

Hi,

I've moved the bitcoin-dev list to bcc:, as this question is better suited
to forums dedicated to Bitcoin Core implementation specifics, rather than
the general bitcoin development list.

Please feel free in the future to ask questions like this on the
bitcoin-core-dev mailing list (https://lists.linuxfoundation
.org/mailman/listinfo/bitcoin-core-dev) or on the #bitcoin-core-dev
freenode IRC channel.

The work limit (that was put in place in https://github.com/bitcoin/
bitcoin/pull/6654, when the concept of "dirty" entries was introduced) was
removed in https://github.com/bitcoin/bitcoin/pull/7594, in preparation for
ancestor-feerate-mining.  So those comments should have been cleaned up to
match the new code.

Please feel free to file an issue or open a PR to update those comments at
https://github.com/bitcoin/bitcoin.

Thanks,
Suhas


On Mon, May 8, 2017 at 5:38 AM, DJ Bitcoin via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi Guys,
>
> I have a question about the use of txmempool. find attached the code in
> txmempool.h
>
>
> ==
> /* Adding transactions from a disconnected block can be very time
> consuming,
>  * because we don't have a way to limit the number of in-mempool
> descendants.
>  * To bound CPU processing, we limit the amount of work we're willing to do
>  * to properly update the descendant information for a tx being added from
>  * a disconnected block.  If we would exceed the limit, then we instead
> mark
>  * the entry as "dirty", and set the feerate for sorting purposes to be
> equal
>  * the feerate of the transaction without any descendants. */
>
> class CTxMemPoolEntry
> {
>private:
>// ...
>// Information about descendants of this transaction that are in the
>// mempool; if we remove this transaction we must remove all of these
>// descendants as well. if nCountWithDescendants is 0, treat this entry
> as
>// dirty, and nSizeWithDescendants and nModFeesWithDescendants will not
> be
>// correct.
>
>int64_t nCountWithDescendants; //!< number of descendant transactions
>// ...
> ==
>
>
> Now, the only place where nCountWithDescendants is modified is the
> following (txmempool.cpp):
>
>
> ==
> void CTxMemPoolEntry::UpdateDescendantState(int64_t modifySize, CAmount
> modifyFee, int64_t modifyCount)
> {
> nSizeWithDescendants += modifySize;
> assert(int64_t(nSizeWithDescendants) > 0);
> nModFeesWithDescendants += modifyFee;
> nCountWithDescendants += modifyCount;
> assert(int64_t(nCountWithDescendants) > 0);
> }
> ==
>
>
> Therefore, nCountWithDescendants is never zero.
> Am i missing something? Where is this concept of "dirty" defined?
>
> Thanks a lot,
> DJ
>
>
>
>
>
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] Segregated witness p2p layer compatibility

[Suhas Daftuar via bitcoin-dev]

Hi,

There have been two threads recently that have made references to
peer-to-peer implementation details in Bitcoin Core's Segregated Witness
code that I would like to clarify.

In the thread "Issolated Bitcoin Nodes" (
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-March/013765.html),
there was some discussion about how Bitcoin Core's block download logic
behaves after segwit activation.  After segwit activation, Bitcoin Core
nodes will not (currently) attempt to download any blocks from non-segwit
peers (nodes that do not set the NODE WITNESS service bit).  This is a
bandwidth optimization to prevent a node from downloading a block that may
be invalid only because the sender omitted the witness, requiring
re-download until the block is received with the required witness data.

But to be clear, non-segwit blocks -- that is, blocks without a witness
commitment in the coinbase, and whose transactions are serialized without
witnesses, and whose transactions are not spending segwit outputs which
require a witness -- are evaluated under the same rules as prior,
pre-segwit versions of the software.  So such non-segwit blocks that are
valid to older, pre-segwit nodes are also valid to segwit-nodes.

In
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-March/013796.html,
Eric Voskuil wrote:

Given the protocol requirements of the segwit proposal this is not the
> case. A miner running pre-segwit code will produce blocks that no
> segwit node will ever receive.


The phrase "protocol requirements of segwit" is confusing here, because
there are two different layers that need consideration: the consensus
protocol layer and the peer-to-peer protocol layer.  But in neither layer
is the behavior of not downloading blocks from non-NODE WITNESS peers a
"requirement".  This is an implementation detail in the Bitcoin Core code
that alternate implementations compliant with BIP 144 could implement
differently.

At the consensus layer, non-segwit blocks (described above) that are valid
to older nodes are also valid to segwit nodes.  That means that if a miner
was using an older, pre-segwit version of Bitcoin Core to produce blocks
after segwit activates, that blocks they find will be valid to all nodes.

At the p2p layer, though, segwit-enabled Bitcoin Core nodes will only try
to download those blocks if announced by a segwit-enabled peer.  But this
is not a protocol requirement; other implementations can remain compatible
even they take different approaches here.  (As an example, I could imagine
an implementation that downloaded a new block from any peer, but if the
block has a witness commitment in the coinbase and was received from a
non-segwit peer, then the node would attempt re-download from a segwit
peer.  I'm sure many other reasonable block download strategies could be
devised.)

Still, if a miner wants to continue mining post-segwit activation, but
using pre-segwit software, they would need a way to relay their blocks to a
segwit-enabled peer.

There are a few ways to do this that I can think of:

- Use the RPC call "submitblock" on a segwit-enabled node.  Calling
"submitblock" on a Bitcoin Core 0.13.1 (0.13.0 in the case of testnet) or
later node works fine as long as the block is valid (whether or not it has
a witness commitment or witness transactions), and once a segwit-enabled
peer has the block it will relay to other segwit peers.

- Explicitly deliver the block to a segwit node over the p2p network, even
if unrequested.  Currently Bitcoin Core at least will process unrequested
blocks, and valid blocks that update the tip will then be relayed to other
peers.

- Run a bridge node, which advertises NODE_WITNESS and can serialize blocks
with witness data, which downloads blocks even from non-NODE WITNESS
peers.  Anyone can do this to bridge the networks for the benefit of the
whole network (I have personally been running a few nodes that do this, for
several months now), but miners concerned about this issue for their own
blocks could explicitly do this themselves to ensure that their own blocks
propagate to the segwit-enabled network.

- Peer directly with other miners, bypassing the p2p network.  Many miners
do this already, using protocols which may already serve to bridge the
network.

So saying that "A miner running pre-segwit code will produce blocks that no
segwit node will ever receive" is not really correct, in my view.  If the
whole network were just running Bitcoin Core software releases, and the
miner was not able/willing to deliver their block to a segwit-enabled node
(eg by using the RPC call "submitblock", or one of the other suggestions I
had above), then I would agree with the statement.  But given that there
are bridge nodes on the network, and that miners have other options to
relay their block, I think this is not an accurate portrayal of what would
actually happen on the network -- I would expect that non-segwit miners'
blocks would still get relayed post-segwit 

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Suhas Daftuar via bitcoin-dev]

Just want to clarify two points:

This change has not yet appeared in any released software (but I assume it
will be in the next release, 0.14.0).

I agree that the performance optimization is not the point of this change;
I can modify the BIP draft to de-emphasize that further (perhaps remove
mention of it entirely).

On Mon, Nov 14, 2016 at 1:47 PM, Eric Voskuil <e...@voskuil.org> wrote:

> NACK
>
> Horrible precedent (hardcoding rule changes based on the assumption that
> large forks indicate a catastrophic failure), extremely poor process
> (already shipped, now the discussion), and not even a material performance
> optimization (the checks are avoidable once activated until a sufficiently
> deep reorg deactivates them).
>
> e
>
> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Hi,
>
> Recently Bitcoin Core merged a simplification to the consensus rules
> surrounding deployment of BIPs 34, 66, and 65 (https://github.com/bitcoin/
> bitcoin/pull/8391), and though the change is a minor one, I thought it
> was worth documenting the rationale in a BIP for posterity.
>
> Here's the abstract:
>
> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
> signaling in block version numbers. Now that the chain has long since
> passed the blocks at which those consensus rules have triggered, we can (as
> a simplification and optimization) replace the trigger mechanism by caching
> the block heights at which those consensus rules became enforced.
>
> The full draft can be found here:
>
> https://github.com/sdaftuar/bips/blob/buried-deployments/
> bip-buried-deployments.mediawiki
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar via bitcoin-dev]

Hi,

I'm proposing the addition of a new, optional p2p message to help improve
the way blocks are announced on the network.  The draft BIP is available
here and pasted below:
https://gist.github.com/sdaftuar/465bf008f0a4768c0def

The goal of this p2p message is to facilitate nodes being able to
optionally announce blocks with headers messages rather than with inv's,
which is particularly beneficial since the introduction of headers-first
download in Bitcoin Core 0.10.  In particular, this allows for more
efficient propagation of reorgs as it would eliminate a round trip in
network communication.

The implementation of this BIP (which includes code to directly fetch
blocks based on announced headers) is in
https://github.com/bitcoin/bitcoin/pull/6494.  For additional background,
please also see https://github.com/bitcoin/bitcoin/issues/5982.

Note that this new p2p message is optional; nodes can feel free to ignore
and continue to use inv messages to announce new blocks.

Thanks to Pieter Wuille for suggesting this idea.

Draft BIP text:


  BIP: 
  Title: sendheaders message
  Author: Suhas Daftuar 
  Status: Draft
  Type: Standards Track
  Created: 2015-05-08


==Abstract==

Add a new message, "sendheaders", which indicates that a node prefers to
receive new block announcements via a "headers" message rather than an
"inv".

==Motivation==

Since the introduction of "headers-first" downloading of blocks in 0.10,
blocks will not be processed unless
they are able to connect to a (valid) headers chain.  Consequently, block
relay generally works
as follows:
# A node (N) announces the new tip with an "inv" message, containing the
block hash
# A peer (P) responds to the "inv" with a "getheaders" message (to request
headers up to the new tip) and a "getdata" message for the new tip itself
# N responds with a "headers" message (with the header for the new block
along with any preceding headers unknown to P) and a "block" message
containing the new block

However, in the case where a new block is being announced that builds on
the tip, it would be generally more efficient if the node N just announced
the block header for the new block, rather than just the block hash, and
saved the peer from generating and transmitting the getheaders message (and
the required block locator).

In the case of a reorg, where 1 or more blocks are disconnected, nodes
currently just send an "inv" for the new tip.  Peers currently are able to
request the new tip immediately, but wait until the headers for the
intermediate blocks are delivered before requesting those blocks.  By
announcing headers from the last fork point leading up to the new tip in
the block announcement, peers are able to request all the intermediate
blocks immediately.

==Specification==

# The sendheaders message is defined as an empty message where pchCommand
== "sendheaders"
# Upon receipt of a "sendheaders" message, the node will be permitted, but
not required, to announce new blocks by sending the header of the new block
(along with any other blocks that a node believes a peer might need in
order for the block to connect).
# Feature discovery is enabled by checking protocol version >= 70012

==Backward compatibility==

Older clients remain fully compatible and interoperable after this change.

==Implementation==

https://github.com/bitcoin/bitcoin/pull/6494
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar via bitcoin-dev]

I considered that as well, but it seemed to me that other software on the
network (say, different wallet implementations) might prefer the option of
being able to bump up their protocol version in the future to pick up some
other change, without having to also opt-in to receiving
headers-announcements for blocks.

In particular, inv-based block announcements aren't going away (even in my
implementation of headers announcements, there are some edge cases where
the code would need to fall back to an inv announcement), so forcing all
software on the network to upgrade to supporting headers announcements,
whether now or in the future, seems too drastic -- I could imagine some
software not being very concerned about optimizing block relay in this
particular way.

On Thu, Sep 24, 2015 at 2:41 PM, Peter Todd <p...@petertodd.org> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
>
>
> On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
> >On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
> >bitcoin-dev@lists.linuxfoundation.org> wrote:
> >
> >>
> >> Is there actually a requirement for the new message?  New nodes could
> >just
> >> unilaterally switch to sending headers and current nodes would be
> >> compatible.
> >>
> >
> >I don't believe that unilaterally switching to headers announcements
> >would
> >work for all network participants -- both for users running older
> >Bitcoin
> >Core versions (anything before 0.10, which I believe all ignore headers
> >messages) and for non-Bitcoin Core software that participates on the
> >network (which may ignore headers messages too, I'm not sure what all
> >is
> >out there).
>
> You can enable the behaviour based on advertised p2p network version.
> -BEGIN PGP SIGNATURE-
>
> iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5
> AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj
> djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE
> LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n
> 3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU
> Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR
> V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA=
> =cF4B
> -END PGP SIGNATURE-
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev