Re: [Bitcoin-development] F2Pool has enabled full replace-by-fee

[Stephen Morse]

It is disappointing that F2Pool would enable full RBF when the safe
alternative, first-seen-safe RBF, is also available, especially since the
fees they would gain by supporting full RBF over FSS RBF would likely be
negligible. Did they consider using FSS RBF instead?

Best,
Stephen

On Fri, Jun 19, 2015 at 6:39 AM, Peter Todd  wrote:

> Yesterday F2Pool, currently the largest pool with 21% of the hashing
> power, enabled full replace-by-fee (RBF) support after discussions with
> me. This means that transactions that F2Pool has will be replaced if a
> conflicting transaction pays a higher fee. There are no requirements for
> the replacement transaction to pay addresses that were paid by the
> previous transaction.
>
>
> I'm a user. What does this mean for me?
> ---
>
> In the short term, very little. Wallet software aimed at average users
> has no ability to reliably detect conditions where an unconfirmed
> transaction may be double-spent by the sender. For example, Schildbach's
> Bitcoin Wallet for Android doesn't even detect double-spends of
> unconfirmed transactions when connected to a RBF or Bitcoin XT nodes
> that propagate them. The least sophisticated double-spend attack
> possibly - simply broadcasting two conflicting transactions at the same
> time - has about 50% probability of success against these wallets.
>
> Additionally, SPV wallets based on bitcoinj can't even detect invalid
> transactions reliably, instead trusting the full node(s) it is connected
> too over the unauthenticated, unencrypted, P2P protocol to do validation
> for them. For instance due to a unfixed bug¹ Bitcoin XT nodes will relay
> double-spends that spend the output of the conflicting transaction. I've
> personally tested this with Schildbach's Bitcoin Wallet for Android,
> which shows such invalid transactions as standard, unconfirmed,
> transactions.
>
> Users should continue to assume that unconfirmed transactions could be
> trivially reversed by the sender until the first confirmation. In
> general, only the sender can reverse a transaction, so if you do trust
> the sender feel free to assume an unconfirmed transaction will
> eventually confirm. However, if you do not trust the sender and/or have
> no other recourse if they double-spend you, wait until at least the
> first confirmation before assuming the transaction will go through.
>
> In the long term, miner support of full RBF has a number of advantages
> to users, allowing you to more efficiently make transactions, paying
> lower fees. However you'll need a wallet supporting these features; none
> exist yet.
>
>
> I'm a business. What does this mean for me?
> ---
>
> If you use your own node to verify transactions, you probably are in a
> similar situation as average users, so again, this means very little to
> you.
>
> If you use a payment processor/transaction API such as BitPay, Coinbase,
> BlockCypher, etc. you may or may not be accepting unconfirmed
> transactions, and they may or may not be "guaranteed" by your payment
> processor even if double-spent. If like most merchants you're using the
> API such that confirmations are required prior to accepting orders (e.g.
> taking a meaningful loss such as shipping a product if the tx is
> reversed) nothing changes for you. If not I recommend you contact your
> payment processor.
>
>
> I'm a miner. Why should I support replace-by-fee?
> -
>
> Whether full or first-seen-safe⁵ RBF support (along with
> child-pays-for-parent) is an important step towards a fully functioning
> transaction fee market that doesn't lead to users' transactions getting
> mysteriously "stuck", particularly during network flooding
> events/attacks. A better functioning fee market will help reduce
> pressure to increase the blocksize, particularly from the users creating
> the most valuable transactions.
>
> Full RBF also helps make use of the limited blockchain space more
> efficiently, with up to 90%+ transaction size savings possible in some
> transaction patterns. (e.g. long payment chains⁶) More users in less
> blockchain space will lead to higher overall fees per block.
>
> Finally as we'll discuss below full RBF prevents a number of serious
> threats to the existing level playing field that miners operate in.
>
>
> Why can't we make accepting unconfirmed txs from untrusted people safe?
> ---
>
> For a decentralized wallet, the situation is pretty bleak. These wallets
> only have a handful of connections to the network, with no way of
> knowing if those connections give an accurate view of what transactions
> miners actually know about.
>
> The only serious attempt to fix this problem for decentralized wallets
> that has been actually deployed is Andresen/Harding's double-spend
> relaying, implemented in Bitcoin XT. It relays up to one double-spend
>

Re: [Bitcoin-development] Ninki Wallet view on blocksize debate

[Stephen Morse]

Ben,

How does your wallet calculate the fee that should be paid to miners? Do
they automatically adjust when transactions take a long time to be
confirmed? And how does it respond when transactions are not mined
successfully, such as when blocks are full?

I strongly urge Gavin to withdraw from this standoff and work with the
> bitcoin core devs via the existing and successful bip process.
>

The BIP process has not resulted in any hard forks, so this is a little
different. While I don't like M&G's proposed solution of convincing miners
and services to switch to Bitcoin-XT, I recognize that it is done out of a
sense of urgency. These types of changes take a long time to roll out, and
we should start them before it is too late.

This whole debate comes down to: what is more risky, a consensus hard fork
or letting bitcoin exceed its imposed capacity limits? The former could
result in many services not being compatible and even loss of funds. The
latter could result in software failures, instability, and inability to
transact: essentially, what bitcoin is supposed to be good at. Both are
dangerous and could result in a significant loss of public confidence.

Something needs to be done, that's for sure. In the short term, I think we
need to do one of two things:

   1. All miners and wallet developers need to upgrade to support
   first-safe RBF, to allow for double spending one's own transactions when
   they lack sufficient fees to merit confirmations. Wallets also need to
   randomly request transactions from blocks to see what kind of fees are
   being paid to get confirmations, so that fees can be paid dynamically
   instead of with hard-coded values.
   2. We can implement either Gavin's or Jeff Garzik's proposal to change
   the consensus parameters around the block size limit.

So Ben, if really don't think that going with #2 is the right way to go
(even though everyone agrees that we will need to increase the block size
limit eventually anyway, why not now?), then I hope you start to work hard
on implementing #1 so that your wallet software can handle hitting capacity
limits gracefully.

Best,
Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Max Block Size: Simple Voting Procedure

[Stephen Morse]

Vincent,

> Some changes:
>
> Votes need to be 100%, not 50.01%. That way small miners have a fair
> chance. A 50.01% vote means large miners call the shots.
>
While I like the idea of possibly requiring more than 50%, you wouldn't
want to have a situation where a minority of uncooperative (or just lazy)
miners don't add their votes and hold up progress. Maybe 2/3 instead of
1/2, though.

> Users (people who make transactions) need to vote. A vote by a miner
> shouldn't count without user votes. Fee incentives should attract
> legitimate votes from miners. A cheating miner will be defeated by another
> miner who includes those votes, and take the fees.
>
> This lets wallet providers and exchanges cast votes (few wallets will
> implement prompts and will just auto vote, so if you don't agree, switch
> wallets. Vote with your wallet).
>
The idea of voting with your wallet, while appealing, is technically
infeasible. Miners can fill their blocks with any type of transactions,
including their own specially designed transactions. And any fees from
these transactions can be collected right back into their coinbase
transaction.

- Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Max Block Size: Simple Voting Procedure

[Stephen Morse]

Pindar,

yes and it's a good idea to separate the hard/soft fork upgrades. The point
> being here is that we're also establishing a process for the community to
> self-determine the way forward in a transparent and verifiable manner.
>
> What's not to like? :)
>
> I'll probably have some time on Sunday to help hack something up but I
> don't think this is that heavy a coding lift? What am I missing?
>

As Matt mentioned, many members of the bitcoin community would be hesitant
about giving miners this much power. It essentially lets them vote to
change the rules of the system. But miners are not the only part of this
ecosystem, and they are not the only ones affected by the choice of block
size limit, so they probably shouldn't be the only ones with a vote.
Instead, we vote with the software we run, and all upgrade.

So, while I think an idea like this has its merits, I would bet that it's
fairly unlikely to get enough support to be merged into bitcoin core.

Best,
Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Max Block Size: Simple Voting Procedure

[Stephen Morse]

>
> Why do it as an OP_RETURN output? It could be a simple token in the
> coinbase input script, similar to how support for P2SH was signaled among
> miners. And why should there be an explicit token for voting for the status
> quo? Simply omitting any indication should be an implicit vote for the
> status quo. A miner would only need to insert an indicator into their block
> if they wished for a larger block.
>

I don't really care the exact location it's put in. I just thought there
wasn't an explicit need to put it in the header (via a bit of nVersion),
and the scriptSig is already used for many things (block height, merged
mining hash, "\"P2SH\"", miner identifier). And voting to keep the block
size the same by not voting is fine by me.


> That said, proposals of this type have been discussed before, and the
> objection is always that miners would want larger blocks than the rest of
> the network could bear. Unless you want Bitcoin to become centralized in
> the hands of a few large mining pools, you shouldn't hand control over the
> block size limits to the miners.
>

Yeah, that was the conclusion we came to chatting on #bitcoin-wizards the
other day. I now think that this could be useful to dynamically increase a
lower limit, but that there should still be a hard upper limit like 20 MB.
I think that just changing the upper limit might be simpler and better,
though.

- Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] [BIP draft] Consensus-enforced transaction replacement signalled via sequence numbers

[Stephen Morse]

>
> That would also introduce the anomaly of a script that was once valid
> becoming later invalid, when nothing varies other than time.  That is
> not super compatible with the current model of reprocessing
> transactions in later blocks if the block they were first in gets
> reorged.
>

Very good point.


>
> (Not a huge flexibility loss as you can implement "not after" by
> making it the previous holders responsibility to spend a "not before"
> back to themselves.)
>

Do you mean something like the below?

scriptPubKey:
  IF
{A's pub} CHECKSIGVERIFY
  ELSE
{curr_height + 100} CLTV {B's pub} CHECKSIGVERIFY

This ensures that Alice has to spend the output in the next 100 blocks or
risk it being taken from her (she just has to put an OP_TRUE on the end of
her scriptSig). So, it seems we can forget about an inverted CLTV/CSV,
great!

Best,
Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] [BIP draft] Consensus-enforced transaction replacement signalled via sequence numbers

[Stephen Morse]

I see, so OP_SEQUENCEVERIFY will have a value pushed on the stack right
before, and then check that the input spending the prevout has nSequence
corresponds to at least the sequence specified by the stack value. Good
idea! Keeps the script code from depending on external chain specific data,
which is nice.

Hopefully we can repurpose one of the OP_NOPs for CHECKLOCKTIMEVERIFY and
one for OP_CHECKSEQUENCEVERIFY. Very complementary.

Best,
Stephen


On Tue, Jun 2, 2015 at 12:16 AM, Mark Friedenbach 
wrote:

> You are correct! I am maintaining a 'checksequenceverify' branch in my git
> repository as well, an OP_RCLTV using sequence numbers:
>
> https://github.com/maaku/bitcoin/tree/checksequenceverify
>
> Most of the interesting use cases for relative lock-time require an RCLTV
> opcode. What is interesting about this architecture is that it possible to
> cleanly separate the relative lock-time (sequence numbers) from the RCLTV
> opcode (OP_CHECKSEQUENCEVERIFY) both in concept and in implementation. Like
> CLTV, the CSV opcode only checks transaction data and requires no
> contextual knowledge about block headers, a weakness of the other RCLTV
> proposals that violate the clean separation between libscript and
> libconsensus. In a similar way, this BIP proposal only touches the
> transaction validation logic without any impact to script.
>
> I would like to propose an additional BIP covering the CHECKSEQUENCEVERIFY
> opcode and its enabling applications. But, well, one thing at a time.
>
> On Mon, Jun 1, 2015 at 8:45 PM, Stephen Morse  > wrote:
>
>> Hi Mark,
>>
>> Overall, I like this idea in every way except for one: unless I am
>> missing something, we may still need an OP_RCLTV even with this being
>> implemented.
>>
>> In use cases such as micropayment channels where the funds are locked up
>> by multiple parties, the enforcement of the relative locktime can be done
>> by the first-signing party. So, while your solution would probably work in
>> cases like this, where multiple signing parties are involved, there may be
>> other, seen or unforeseen, use cases that require putting the relative
>> locktime right into the spending contract (the scriptPubKey itself).
>> When there is only one signer, there's nothing that enforces using an
>> nSequence and nVersion=2 that would prevent spending the output until a
>> certain time.
>>
>> I hope this is received as constructive criticism, I do think this is an
>> innovative idea. In my view, though, it seems to be less fully-featured
>> than just repurposing an OP_NOP to create OP_RCLTV. The benefits are
>> obviously that it saves transaction space by repurposing unused space, and
>> would likely work for most cases where an OP_RCLTV would be needed.
>>
>> Best,
>> Stephen
>>
>> On Mon, Jun 1, 2015 at 9:49 PM, Mark Friedenbach 
>> wrote:
>>
>>> I have written a reference implementation and BIP draft for a soft-fork
>>> change to the consensus-enforced behaviour of sequence numbers for the
>>> purpose of supporting transaction replacement via per-input relative
>>> lock-times. This proposal was previously discussed on the mailing list in
>>> the following thread:
>>>
>>> http://sourceforge.net/p/bitcoin/mailman/message/34146752/
>>>
>>> In short summary, this proposal seeks to enable safe transaction
>>> replacement by re-purposing the nSequence field of a transaction input to
>>> be a consensus-enforced relative lock-time.
>>>
>>> The advantages of this approach is that it makes use of the full range
>>> of the 32-bit sequence number which until now has rarely been used for
>>> anything other than a boolean control over absolute nLockTime, and it does
>>> so in a way that is semantically compatible with the originally envisioned
>>> use of sequence numbers for fast mempool transaction replacement.
>>>
>>> The disadvantages are that external constraints often prevent the full
>>> range of sequence numbers from being used when interpreted as a relative
>>> lock-time, and re-purposing nSequence as a relative lock-time precludes its
>>> use in other contexts. The latter point has been partially addressed by
>>> having the relative lock-time semantics be enforced only if the
>>> most-significant bit of nSequence is set. This preserves 31 bits for
>>> alternative use when relative lock-times are not required.
>>>
>>> The BIP draft can be found at the following gist:
>>>
>>> https://gist.github.com/maaku/be15629fe64618b14f5a
>>>
>>> The reference implemen

Re: [Bitcoin-development] [BIP draft] Consensus-enforced transaction replacement signalled via sequence numbers

[Stephen Morse]

Hi Mark,

Overall, I like this idea in every way except for one: unless I am missing
something, we may still need an OP_RCLTV even with this being implemented.

In use cases such as micropayment channels where the funds are locked up by
multiple parties, the enforcement of the relative locktime can be done by
the first-signing party. So, while your solution would probably work in
cases like this, where multiple signing parties are involved, there may be
other, seen or unforeseen, use cases that require putting the relative
locktime right into the spending contract (the scriptPubKey itself). When
there is only one signer, there's nothing that enforces using an nSequence
and nVersion=2 that would prevent spending the output until a certain time.

I hope this is received as constructive criticism, I do think this is an
innovative idea. In my view, though, it seems to be less fully-featured
than just repurposing an OP_NOP to create OP_RCLTV. The benefits are
obviously that it saves transaction space by repurposing unused space, and
would likely work for most cases where an OP_RCLTV would be needed.

Best,
Stephen

On Mon, Jun 1, 2015 at 9:49 PM, Mark Friedenbach 
wrote:

> I have written a reference implementation and BIP draft for a soft-fork
> change to the consensus-enforced behaviour of sequence numbers for the
> purpose of supporting transaction replacement via per-input relative
> lock-times. This proposal was previously discussed on the mailing list in
> the following thread:
>
> http://sourceforge.net/p/bitcoin/mailman/message/34146752/
>
> In short summary, this proposal seeks to enable safe transaction
> replacement by re-purposing the nSequence field of a transaction input to
> be a consensus-enforced relative lock-time.
>
> The advantages of this approach is that it makes use of the full range of
> the 32-bit sequence number which until now has rarely been used for
> anything other than a boolean control over absolute nLockTime, and it does
> so in a way that is semantically compatible with the originally envisioned
> use of sequence numbers for fast mempool transaction replacement.
>
> The disadvantages are that external constraints often prevent the full
> range of sequence numbers from being used when interpreted as a relative
> lock-time, and re-purposing nSequence as a relative lock-time precludes its
> use in other contexts. The latter point has been partially addressed by
> having the relative lock-time semantics be enforced only if the
> most-significant bit of nSequence is set. This preserves 31 bits for
> alternative use when relative lock-times are not required.
>
> The BIP draft can be found at the following gist:
>
> https://gist.github.com/maaku/be15629fe64618b14f5a
>
> The reference implementation is available at the following git repository:
>
> https://github.com/maaku/bitcoin/tree/sequencenumbers
>
> I request that the BIP editor please assign a BIP number for this work.
>
> Sincerely,
> Mark Friedenbach
>
>
> --
>
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Why do we need a MAX_BLOCK_SIZE at all?

[Stephen Morse]

This exact question came up on the Bitcoin Stack Exchange once. I gave an
answer here:
http://bitcoin.stackexchange.com/questions/37292/whats-the-purpose-of-a-maximum-block-size/37303#37303

On Mon, Jun 1, 2015 at 2:32 PM, Jim Phillips  wrote:

> Ok, I understand at least some of the reason that blocks have to be kept
> to a certain size. I get that blocks which are too big will be hard to
> propagate by relays. Miners will have more trouble uploading the large
> blocks to the network once they've found a hash. We need block size
> constraints to create a fee economy for the miners.
>
> But these all sound to me like issues that affect some, but not others. So
> it seems to me like it ought to be a configurable setting. We've already
> witnessed with last week's stress test that most miners aren't even
> creating 1MB blocks but are still using the software defaults of 730k. If
> there are configurable limits, why does there have to be a hard limit?
> Can't miners just use the configurable limit to decide what size blocks
> they can afford to and are thus willing to create? They could just as
> easily use that to create a fee economy. If the miners with the most
> hashpower are not willing to mine blocks larger than 1 or 2 megs, then they
> are able to slow down confirmations of transactions. It may take several
> blocks before a miner willing to include a particular transaction finds a
> block. This would actually force miners to compete with each other and find
> a block size naturally instead of having it forced on them by the protocol.
> Relays would be able to participate in that process by restricting the
> miners ability to propagate large blocks. You know, like what happens in a
> FREE MARKET economy, without burdensome regulation which can be manipulated
> through politics? Isn't that what's really happening right now? Different
> political factions with different agendas are fighting over how best to
> regulate the Bitcoin protocol.
>
> I know the limit was originally put in place to prevent spamming. But that
> was when we were mining with CPUs and just beginning to see the occasional
> GPU which could take control over the network and maliciously spam large
> blocks. But with ASIC mining now catching up to Moore's Law, that's not
> really an issue anymore. No one malicious entity can really just take over
> the network now without spending more money than it's worth -- and that's
> just going to get truer with time as hashpower continues to grow. And it's
> not like the hard limit really does anything anymore to prevent spamming.
> If a spammer wants to create thousands or millions of transactions, a hard
> limit on the block size isn't going to stop him.. He'll just fill up the
> mempool or UTXO database instead of someone's block database.. And block
> storage media is generally the cheapest storage.. I mean they could be
> written to tape and be just as valid as if they're stored in DRAM. Combine
> that with pruning, and block storage costs are almost a non-issue for
> anyone who isn't running an archival node.
>
> And can't relay nodes just configure a limit on the size of blocks they
> will relay? Sure they'd still need to download a big block occasionally,
> but that's not really that big a deal, and they're under no obligation to
> propagate it.. Even if it's a 2GB block, it'll get downloaded eventually.
> It's only if it gets to the point where the average home connection is too
> slow to keep up with the transaction & block flow that there's any real
> issue there, and that would happen regardless of how big the blocks are. I
> personally would much prefer to see hardware limits act as the bottleneck
> than to introduce an artificial bottleneck into the protocol that has to be
> adjusted regularly. The software and protocol are TECHNICALLY capable of
> scaling to handle the world's entire transaction set. The real issue with
> scaling to this size is limitations on hardware, which are regulated by
> Moore's Law. Why do we need arbitrary soft limits? Why can't we allow
> Bitcoin to grow naturally within the ever increasing limits of our
> hardware? Is it because nobody will ever need more than 640k of RAM?
>
> Am I missing something here? Is there some big reason that I'm overlooking
> why there has to be some hard-coded limit on the block size that affects
> the entire network and creates ongoing issues in the future?
>
> --
>
> *James G. Phillips IV*
> 
>
> *"Don't bunt. Aim out of the ball park. Aim for the company of immortals."
> -- David Ogilvy*
>
>  *This message was created with 100% recycled electrons. Please think
> twice before printing.*
>
>
> --
>
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[Bitcoin-development] Max Block Size: Simple Voting Procedure

[Stephen Morse]

This is likely very similar to other proposals, but I want to bring voting
procedures back into the discussion. The goal here is to create a voting
procedure that is as simple as possible to increase the block size limit.

Votes are aggregated over each 2016 block period. Each coinbase transaction
may have an output at tx.vout[0] with OP_RETURN data in it of the format:

  OP_RETURN {OP_1 or OP_2}

OP_2 means the miner votes to increase the block size limit. OP_1 means the
miner votes to not increase the block size limit. *Not including such a
vote is equivalent to voting to NOT increase the block size. *I first
thought that not voting should mean that you vote with your block size, but
then decided that it would be too gameable by others broadcasting
transactions to affect your block size.

If in a 2016 block round there were more than 1008 blocks that voted to
increase the block size limit, then the max block size increases by 500 kb.
The votes can start when there is a supermajority of miners signaling
support for the voting procedure.

A few important properties of this simple voting:

   - It's not gameable via broadcasting transactions (assuming miners don't
   set their votes to be automatic, based on the size of recent blocks).
   - Miners don't have to bloat their blocks artificially just to place a
   vote for larger block sizes, and, similarly, don't need to exclude
   transactions even when they think the block size does not need to be raised.
   - The chain up until the point that this goes into effect may be
   interpreted as just lacking votes to increase the block size.

We can't trust all miners, but we have to trust that >50% of them are
honest for the system to work. This system makes it so that altering the
maximum block size requires >50% of miners (hash power) to vote to increase
the consensus-limit.

Thanks for your time. I think this is an important time in Bitcoin's
history. I'm not married to this proposal, but I think it would work. I
think a lot of the proposals mentioned on this mailing list would work. I
think it's time we just pick one and run with it.

Please let me know your thoughts. I will start working on a pull request if
this receives any support from miners/core devs/community members, unless
someone with more experience volunteers.

Best,
Stephen
--
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] [BIP] Normalized Transaction IDs

[Stephen Morse]

>
> An option would be that the height is included in the scriptSig for all
>> transactions, but for non-coinbase transctions, the height used is zero.
>>
> No need to add an extra field to the transaction just to include the
> height. We can just add a rule that the height specified in the scriptSig
> in coinbase transactions (and only coinbase transactions) is copied into
> the locktime of the transaction before computing the normalized transaction
> ID and leave the locktime untouched for all normal transactions
>

No need to replace lock times (or any other part of the transaction) at
all. If you have to, just serialize the height right before serializing the
transaction (into the same buffer). And you could pre-serialize 0 instead
of the height for all non-coinbase transactions. I don't really see what
that gets you, though, because the 0 is not really doing anything.

But, I don't see any reason you have to mess with the serialization this
much at all. Just do:

uint256 normalized_txid(CTransaction tx)
{
  // Coinbase transactions are already normalized
  if (!tx.IsCoinbase())
  {
foreach(CTxIn in : tx.vin)
{
  if (!ReplacePrevoutHashWithNormalizedHash(in.prevout))
throw NormalizationError("Could not lookup prevout");
  in.scriptSig.clear();
}
  }

  // Serialize
  CHashWriter ss(SER_GETHASH, 0);
  ss << tx;
  return ss.GetHash();
}

An alternative could be (although I like the above option better):

uint256 normalized_txid(CTransaction tx, int nHeight)
{
  foreach(CTxIn in : tx.vin)
  {
if (!in.prevout.IsNull() &&
!ReplacePrevoutHashWithNormalizedHash(in.prevout))
  throw NormalizationError("Could not lookup prevout");
in.scriptSig.clear();
  }

  // Serialize
  CHashWriter ss(SER_GETHASH, 0);

if (tx.IsCoinbase())
ss << nHeight;
// or:
// ss << (tx.IsCoinbase() ? nHeight : 0);

  ss << tx;
  return ss.GetHash();
}
--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] 75%/95% threshold for transaction versions

[Stephen Morse]

Hi Gregory,

In particular not covering the ID allows for transaction replay which
> can result in monetary losses far more severe than any possible
> mishandling of malleability could result in. Byzantine attackers can
> costlessly replay your old transactions any time anyone reuses an
> address, even accidentally (which cannot be easily prevented since
> they can race).
>

With the SIGHASH_WITHOUT_PREV_VALUE flag, signatures have to explicitly
specify that they are to be signed without the previous UTXO's
value/amount. This means that, at worst, replay attacks can send the money
to the same place it was sent before (which in many cases is likely not be
a loss of funds), and only if the amount sent to the reused address is the
exact same as it was before. I don't think this is worse than an attacker
being able to mutate their transaction and extort a merchant who accepts
zero-conf transactions. Anyway, not signing the input ID wouldn't exactly
be the norm, there would be a defined set of flags for standard use cases.
Not signing the input TXID would only be used in specialized cases, such as
setting up micropayment channels.


> There are no free lunches;  the proposal linked to there is itself a
> game of wack-a-mole with assorted masking flags;


I agree that it is also a bit of wac-a-mole, but the defined space of
issues is possibly more limited here. There are only X number of things
that can be signed/not signed in a transaction, and the 'Build your own
nHashType' proposal enables you to fully specify which of those are being
signed. If you don't want to get burned by not fully signing your
transactions, then don't use the non-standard sighash flags.

many of which we have
> no notion of if they're useful for any particular application(s);


A few of the flags, indeed, may not ever be useful. But we can't predict
the future, and I think it's better to build in a more flexible solution
now than to wish we had more flexible nHashTypes later.

To the original point of this thread, hopefully the suggested proposal
won't be necessary as wallets will upgrade to use version 3 transactions
and the rules associated with them over time.

Best,
Stephen
--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] 75%/95% threshold for transaction versions

[Stephen Morse]

Hi William,

I personally prefer this solution, since it nails the problem
> completely with one simple and obvious change. The BIP 62 approach is
> more like a game of wac-a-mole.
>

The two are complementary, not competing. BIP62 prevents *non-signers* from
mutating the transactions, which is very important. The 'Build your own
nHashType' proposal enables chained transactions even in the face of
*signers* mutating the transaction. I believe that integrating both will
lead to the best defense against transaction malleability, and will enable
more complicated uses of chained transactions (such as micropayment
channels).

Best,
Stephen
--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Build your own nHashType

[Stephen Morse]

Hi Mike,

Hi Stephen,
>
> It's an interesting idea. I'm not sure that all the combinations make
> sense. Excluding the connected output script or value but still signing the
> prev tx hash appears pointless: the script cannot change anyway, and you
> still need to know what it is to actually calculate the inputs to it, so
> what is the point of this?
>

That's a good point, maybe SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY and
SIGHASH_WITHOUT_PREV_VALUE should be assumed false, since you need the data
anyway. That gets the total number of flags down to 17. If we eliminate
SIGHASH_WITHOUT_TX_VERSION (I can't think of any good reason for this one),
then we're down to a 2-byte nHashType. SIGHASH_SIGN_STACK_ELEMENT could
also be removed, I'm not convinced of the usefulness of that one either.


>
> I also worry that quite a few of these combinations could be unexpectedly
> dangerous. If you don't sign the prevout hash or value and combine it with
> a regular pay-to-address output then you've effectively written a blank
> cheque that can be used by anyone, to claim any money ever sent to that
> address ... no? And then any p2p
>
node or miner could do so, making the transaction pretty useless.
>
> That isn't inherently a problem as long as people understand which
> combinations have what effects or cannot be used for various reasons. But
> it would need good documentation and careful thought to explore each
> possibility people might use.
>

I don't think it's quite a blank check, but it would enable replay attacks
in the form of sending the money to the same place it was sent before if an
address ever receives coins again. Almost like auto-forwarding addresses.
If, in addition, you signed with just that input and no outputs as well,
then you're basically forfeiting your rights to any coins sent to that
address.

It allows for some dangerous combinations, but we already have some
dangerous nHashTypes. e.g. SIGHASH_NONE | SIGHASH_ANYONECANPAY. Good
documentation and careful developers shouldn't have any issues if they use
a standard set of sighash flag combinations for their standard use cases.
But developers that need special combinations can now use them, so long as
they are careful and think things through.


>
> I'll leave the soft fork business to one side for now. I think any change
> in CHECKSIG or new version of it would likely be ready around the same time
> as the hard fork we need for changing the block size limit anyway, and it's
> much cleaner to do it that way.
>
> The most important change that we need in sighash calculation, IMO, is
> ensuring that you don't have to hash data over and over again without a
> good reason. The current sighash definition is unfortunate because it's
> possible to make small transactions that involve hashing huge amounts of
> data. It's not clear to me that your proposal fixes that: ideally there
> would be one exactly one sighash for one transaction no matter how many
> checksigs are involved in verifying it.
>
>
It's hard, though, because there is different data needs to be signed for
each input. Although, I suppose if you signed your input with
SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY, SIGHASH_WITHOUT_PREV_VALUE, and the
equivalent of SIGHASH_ALL, then the hash that needs to be signed would be
the same for all of your inputs. Strangely enough, I think we might have
just found use cases for the flags that we had nearly dismissed.

Another possibility would be to put the previous scriptPubKey and previous
output value at the END of the serialized transaction, so that you could
make use of some sort of a signature hash midstate. But that feels a little
messy. It sort of makes sense to have a base serialization for a
transaction and then append it with whatever input/output specific
information you have, but still, messy.

Is hashing transaction data once for each input really a huge bottleneck,
though? Do mobile devices have an issue with this?

Best,
Stephen
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Build your own nHashType

[Stephen Morse]

Seeking feedback on a proposal that will allow a transaction signer to
explicitly specify what is to be serialized for the signature hash. The
basic idea is to make the nHashType general enough that we won't need a new
sighash flag every time a new use case comes up.

If implemented into bitcoin (via a soft fork), this would make malleability
almost a non-issue (the TXID referenced by inputs just need to be updated
previous TX changes) and would enable hardware wallets to securely sign
without needing to download/process each transaction it spends from.

Please let me know your thoughts.

https://github.com/scmorse/bitcoin-misc/blob/master/sighash_proposal.md
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP: Voluntary deposit bonds

[Stephen Morse]

I agree with Gregory Maxwell, I don't think it would be as easy as just
changing IsCoinBase(), since there are places where the code assumes that
the coinbase's vin doesn't spend any prevouts and/or has size 1. For
example, here
 and
here 
.

I think the motivation behind the original suggestion is to have a way to
pay specific miners upon solving a block without risking possibly paying
other miners through pay-to-fee. What I'm not sure about, though, is why
not just send them a transaction once you see that the miner has solved a
block? Not a pay-to-fee transaction, a pay to pubkeyhash or whatever type
of transaction you need to make to send the miner some coins.

Although I don't completely understand the motivation for making such
transactions, maybe this would this work. Have outputs in the coinbase
transaction which have nValue == 0, then only apply the COINBASE_MATURITY
rule to spending coinbase outputs which have non-zero value. That way you
could make a transactions which is only valid after the miner specified
solves a block with the coinbase having the same TxID referenced by the new
transaction's input. It's still a hard fork, but might be easier than
allowing the coinbase to spend prevouts. I guess, at that point though, why
not just hard fork to allow the coinbase to spend prevouts...

Best,
Stephen

On Tue, Dec 30, 2014 at 1:28 PM, Gregory Maxwell  wrote:

> On Tue, Dec 30, 2014 at 4:25 PM, Sergio Lerner
>  wrote:
> > Slight off-topic:
> > That looks like an abuse of the VM. Even P2SH is an abuse of the VM.
> > Gavin's OP_EVAL (hard-fork) should had been chosen. I'm taking about a
> > simple change that goes along the lines of Satoshi's original design.
> > Bitcoin was a beautiful design, and extra complexity is making it ugly.
> > We need Bitcoin to be simple to understand for new programmers so they
> > can keep the project going. It doesn't help the project that one needs
> > to be a guru to code for Bitcoin.
>
> Sergio there is no "abuse" there,  OP_NOP3 in that case would be
> redefined to OP_COINBASE_FOO_CONSISTENCY.
>
> (I say FOO because it's not clear what rule you actually hope to apply
> there.)
>
> What you suggested has no purpose by itself: it would need an
> additional change which overlays functionality in order to actually do
> something. Such a change would likely be "ugly"-- it's easy to be
> elegant when you do nothing.
>
>
> --
> Dive into the World of Parallel Programming! The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development