Re: [bitcoin-dev] CHECKSEQUENCEVERIFY - We need more usecases to motivate the change

[Jorge Timón via bitcoin-dev]

On Fri, Oct 16, 2015 at 3:26 AM, Rusty Russell via bitcoin-dev
 wrote:
> ... Gavin just told me about setmocktime.  That's fast service!

Once more functions (specially consensus-critical functions) take
nTime explicitly as parameter instead of relying on the
library-unfriendly GetAdjustedTime(), then SetMockTime() will be less
necessary for testing. For example, see
https://github.com/jtimon/bitcoin/commit/88a35548518a27c7d24efe064e1bf4e5b3029578#diff-524ba4b43aa70d393ef51ab42a6d25f2L52
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] Bitcoin dev IRC meeting in layman's terms (2015-10-15)

[G1lius Caesar via bitcoin-dev]

Once again my attempt to summerize and explain the weekly bitcoin developer
meeting in layman's terms.
Link to last weeks summerization (
https://www.reddit.com/r/Bitcoin/comments/3o7bi6/bitcoin_dev_meeting_in_laymans_terms_2015108/
)
Link to this weeks on reddit:
https://www.reddit.com/r/Bitcoin/comments/3pcinz/bitcoin_dev_irc_meeting_in_laymans_terms_20151015/

*Disclaimer*

Please bear in mind I'm not a developer and I'd have problems coding "hello
world!", so some things might be incorrect or plain wrong.
Like any other write-up it likely contains personal biases, although I try
to stay as neutral as I can.
There are no decisions being made in these meetings, so if I say "everyone
agrees" this means everyone present in the meeting, that's not consensus,
but since a fair amount of devs are present it's a good representation.
The dev IRC and mailinglist are for bitcoin development purposes. If you
have not contributed actual code to a bitcoin-implementation, this is
probably not the place you want to reach out to. There are many places to
discuss things that the developers read, including this sub-reddit.


link to this week logs
http://bitcoinstats.com/irc/bitcoin-dev/logs/2015/10/15#l1444935660.0
Meeting minutes by meetbot
http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-10-15-19.01.html


Main topics discussed where:
Mempool limiting
sendheaders BIP
versionbits
dev/discuss list policy
CHECKSEQUENCEVERIFY


**Mempool limiting**

- background

When a transaction is relayed across the network it is held by the nodes in
memory, until it gets into a block. All these transactions that sit in
memory are called the memorypool or mempool for short.
Like we could see during the spam-attack if there's a big back-log of
transactions that couldn't make it in the blockchain this mempool can get
pretty big resulting in nodes crashing.

To stop this from happening devs are trying to find a way to limit this
mempool, so a mechanism to reject and/or remove transactions from the
mempool. The hard part here is to make it so nodes can't be attacked by
abusing this mechanism.
So far the devs are going with TheBlueMatt's proposal of throwing away the
cheapest txn and setting the min relay fee to it
https://github.com/bitcoin/bitcoin/pull/6722


- meeting comments

While testing, sipa encountered transactions that took 200ms to be accepted
into the mempool.
As it's the first time he has benchmarked this and the pull-request
shouldn't make an impact on these times it likely doesn't have anything to
do with this. However, such times are bad either way.
The average time in sipa's tests is 4ms. (After the meeting Morcos did some
benchmarking (
https://github.com/bitcoin/bitcoin/pull/6722#issuecomment-148874040 ) and
confirmed it was not specific to this PR, and pointed out the outliers come
from CheckInputs and HaveInputs (as you might guess, having to do with
checking the inputs)
Question on why we should revert the minrelay (minimum fee for nodes to
relay a transaction) back to 1000 (it has been set to 5000 to quick-fix the
mempool issues), sipa thinks it should be floating as well or the dust
limit becomes ineffective.


- meeting conclusion

Review PR 6722 Limit mempool by throwing away the cheapest txn and setting
min relay fee to it https://github.com/bitcoin/bitcoin/pull/6722
Morcos/sipa will do some more benchmarks and comment on the PR ( morcos'
benchmark results
https://github.com/bitcoin/bitcoin/pull/6722#issuecomment-148874040 )


**sendheaders BIP**

- background

send headers BIP
https://github.com/sdaftuar/bips/blob/add-sendheaders/bip-sendheaders.mediawiki
Copy/paste from the BIP:
Since the introduction of "headers-first" downloading of blocks in 0.10,
blocks will not be processed unless they are able to connect to a (valid)
headers chain. Consequently, block relay generally works as follows:

1. A node (N) announces the new tip with an "inv" message, containing the
block hash
2. A peer (P) responds to the "inv" with a "getheaders" message (to request
headers up to the new tip) and a "getdata" message for the new tip itself
3. N responds with a "headers" message (with the header for the new block
along with any preceding headers unknown to P) and a "block" message
containing the new block
However, in the case where a new block is being announced that builds on
the tip, it would be generally more efficient if the node N just announced
the block header for the new block, rather than just the block hash, and
saved the peer from generating and transmitting the getheaders message (and
the required block locator).



- meeting comments

Question on how to move forward. How to let the nodes know you want the
blockheader instead of the blockhash.
Options:
Extend the version message.
Have an "options" message that can send flags.
Send a "sendheaders" message early when connecting so the way peers want
their block announcement is immediately known.
Send a "sendheaders" message at any time, changing the 

[bitcoin-dev] [BIP] Normalized transaction IDs

[Christian Decker via bitcoin-dev]

After spending some more time on the normalized transaction ID proposal and
reworking it to be a soft-fork (thanks sipa for helping me figuring out
how), I'd like to propose the BIP again.

As with the previous version, which was using a hard-fork, the normalized
transaction ID is computed only considering the non-malleable parts of a
transaction, i.e., stripping the signatures before computing the hash of
the transaction. This ensures that if a transaction is modified, either by
a third party fishing transactions from the network and re-injecting
modified versions or by one of the signers re-signing it, any transaction
that builds on top of it still remains valid. Furthermore it allows the use
of template transactions, unsigned transactions upon which further
transaction can be built before signing the template transaction and
locking the contract.

Unlike the previous proposal, this is a softfork proposal that redefines
OP_NOP4 with an extensible and parameterized version of the signature
checking opcodes, called OP_CHECKSIGEX. Among other things the parameters
allow to specify that an output with an OP_CHECKSIGEX is to be referenced
by the normalized transaction ID that created it, instead of the instance
transaction ID containing malleable signatures. This BIP uses the
normalized transaction IDs exclusively while signing or checking
signatures, they are not used in any network level message as the previous
version would have done, hence there is no change at network level and old
clients should be able to exchange transactions as before and blocks still
reference the transaction instances.

The proposal is implemented (see below), by computing the normalized
transaction ID when adding them to the UTXO and storing them along with the
coin state. OP_CHECKSIGEX mostly duplicates OP_CHECKSIG and
OP_CHECKMULTISIG, but I'm hoping somebody can give me some pointers into
how to best refactor the common functionality into reusable blocks. And the
annotating incoming transactions with their normalized inputs is a bit
cumbersome, maye somebody has some pointers here as well?

BIP Pull request: https://github.com/bitcoin/bips/pull/224
Implementation: https://github.com/cdecker/bitcoin/commits/normtx

I think in the discussion of my previous proposal, most of you welcomed the
introduction of normalized transaction IDs, were it not for the hardfork. I
hope this proposal adresses the previous concerns and that we can move
forward in adding the normalized transaction IDs to the bitcoin protocol.
That being said, I'm always open to suggestions :-)

Regards,
Christian
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP] Normalized transaction IDs

[Tier Nolan via bitcoin-dev]

On Mon, Oct 19, 2015 at 3:01 PM, Christian Decker via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> As with the previous version, which was using a hard-fork, the normalized
> transaction ID is computed only considering the non-malleable parts of a
> transaction, i.e., stripping the signatures before computing the hash of
> the transaction.
> 


Is this proposal recursive?


*Coinbase transaction *

* n-txid = txid


*Non-coinbase transactions*
* replace sigScripts with empty strings
* replace txids in TxIns with n-txid for parents

The 2nd step is recursive starting from the coinbases.

In effect, the rule is that txids are what they would have been if n-txids
had been used right from the start.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin via bitcoin-dev]

My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan 
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (ad...@multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.3, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880==definitely lost: 0 bytes in 0 blocks
> ==6880==indirectly lost: 0 bytes in 0 blocks
> ==6880==  possibly lost: 288 bytes in 1 blocks
> ==6880==still reachable: 10,552 bytes in 39 blocks
> ==6880== suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==definitely lost: 0 bytes in 0 blocks
> ==6778==indirectly lost: 0 bytes in 0 blocks
> ==6778==  possibly lost: 320 bytes in 1 blocks
> ==6778==still reachable: 10,080 bytes in 32 blocks
> ==6778== suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP] Normalized transaction IDs

[Christian Decker via bitcoin-dev]

Yes, this has been pointed out in the PR as well. Transactions inputs must
also be normalized by replacing malleable hashes with the normalized
hashes. I will fix the spec and the implementation to reflect this :-)

Regards,
Christian

On Mon, Oct 19, 2015 at 5:24 PM Tier Nolan via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Mon, Oct 19, 2015 at 3:01 PM, Christian Decker via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> As with the previous version, which was using a hard-fork, the normalized
>> transaction ID is computed only considering the non-malleable parts of a
>> transaction, i.e., stripping the signatures before computing the hash of
>> the transaction.
>> 
>
>
> Is this proposal recursive?
>
>
> *Coinbase transaction *
>
> * n-txid = txid
>
>
> *Non-coinbase transactions*
> * replace sigScripts with empty strings
> * replace txids in TxIns with n-txid for parents
>
> The 2nd step is recursive starting from the coinbases.
>
> In effect, the rule is that txids are what they would have been if n-txids
> had been used right from the start.
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP] Normalized transaction IDs

[s7r via bitcoin-dev]

So what exactly is used to create the normalized txid (sha256 hash of
what data)? I've read in the linked BIP draft that it will strip the
'malleable parts' but didn't understand what exactly will be used to
calculate the normalized transactions ids and how will the change apply
retro-active for the transactions so deep buried in the blockchain?

Pubkeys (addresses) can be reused infinitely so what guarantees us
unique normalized txids all the time and protection against replay
attacks? The question is not if this issue is covered or not, I know it
is, I am just asking how, in simpler terms.

SCRIPT_CHECKSIGEX_NORMALIZE could be explained better in the document.

Will it also fix > third level malleability (a tx which spends from
another unconfirmed tx which spends from yet another unconfirmed tx)?


On 10/19/2015 6:23 PM, Tier Nolan via bitcoin-dev wrote:
> On Mon, Oct 19, 2015 at 3:01 PM, Christian Decker via bitcoin-dev
>  > wrote:
> 
> As with the previous version, which was using a hard-fork, the
> normalized transaction ID is computed only considering the
> non-malleable parts of a transaction, i.e., stripping the signatures
> before computing the hash of the transaction.
> 
> 
> 
> Is this proposal recursive? 
> 
> *Coinbase transaction
> *
> 
> * n-txid = txid
> 
> *Non-coinbase transactions
> *
> * replace sigScripts with empty strings
> * replace txids in TxIns with n-txid for parents
> 
> The 2nd step is recursive starting from the coinbases.
> 
> In effect, the rule is that txids are what they would have been if
> n-txids had been used right from the start.
> 
> 
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin via bitcoin-dev]

I should also mention that this is definitely not an attack coming from
connected nodes.  My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).

--Adam

On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin 
wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan 
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (ad...@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.3, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==definitely lost: 0 bytes in 0 blocks
>> ==6880==indirectly lost: 0 bytes in 0 blocks
>> ==6880==  possibly lost: 288 bytes in 1 blocks
>> ==6880==still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==definitely lost: 0 bytes in 0 blocks
>> ==6778==indirectly lost: 0 bytes in 0 blocks
>> ==6778==  possibly lost: 320 bytes in 1 blocks
>> ==6778==still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> ___
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev