Re: [bitcoin-dev] Minimizing the redundancy in Golomb Coded Sets

2018-05-29 Thread Lucas Ontivero via bitcoin-dev
Hi Jim,

Yes please, could you share CSV? We are developing a Wallet that uses
Golomb-Rice filters it would help a lot for determine the best P value
depending on the estimated number of elements the client needs to watch.

2018-05-29 19:38 GMT-03:00 Jim Posen via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org>:

> This is a really cool finding, thanks Pieter!
>
> I did some more analysis on selecting a good P value to reduce total data
> downloaded considering both filters themselves and blocks in the case of
> false positive matches, using data from mainnet. The quantity it minimizes
> is:
>
> filter_size(N, B) + block_size * false_positive_probability(C, N, B)
>
> N is the number of filter elements per block
> B is the Golomb-Rice coding parameter
> C is the number of filter elements watched by the client
>
> The main result is that:
>
> For C = 10, B = 13 is optimal
> For C = 100, B = 16 is optimal
> For C = 1,000, B = 20 is optimal
> For C = 10,000, B = 23 is optimal
>
> So any value of B in the range 16 to 20 seems reasonable, with M = 1.4971
> * 2^B for optimal compression, as Pieter derived. The selection of the
> parameter depends on the target number of elements that a client may watch.
>
> I attached some of the results, and would be happy to share the CSV and
> raw notebook if people are interested.
>
>
> On Fri, May 25, 2018 at 2:14 PM Gregory Maxwell via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Fri, May 25, 2018 at 6:42 PM, Gregory Maxwell  wrote:
>> > configuration is roughly right, then M=1569861 and rice parameter 19
>> > should be used.
>>
>> That should have been M=784931 B=19  ... paste error.
>> ___
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


Re: [bitcoin-dev] Making OP_TRUE standard?

2018-05-29 Thread Rusty Russell via bitcoin-dev
Peter Todd  writes:
> On Mon, May 21, 2018 at 01:14:06PM +0930, Rusty Russell via bitcoin-dev wrote:
>> Jim Posen  writes:
>> > I believe OP_CSV with a relative locktime of 0 could be used to enforce RBF
>> > on the spending tx?
>> 
>> Marco points out that if the parent is RBF, this child inherits it, so
>> we're actually good here.
>> 
>> However, Matt Corallo points out that you can block RBF will a
>> large-but-lowball tx, as BIP 125 points out:
>> 
>>will be replaced by a new transaction...:
>> 
>>3. The replacement transaction pays an absolute fee of at least the sum
>>   paid by the original transactions.
>> 
>> I understand implementing a single mempool requires these kind of
>> up-front decisions on which tx is "better", but I wonder about the
>> consequences of dropping this heuristic?  Peter?
>
> We've discussed this before: that rule prevents bandwidth usage DoS attacks on
> the mempool; it's not a "heuristic". If you drop it, an attacker can 
> repeatedly
> broadcast and replace a series of transactions to use up tx relay bandwidth 
> for
> significantly lower cost than otherwise.
>
> Though these days with relatively high minimum fees that may not matter.

AFAICT the optimal DoS is where:

1.  Attacker sends a 100,000 vbyte tx @1sat/vbyte.
2.  Replaces it with a 108 vbyte tx @2sat/vbyte which spends one of
those inputs.
3.  Replaces that spent input in the 100k tx and does it again.

It takes 3.5 seconds to propagate to 50% of network[1] (probably much worse
given 100k txs), so they can only do this about 86 times per block.

That means they send 86 * (10 + 108) = 8609288 vbytes for a cost of
86 * 2 * 108 + 10 / 2 = 68576 satoshi (assuming 50% chance 100k tx
gets mined).

That's a 125x cost over just sending 1sat/vbyte txs under optimal
conditions[2], but it doesn't really reach most low-bandwidth nodes
anyway.

Given that this rule is against miner incentives (assuming mempool is
full), and makes things more complex than they need to be, I think
there's a strong argument for its removal.

Cheers,
Rusty.
[1] http://bitcoinstats.com/network/propagation/
[2] Bandwidth overhead for just sending a 108-vbyte tx is about 160
bytes, so our actual bandwidth per satoshi is closer to 60x
even under optimal conditions.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[bitcoin-dev] New serialization/encoding format for key material

2018-05-29 Thread Jonas Schnelli via bitcoin-dev
Hi

Extended public and private keys are defined in BIP32 [1].

Encoded extended private keys should not be confused with a wallet „seed“
(proposals like BIP39) while they can also partially serve the purpose to
„seed“ a wallet (there may be an overlap in the use-case).

Recovering a wallet by its extended private master key (xpriv; may or may not
be at depth 0) is a complex task with risks of failing to recover all available
funds.

It may be reasonable to consider that recovering a wallet purely based on the
existence of an extended private master key is a forensic funds recovery
process and should probably be the last resort in case of a backup-recovery
situation. A simple example here is, that it was/is possible to have used an
xpriv (referring to extended private master key) in production that is/was used
to derive BIP45 based P2SH multisig addresses (1of1, used by Bitpays BWS for
while), later used for bare BIP45ish multisig 1of1 as well as for P2PKH after
BIP44 & vanilla BIP32 P2WPKH (m/0’/k’).
I’m not aware of any wallet that would recover 100% of those funds, leading to
the risk that forwarding the unspents and destroying the extended master key
may result in coins forever lost.

The case above may be an edge case, but I’m generally under the assumption that
recovering funds based on the sole existence of an xpriv (or seed) without 
further
metadata is a fragile concept.

Second, the missing birthday-metadata tend to lead to non-optimal blockchain
scans (eventually increased p2p traffic). Recovering funds can take hours.

Additionally, the BIP44 gap limit seems to be a weak construct. The current gap
limit in BIP44 is set to 20 [2] which basically means, handing out more then 20
incoming payment requests (addresses) results in taking the risks that funds
may be destroyed (or at least not detected) during a recovery.
The Gap limit value may also depend on the use case, but the current proposals
do not allow to set an arbitrary value. High load merchants very likely need a
different gap limit value then individuals create a transaction once a year.

During creation time of an xpriv/xpub, it is impossible to know if the created
xpriv will be used for an unforeseen derivation scheme. Future proposals may
want to limit an extended key to a single derivation scheme.


This is an early draft in order to allow discussion that may lead to a possible
proposal.
This proposals could also make BIP 178 obsolete since it can be replace the
WIF[3] standard.


Thanks for feedback
/jonas





Titel
##
Bech32 encoded key material including metadata

Abstract

An error tolerant encoding format for key material up to 520bits with a minimal
amount of metadata.

Motivation
##
(See above; intro text)


Specification
#

## Serialization format

1 bit version bit
15 bits (bit 1 to 16) key-birthday (0-32767)
(12 bit gap limit)
3 or 5 bits script type
256 or 512 or 520 bits key material
= Total 275, 545, 553 bits

The initial version bit allows extending the serialization-format in future.
The encoding format must hint the total length and thus allow to calculate the
length of the key material.

The total length for 256 or 512 bit key material is optimised for Bech32 (power
of 5).

### Key material
If the key material length is 520 bits, it must contain an extended public key
If the key material length is 512 bits, it must contain an extended private key
Key material length other then 256, 512, 520 bits and invalid.

If 520 bits are present, first 256 bits are the BIP32 chain code, to second 264
bits (33 bytes) define the public key (according to BIP32)

If 512 bits are present, first 256 bits are the BIP32 chain code, to second 256
bits define the private key

If 256 bits are present, those bits represent a pure private key (or seed)

### Key birthday
A 15 bit timestamp expressed in days since genesis (valid up to ~2098). The
birthday must be set to the first possible derivation of the according extended
key, if unknown, the used seed birthday must be used. If both unknown, 0
(16x0bit) must be used.

### Gap limit delta
12 bits, results in a possible range from 0 to 4095.

If the total decoded serialization length is 275 bits (decode) or if the key
material is 256 bits (encode), the gap limit must not be present.

The base gap limit value is 20 (to disallow insane gap limits). The final gap
limit is the base value + the gap limit delta stored in those 12 bits.
Key derivation gap limit must not be exceeded when deriving child keys and must
be respected during transaction rescans.
Child key derivation must not be possible if gap limit is hit.

### Script type restriction
3 or 5 bits (range 0-7 / 0-31)
0 no restriction
1 P2PKH compressed
2 P2PKH | P2SH
3 P2WPKH P2WSH nested in P2SH
4 P2WPKH | P2WSH

If the total decoded serialization length is 275 bits (decode) or if the key
material is 256 bits (encode), 3 bits are used for the script type. 5 bits are
used