Re: [bitcoin-dev] New serialization/encoding format for key material
On Fri, Jun 15, 2018 at 8:54 AM, Russell O'Connor wrote: > >> For codes designed for length 341 (the first length enough to support >> 512 bits of data): >> * correct 1 error = 3 checksum characters >> * correct 2 errors = 7 checksum characters >> * correct 3 errors = 11 checksum characters >> * correct 4 errors = 15 checksum characters >> * correct 5 errors = 19 checksum characters >> * ... >> * correct 7 errors = 26 checksum characters (~ length * 1.25) >> * correct 13 errors = 51 checksum characters (~ length * 1.5) >> * correct 28 errors = 102 checksum characters (~ length * 2) >> >> So it really boils down to a trade-off between length of the code, and >> recovery properties. > > > At the risk of making the proposal more complex, I wonder if it might be > better to support multiple checksum variants? The trade-off between code > length and recovery seems to be largely determined by the user's medium of > storage, which is likely to vary from person to person. I personally would > probably be interested in the 51 or even 102 character checksums variants. Here are some more numbers then. It's important to note that the number of correctable errors includes errors inside the checksum characters themselves. So if you want to aim for a certain percentage of correctable characters, the numbers go up much more dramatically. For codes restricted to 341 characters total (including the checksum characters), and assuming 103 data characters (enough for 512 bits): * With 26 checksum characters (adding 25%, 20% of overall string), 7 errors can be corrected (5% of overall string) * With 62 checksum characters (adding 60%, 38% of overall string), 17 errors can be corrected (10% of overall string) * With 116 checksum characters (adding 113%, 53% of overall string), 33 errors can be corrected (15% of overall string) * With 195 checksum characters (adding 189%, 65% of overall string), 60 errors can be corrected (20% of overall string) For codes restricted to 1023 characters total (including the checksum characters), and assuming 103 data characters (enough for 512 bits): * With 27 checksum characters (adding 26%, 21% of overall string), 7 errors can be corrected (5% of overall string) * With 64 checksum characters (adding 62%, 38% of overall string), 17 errors can be corrected (10% of overall string) * With 127 checksum characters (adding 123%, 57% of overall string), 36 errors can be corrected (15% of overall string) * With 294 checksum characters (adding 285%, 74% of overall string), 80 errors can be corrected (20% of overall string) * With 920 checksum characters (adding 893%, 90% of overall string), 255 errors can be corrected (25% of overall string) I'll gladly construct reference source code for any of these. Cheers, -- Pieter ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
> For codes designed for length 341 (the first length enough to support > 512 bits of data): > * correct 1 error = 3 checksum characters > * correct 2 errors = 7 checksum characters > * correct 3 errors = 11 checksum characters > * correct 4 errors = 15 checksum characters > * correct 5 errors = 19 checksum characters > * ... > * correct 7 errors = 26 checksum characters (~ length * 1.25) > * correct 13 errors = 51 checksum characters (~ length * 1.5) > * correct 28 errors = 102 checksum characters (~ length * 2) > > So it really boils down to a trade-off between length of the code, and > recovery properties. > At the risk of making the proposal more complex, I wonder if it might be better to support multiple checksum variants? The trade-off between code length and recovery seems to be largely determined by the user's medium of storage, which is likely to vary from person to person. I personally would probably be interested in the 51 or even 102 character checksums variants. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
On Tue, May 29, 2018 at 5:13 AM, Jonas Schnelli via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Hi > > If 520 bits are present, first 256 bits are the BIP32 chain code, to > second 264 > bits (33 bytes) define the public key (according to BIP32) > In a 33 byte compressed public key, only 1 bit from the first byte conveys information. The other 7 bits can be discarded. This will allow you to reduce the bech32 encoded result by one character. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
On Sun, Jun 3, 2018 at 2:30 PM, Jonas Schnelli wrote: >> If there is interest, I can construct a code + implementation for any >> of these in a few days probably, once the requirements are clear. > > Yes. Please. Here is an example BCH code for base32 data which adds 27 checksum characters, and can correct up to 7 errors occurring in strings up to length 1023 (including the checksum characters themselves): https://gist.github.com/sipa/d62f94faa1dcfd9ee4012d4c88955ba6 It can encode sequences of integers (between 0 and 31): ref.py encode 13 7 22 23 11 29 21 15 3 26 20 26 4 7 6 11 19 1 6 8 31 13 4 19 > d8khta40r656y8xtnpxgldyne96vsfr83uch908se82g98rmnaa Decode it again: ref.py decode d8khta40r656y8xtnpxgldyne96vsfr83uch908se82g98rmnaa > Decoded: 13 7 22 23 11 29 21 15 3 26 20 26 4 7 6 11 19 1 6 8 31 13 4 19 Or correct errors: ref.py decode d8khta50r656y8xtmpxhlcyne96vsfr84udh908se82g98rmnat > Errors found: d8khta?0r656y8xt?px?l?yne96vsfr8?u?h908se82g98rmna? > Correction: d8khta40r656y8xtnpxgldyne96vsfr83uch908se82g98rmnaa > Decoded: 13 7 22 23 11 29 21 15 3 26 20 26 4 7 6 11 19 1 6 8 31 13 4 19 The code above is just a randomly picked BCH code, and has no special properties beyond the ones it is designed for. I can easily generate similar code for BCH codes with different properties. Cheers, -- Pieter ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
> I have some concerns about the use of Bech32. It is designed for > detecting 3 errors up to length 1023 (but is then picked specifically > to support 4 errors up to length 89). However, for error correction > this translates to just being able to efficiently correct 1 error > (3/2, rounded down) up to length 1023. You can of course always try > all combinations of up to N changes to the input (for any N), testing > the checksum, and comparing the results against the UTXO set or other > wallet information that may have been recovered. However, the checksum > at best gives you a small constant speedup here, not a fundamentally > improved way for recovery. Thanks Peter I removed the part in the proposals that made false claims about the error correction or cpu-intense key recovery. I wrote some test code and figured out that my Core i7 machine can do 31’775 operations per seconds of a addr-derivation-comparison (bech32 decode, bip32 ckd, hash160, Base58check). This is non-optimized code running non-parallelized. Just in case someone wants to do more math here. Without knowing to much about BCHs, ideally there would be a code that includes the fact that computational costs for error correction can be very high during a disaster recovery and that we can probably assume that the user can provide a derivation element like a used address or pubkey. Deriving one million child keys and comparing them against an address table will take less than a minute on consumer systems. > * correct 7 errors = 26 checksum characters (~ length * 1.25) > > So it really boils down to a trade-off between length of the code, and > recovery properties. I think 5% error correction (7 errors at 555bits) with a 26 char checksum is probably an acceptable tradeoff. Resulting string with 26 checksum chars (mockup): xp1qq8z4rsgv54z9a92yla4m2yrsqdlwdl7gn6qldvwkuh3zrg66z8ad2snf832tgaxcuv3kmwugzl5x8wtnkj2q3a03ky0kg8p7dvv4czpjqgvv4zgnvv4zgnvv4zgnvv4zgngn (140 chars) Versus the bech32 (6 char checksum): xp1qq8z4rsgv54z9a92yla4m2yrsqdlwdl7gn6qldvwkuh3zrg66z8ad2snf832tgaxcuv3kmwugzl5x8wtnkj2q3a03ky0kg8p7dvv4czpjqgvv4zgn (120 chars) Versus an xpriv: xprv9wHokC2KXdTSpEepFcu53hMDUHYfAtTaLEJEMyxBPAMf78hJg17WhL5FyeDUQH5KWmGjGgEb2j74gsZqgupWpPbZgP6uFmP8MYEy5BNbyET (111 chars) Not sure if the additional 20 characters make the UX worse. Typing in +20 chars in a disaster recovery is probably acceptable. > If there is interest, I can construct a code + implementation for any > of these in a few days probably, once the requirements are clear. Yes. Please. Lets first wait for more feedback about the error robustness though. Thanks - Jonas signature.asc Description: Message signed with OpenPGP ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
On Sun, Jun 3, 2018 at 9:51 AM, Jonas Schnelli via bitcoin-dev wrote: > Hi > > The BIP proposal is now available here: > https://gist.github.com/jonasschnelli/68a2a5a5a5b796dc9992f432e794d719 > > Reference C code is available here: > https://github.com/jonasschnelli/bech32_keys > > Feedback, criticism, etc. welcome! First of all, thanks for working on this. I have some concerns about the use of Bech32. It is designed for detecting 3 errors up to length 1023 (but is then picked specifically to support 4 errors up to length 89). However, for error correction this translates to just being able to efficiently correct 1 error (3/2, rounded down) up to length 1023. You can of course always try all combinations of up to N changes to the input (for any N), testing the checksum, and comparing the results against the UTXO set or other wallet information that may have been recovered. However, the checksum at best gives you a small constant speedup here, not a fundamentally improved way for recovery. However, we can design other base32 BCH codes easily with different properties. As we mostly care about efficient algorithms for recovery (and not just error detection properties), it seems more important to have good design strength (as opposed to picking a code from a large set which happens to have better properties, but without efficient algorithm, like Bech32). This is what I find for codes designed for length 93 (the first length for which efficient codes exist with length long enough to support 256 bits of data): * correct 1 error = 3 checksum characters * correct 2 errors = 6 checksum characters * correct 3 errors = 10 checksum characters * correct 4 errors = 13 checksum characters * correct 5 errors = 16 checksum characters * ... * correct 8 errors = 26 checksum characters (~ length * 1.5) * correct 11 errors = 36 checksum characters (~ maximum length without pushing checksum + data over 93 characters) For codes designed for length 341 (the first length enough to support 512 bits of data): * correct 1 error = 3 checksum characters * correct 2 errors = 7 checksum characters * correct 3 errors = 11 checksum characters * correct 4 errors = 15 checksum characters * correct 5 errors = 19 checksum characters * ... * correct 7 errors = 26 checksum characters (~ length * 1.25) * correct 13 errors = 51 checksum characters (~ length * 1.5) * correct 28 errors = 102 checksum characters (~ length * 2) So it really boils down to a trade-off between length of the code, and recovery properties. These two sets of codes are distinct (a code designed for length 93 has zero error correction properties when going above 93), so either we can pick a separate code for the two purposes, or be limited to the second set. If there is interest, I can construct a code + implementation for any of these in a few days probably, once the requirements are clear. Cheers, -- Pieter ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
Hi The BIP proposal is now available here: https://gist.github.com/jonasschnelli/68a2a5a5a5b796dc9992f432e794d719 Reference C code is available here: https://github.com/jonasschnelli/bech32_keys Feedback, criticism, etc. welcome! Thanks — Jonas signature.asc Description: Message signed with OpenPGP ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
Hi > - Visually Comparing two keys to find if they are same (Important) > - Different wallet software could set different birthday/gap limit. creating > different xpub/xprv for the same set of mathematically derived individual > keys. This removes the decoupling between key and wallet metadata What would be the downside of encoding the same key with different metadata (resulting in different "visual strings“)? If you import it into the same software, it would be trivial to detect it. If you import it into another software, it probably doesn’t matter. Visual comparing is eventually a broken concept (agree with Greg) and I doubt that this property is important, and IMHO basic metadata seems more important then this - very likely irrelevant - visual property. Also, I think a recovery based on a sole xpriv (or + limited amount of meta-data as described in this proposal) is a disaster recovery (or forensic recovery). Long term, I would wish, if wallet-metadata including transaction based user metadata would be backed up - after encrypted with a key that can be derived from the seed - in a way, where you need the seed to recover that backup thus it can be stored in cheap, insecure spaces. > > In fact, same could be argued to add birthday to WIF private key format to > let wallet discover funds faster. > The proposal I made can be seen as a replacement for WIF (it can replace WIF and xpriv/xpub) since it can encode a single private key into 275bits (still pretty short Bech32 string). /jonas signature.asc Description: Message signed with OpenPGP ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
On Wed, May 30, 2018 at 6:30 AM, shiva sitamraju via bitcoin-dev wrote: > The idea to add birthdate and gap limit sounds very good and addresses lots > of problems users are facing. > > However, adding birthday to keys breaks two basic properties > > - Visually Comparing two keys to find if they are same (Important) Can you explain exactly what you mean there? I can think of to plausible meanings (that two valid keys could differ by only a single symbol, which wouldn't be true due to the checksum and could be made even stronger if we thought that would be useful or I think you could also be complaining that the same "key material" could be encoded two ways which I think is both harmless and unavoidable for anything versioned). > - Different wallet software could set different birthday/gap limit. creating > different xpub/xprv for the same set of mathematically derived individual > keys. This removes the decoupling between key and wallet metadata Personally, I think it's a mistake to believe that any key format can really make private keying material strongly compatible between wallets. At best you can hope for a mostly compatible kind of recovery handling. But the lookahead amount may be pretty integral to the design of the software, so signaling it may not mean the other side can obey the signal... but that wouldn't make the signal completely useless. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] New serialization/encoding format for key material
The idea to add birthdate and gap limit sounds very good and addresses lots of problems users are facing. However, adding birthday to keys breaks two basic properties - Visually Comparing two keys to find if they are same (Important) - Different wallet software could set different birthday/gap limit. creating different xpub/xprv for the same set of mathematically derived individual keys. This removes the decoupling between key and wallet metadata In fact, same could be argued to add birthday to WIF private key format to let wallet discover funds faster. Is it possible to have a serialization so that in the encoding, the key part is still visually the same ? On Tue, May 29, 2018 at 5:30 PM, < bitcoin-dev-requ...@lists.linuxfoundation.org> wrote: > Send bitcoin-dev mailing list submissions to > bitcoin-dev@lists.linuxfoundation.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > or, via email, send a message with subject or body 'help' to > bitcoin-dev-requ...@lists.linuxfoundation.org > > You can reach the person managing the list at > bitcoin-dev-ow...@lists.linuxfoundation.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of bitcoin-dev digest..." > > > Today's Topics: > >1. New serialization/encoding format for key material > (Jonas Schnelli) > > > -- > > Message: 1 > Date: Tue, 29 May 2018 11:13:37 +0200 > From: Jonas Schnelli > To: Bitcoin Protocol Discussion > > Subject: [bitcoin-dev] New serialization/encoding format for key > material > Message-ID: <03557e21-8cfc-4822-8494-f4a78e238...@jonasschnelli.ch> > Content-Type: text/plain; charset="utf-8" > > Hi > > Extended public and private keys are defined in BIP32 [1]. > > Encoded extended private keys should not be confused with a wallet ?seed? > (proposals like BIP39) while they can also partially serve the purpose to > ?seed? a wallet (there may be an overlap in the use-case). > > Recovering a wallet by its extended private master key (xpriv; may or may > not > be at depth 0) is a complex task with risks of failing to recover all > available > funds. > > It may be reasonable to consider that recovering a wallet purely based on > the > existence of an extended private master key is a forensic funds recovery > process and should probably be the last resort in case of a backup-recovery > situation. A simple example here is, that it was/is possible to have used > an > xpriv (referring to extended private master key) in production that is/was > used > to derive BIP45 based P2SH multisig addresses (1of1, used by Bitpays BWS > for > while), later used for bare BIP45ish multisig 1of1 as well as for P2PKH > after > BIP44 & vanilla BIP32 P2WPKH (m/0?/k?). > I?m not aware of any wallet that would recover 100% of those funds, > leading to > the risk that forwarding the unspents and destroying the extended master > key > may result in coins forever lost. > > The case above may be an edge case, but I?m generally under the assumption > that > recovering funds based on the sole existence of an xpriv (or seed) without > further > metadata is a fragile concept. > > Second, the missing birthday-metadata tend to lead to non-optimal > blockchain > scans (eventually increased p2p traffic). Recovering funds can take hours. > > Additionally, the BIP44 gap limit seems to be a weak construct. The > current gap > limit in BIP44 is set to 20 [2] which basically means, handing out more > then 20 > incoming payment requests (addresses) results in taking the risks that > funds > may be destroyed (or at least not detected) during a recovery. > The Gap limit value may also depend on the use case, but the current > proposals > do not allow to set an arbitrary value. High load merchants very likely > need a > different gap limit value then individuals create a transaction once a > year. > > During creation time of an xpriv/xpub, it is impossible to know if the > created > xpriv will be used for an unforeseen derivation scheme. Future proposals > may > want to limit an extended key to a single derivation scheme. > > > This is an early draft in order to allow discussion that may lead to a > possible > proposal. > This proposals could also make BIP 178 obsolete since it can be replace the > WIF[3] standard. > > > Thanks for feedback > /jonas > > > > > > Titel > ## > Bech32 encoded key material including metadat
[bitcoin-dev] New serialization/encoding format for key material
Hi Extended public and private keys are defined in BIP32 [1]. Encoded extended private keys should not be confused with a wallet „seed“ (proposals like BIP39) while they can also partially serve the purpose to „seed“ a wallet (there may be an overlap in the use-case). Recovering a wallet by its extended private master key (xpriv; may or may not be at depth 0) is a complex task with risks of failing to recover all available funds. It may be reasonable to consider that recovering a wallet purely based on the existence of an extended private master key is a forensic funds recovery process and should probably be the last resort in case of a backup-recovery situation. A simple example here is, that it was/is possible to have used an xpriv (referring to extended private master key) in production that is/was used to derive BIP45 based P2SH multisig addresses (1of1, used by Bitpays BWS for while), later used for bare BIP45ish multisig 1of1 as well as for P2PKH after BIP44 & vanilla BIP32 P2WPKH (m/0’/k’). I’m not aware of any wallet that would recover 100% of those funds, leading to the risk that forwarding the unspents and destroying the extended master key may result in coins forever lost. The case above may be an edge case, but I’m generally under the assumption that recovering funds based on the sole existence of an xpriv (or seed) without further metadata is a fragile concept. Second, the missing birthday-metadata tend to lead to non-optimal blockchain scans (eventually increased p2p traffic). Recovering funds can take hours. Additionally, the BIP44 gap limit seems to be a weak construct. The current gap limit in BIP44 is set to 20 [2] which basically means, handing out more then 20 incoming payment requests (addresses) results in taking the risks that funds may be destroyed (or at least not detected) during a recovery. The Gap limit value may also depend on the use case, but the current proposals do not allow to set an arbitrary value. High load merchants very likely need a different gap limit value then individuals create a transaction once a year. During creation time of an xpriv/xpub, it is impossible to know if the created xpriv will be used for an unforeseen derivation scheme. Future proposals may want to limit an extended key to a single derivation scheme. This is an early draft in order to allow discussion that may lead to a possible proposal. This proposals could also make BIP 178 obsolete since it can be replace the WIF[3] standard. Thanks for feedback /jonas Titel ## Bech32 encoded key material including metadata Abstract An error tolerant encoding format for key material up to 520bits with a minimal amount of metadata. Motivation ## (See above; intro text) Specification # ## Serialization format 1 bit version bit 15 bits (bit 1 to 16) key-birthday (0-32767) (12 bit gap limit) 3 or 5 bits script type 256 or 512 or 520 bits key material = Total 275, 545, 553 bits The initial version bit allows extending the serialization-format in future. The encoding format must hint the total length and thus allow to calculate the length of the key material. The total length for 256 or 512 bit key material is optimised for Bech32 (power of 5). ### Key material If the key material length is 520 bits, it must contain an extended public key If the key material length is 512 bits, it must contain an extended private key Key material length other then 256, 512, 520 bits and invalid. If 520 bits are present, first 256 bits are the BIP32 chain code, to second 264 bits (33 bytes) define the public key (according to BIP32) If 512 bits are present, first 256 bits are the BIP32 chain code, to second 256 bits define the private key If 256 bits are present, those bits represent a pure private key (or seed) ### Key birthday A 15 bit timestamp expressed in days since genesis (valid up to ~2098). The birthday must be set to the first possible derivation of the according extended key, if unknown, the used seed birthday must be used. If both unknown, 0 (16x0bit) must be used. ### Gap limit delta 12 bits, results in a possible range from 0 to 4095. If the total decoded serialization length is 275 bits (decode) or if the key material is 256 bits (encode), the gap limit must not be present. The base gap limit value is 20 (to disallow insane gap limits). The final gap limit is the base value + the gap limit delta stored in those 12 bits. Key derivation gap limit must not be exceeded when deriving child keys and must be respected during transaction rescans. Child key derivation must not be possible if gap limit is hit. ### Script type restriction 3 or 5 bits (range 0-7 / 0-31) 0 no restriction 1 P2PKH compressed 2 P2PKH | P2SH 3 P2WPKH P2WSH nested in P2SH 4 P2WPKH | P2WSH If the total decoded serialization length is 275 bits (decode) or if the key material is 256 bits (encode), 3 bits are used for the script type. 5 bits are used
