[Bitcoin-development] Bitcoin address TTL key expiration?
Proxying another's idea, from CoinSummit. The request: It would be useful to limit the lifetime of a bitcoin address. Intentionally prevent (somehow) bitcoins being sent to a pubkey/pkh after the key expires. You could append don't [permit|confirm] after X [time|block] to the address I suppose. The metadata would not be digitally signed, but it would be hash-sealed. As address is a client-side notion, wallet clients would be the ones enforcing such a rule. Bitcoin protocol of course knows about keys, and key expiration is a well known and useful concept in public key cryptography. The best insertion point in the protocol for key expiration is an open question, if it's even a good idea at that level at all. Some flag no more TxOuts exactly like this [after X block?]? I readily admit I don't have good answers, but it does seem valuable IMO to * Prevent users from accidentally sending to an expired TxOut/pkh. This happens in the field. * Discourage address reuse * Enable sites that generate lots of keys to rotate ancient keys off their core systems. (HD wallets mitigate this) -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 10:00 AM, Jeff Garzik jgar...@bitpay.com wrote: Proxying another's idea, from CoinSummit. The request: It would be useful to limit the lifetime of a bitcoin address. Intentionally prevent (somehow) bitcoins being sent to a pubkey/pkh after the key expires. Payment request expiration was meant to address this. Adding an optional expiration timestamp to addresses would be possible, however, it would be a non-backward-compatible change and lots of software would have to be changed at this point. In my opinion encouraging the use of the payment protocol and deprecating the use of addresses is the best way forward, and not just for this reason. Wladimir -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 04:00:41AM -0400, Jeff Garzik wrote: Proxying another's idea, from CoinSummit. The request: It would be useful to limit the lifetime of a bitcoin address. Intentionally prevent (somehow) bitcoins being sent to a pubkey/pkh after the key expires. You could append don't [permit|confirm] after X [time|block] to the address I suppose. The metadata would not be digitally signed, but it would be hash-sealed. As address is a client-side notion, wallet clients would be the ones enforcing such a rule. Note that digitally signed has no value here without some kind of PKI/WoT/something else to know what key is doing the signing. I believe Jeff is really referring to the checksum by hash-sealed here, which is as good as is worth getting. Bitcoin protocol of course knows about keys, and key expiration is a well known and useful concept in public key cryptography. The best insertion point in the protocol for key expiration is an open question, if it's even a good idea at that level at all. Some flag no more TxOuts exactly like this [after X block?]? I readily admit I don't have good answers, but it does seem valuable IMO to * Prevent users from accidentally sending to an expired TxOut/pkh. This happens in the field. * Discourage address reuse * Enable sites that generate lots of keys to rotate ancient keys off their core systems. (HD wallets mitigate this) A few months ago I looked into what low-level details it'd take to add Bitcoin addresses to OpenPGP keys a few months ago; one of the requirements we came up with was to make sure the standard OpenPGP expiration machinery would still work. Basically in that model the Bitcoin address - most likely a stealth address for privacy - is added to the key as signed data. All signatures in OpenPGP have optional expiration times, and additionally they can be revoked after the fact if needed as well. Of course, such ideas aren't limited to OpenPGP - all payment protocols should consider adopting them. As for protocol level hacks, keep in mind that anything that makes a transaction invalid because of the presence of a specific scriptPubKey in a txout has the potential to make a whole chain of transactions become invalid during a reorg. Adding such protection in the form of IsStandard() policy would be ok, but as a protocol rule it'd be pretty dangerous. IMO much better to just solve the problem at the UI level. -- 'peter'[:-1]@petertodd.org 32d9d8942fe9461cce9db22a6cd86eacb5c18b415ebb649d signature.asc Description: Digital signature -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 4:19 AM, Wladimir laa...@gmail.com wrote: In my opinion encouraging the use of the payment protocol and deprecating the use of addresses is the best way forward, and not just for this reason. There are major gaps that the payment protocol doesn't cover. There are several deployed use cases where you are provided/request an address, an API provides one, and one or more incoming payments arrive as the user sends them over minutes/hours/days/weeks. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 10:23 AM, Jeff Garzik jgar...@bitpay.com wrote: On Tue, Jul 15, 2014 at 4:19 AM, Wladimir laa...@gmail.com wrote: There are major gaps that the payment protocol doesn't cover. There are several deployed use cases where you are provided/request an address, an API provides one, and one or more incoming payments arrive as the user sends them over minutes/hours/days/weeks. Couldn't these services return a payment message instead of an address? I agree that there is currently an UI issue here: there is no way in current wallets to store a payment message and pay to it later. We will need something like that for recurring payments as well. Bitcoin addresses were never designed with extensibility in mind. Before the payment protocol there have been lots of ideas to add functionality to them, but the underlying idea that they have to be handled by users manually means that they have to be as short as possible, which is a conflicting aim with extensibility... Wladimir -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
Payment Protocol would probably be the communication format for any new meta-data. What's the likelihood of something like this every making it on the blockchain? -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
The request: It would be useful to limit the lifetime of a bitcoin address. Not only useful but essential! Otherwise mobile clients can run out of RAM and have to cycle around and reuse addresses. Which is indeed why BIP70 has this feature. It was thought about quite some time ago. Addresses are an evolutionary dead end, they will never do everything we need them to do. If there's somewhere that's using addresses, that's somewhere we will eventually need to upgrade to use BIP70 instead. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] BIP 38 NFC normalisation issue
[+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: - Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK http://codepoints.net/U+03D2, COMBINING ACUTE ACCENT http://codepoints.net/U+0301, NULL http://codepoints.net/U+, DESERET CAPITAL LETTER LONG I http://codepoints.net/U+10400, PILE OF POO http://codepoints.net/U+1F4A9) - Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF - Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 - *Note:* The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
I think generally control-characters (such as \u) should be disallowed in passphrases. (Even the use of whitespaces is very questionable.) I'm ok with allowing pile-of-poo's. On mobile phones there is keyboards just containing emoticons -- why not allow those? Assuming NFC works of course. On 07/15/2014 03:07 PM, Eric Winer wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net mailto:m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: * Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK http://codepoints.net/U+03D2, COMBINING ACUTE ACCENT http://codepoints.net/U+0301, NULL http://codepoints.net/U+, DESERET CAPITAL LETTER LONG I http://codepoints.net/U+10400, PILE OF POO http://codepoints.net/U+1F4A9) * Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn * Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF * Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 * /Note:/ The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net mailto:Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
I have a python implementation that seems to pass this test vector: https://github.com/wozz/electrum/blob/bip38_import/lib/bip38.py#L299 On Jul 15, 2014, at 9:19 AM, Andreas Schildbach andr...@schildbach.de wrote: I think generally control-characters (such as \u) should be disallowed in passphrases. (Even the use of whitespaces is very questionable.) I'm ok with allowing pile-of-poo's. On mobile phones there is keyboards just containing emoticons -- why not allow those? Assuming NFC works of course. On 07/15/2014 03:07 PM, Eric Winer wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net mailto:m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: * Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK http://codepoints.net/U+03D2, COMBINING ACUTE ACCENT http://codepoints.net/U+0301, NULL http://codepoints.net/U+, DESERET CAPITAL LETTER LONG I http://codepoints.net/U+10400, PILE OF POO http://codepoints.net/U+1F4A9) * Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn * Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF * Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 * /Note:/ The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net mailto:Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
BIP70 does not work well for unknown number of future payments of unknown, unpredictable value. On Tue, Jul 15, 2014 at 6:25 AM, Mike Hearn m...@plan99.net wrote: The request: It would be useful to limit the lifetime of a bitcoin address. Not only useful but essential! Otherwise mobile clients can run out of RAM and have to cycle around and reuse addresses. Which is indeed why BIP70 has this feature. It was thought about quite some time ago. Addresses are an evolutionary dead end, they will never do everything we need them to do. If there's somewhere that's using addresses, that's somewhere we will eventually need to upgrade to use BIP70 instead. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 4:02 PM, Jeff Garzik jgar...@bitpay.com wrote: BIP70 does not work well for unknown number of future payments of unknown, unpredictable value. You can specify zero as an output value, in which case it's the same as no value specified. You can then just reuse the PaymentRequest until it expires. So I think it provides the same functionality already. Now sure, you'll get address reuse in this scenario, but that's no worse than with an extended textual address. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tuesday, July 15, 2014 8:00:41 AM Jeff Garzik wrote: Proxying another's idea, from CoinSummit. The request: It would be useful to limit the lifetime of a bitcoin address. Intentionally prevent (somehow) bitcoins being sent to a pubkey/pkh after the key expires. You could append don't [permit|confirm] after X [time|block] to the address I suppose. The metadata would not be digitally signed, but it would be hash-sealed. As address is a client-side notion, wallet clients would be the ones enforcing such a rule. I agree this would be useful for the permit case, but not the confirm case - it's important that transactions valid in block X also be equally valid in block X+1 to avoid reorg issues. Bitcoin protocol of course knows about keys, and key expiration is a well known and useful concept in public key cryptography. The best insertion point in the protocol for key expiration is an open question, if it's even a good idea at that level at all. Some flag no more TxOuts exactly like this [after X block?]? This would force every wallet to keep an index of all TXOs ever. I readily admit I don't have good answers, but it does seem valuable IMO to * Prevent users from accidentally sending to an expired TxOut/pkh. This happens in the field. * Discourage address reuse Actually, I think this may make address reuse easier, as with base58 adding data will make it impossible to tell at a glance when someone is reusing a key with just a different expiration... I suppose something other than base58 *could* be used to resolve this, however. * Enable sites that generate lots of keys to rotate ancient keys off their core systems. (HD wallets mitigate this) They can already do this. It's perfectly valid for wallets/services to ignore (and not consider as payment) transactions using an address more than once. There might be race attacks if this is implemented in an immediate fashon (attacker transaction gets mined first to kill a payment), but should be pretty safe after a few blocks. Luke -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
Unicode guarantees that null-terminated strings still work. U+ terminates a unicode (or C) string. strlen() gets the string byte count. mbstowcs() gets the character count. Whitespace can be problematic, but should be allowed. Control characters should be filtered. Emoticons probably cannot be filtered without substandard approaches such as character blacklists, a road you do not want to travel. (all this is simply standard practice) On Tue, Jul 15, 2014 at 9:07 AM, Eric Winer enwi...@gmail.com wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER LONG I, PILE OF POO) Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 Note: The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
Unicode guarantees that null-terminated strings still work. UTF-8 guarantees that. Other encodings do not, you can have null bytes in UTF-16 strings for example. Indeed most languages that use pascal-style encodings internally allow null characters in strings, it's just not a good idea to exploit that fact ... -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
Can you provide the rationale for standard practice? For example, why should whitespace be allowed? I regularly use trim() on any passphrase (or other input ftm). So what's the action point? Should we amend the spec to filter control characters? That would get rid of the \u problem. On 07/15/2014 05:17 PM, Jeff Garzik wrote: Unicode guarantees that null-terminated strings still work. U+ terminates a unicode (or C) string. strlen() gets the string byte count. mbstowcs() gets the character count. Whitespace can be problematic, but should be allowed. Control characters should be filtered. Emoticons probably cannot be filtered without substandard approaches such as character blacklists, a road you do not want to travel. (all this is simply standard practice) On Tue, Jul 15, 2014 at 9:07 AM, Eric Winer enwi...@gmail.com wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER LONG I, PILE OF POO) Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 Note: The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tuesday, July 15, 2014 3:11:25 PM Jeff Garzik wrote: On Tue, Jul 15, 2014 at 10:48 AM, Luke Dashjr l...@dashjr.org wrote: They can already do this. It's perfectly valid for wallets/services to ignore (and not consider as payment) transactions using an address more than once. There might be race attacks if this is implemented in an immediate fashon (attacker transaction gets mined first to kill a payment), but should be pretty safe after a few blocks. Sure it's valid. However, few users will appreciate you ignored my deposit as a feature. Payment protocol just doesn't well the use cases of, * an on-going payment stream from, e.g. Eligius to coinbase https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#Serialization_format * deposit addresses and deposit situations There's no reason deposits cannot use a unique payment request or address every time... -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
On whitespace: Security UX testing I've seen shows it is mentally easier for some users to memorize and use longer passphrases, if they are permitted spaces. I've not seen anything written on use of tabs/NLs/FFs in passphrases. I can see the logic of some systems, that convert \s+ into ' ' for purposes of password hashing, even though that might frustrate a security nerd or two. http://security.stackexchange.com/questions/32691/why-not-allow-spaces-in-a-password I do think control characters should be filtered. On Tue, Jul 15, 2014 at 11:32 AM, Andreas Schildbach andr...@schildbach.de wrote: Can you provide the rationale for standard practice? For example, why should whitespace be allowed? I regularly use trim() on any passphrase (or other input ftm). So what's the action point? Should we amend the spec to filter control characters? That would get rid of the \u problem. On 07/15/2014 05:17 PM, Jeff Garzik wrote: Unicode guarantees that null-terminated strings still work. U+ terminates a unicode (or C) string. strlen() gets the string byte count. mbstowcs() gets the character count. Whitespace can be problematic, but should be allowed. Control characters should be filtered. Emoticons probably cannot be filtered without substandard approaches such as character blacklists, a road you do not want to travel. (all this is simply standard practice) On Tue, Jul 15, 2014 at 9:07 AM, Eric Winer enwi...@gmail.com wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER LONG I, PILE OF POO) Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 Note: The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
On Tue, Jul 15, 2014 at 11:41 AM, Luke Dashjr l...@dashjr.org wrote: There's no reason deposits cannot use a unique payment request or address every time... Actually, and this is key, there _are_ reasons why deposits might not be able to use PaymentRequests. Payments happen even when wallets/computers are offline. If you have negotiated HD wallet details, you can use a new address every time, as mentioned. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin address TTL key expiration?
Actually, and this is key, there _are_ reasons why deposits might not be able to use PaymentRequests. Payments happen even when wallets/computers are offline. I don't understand this point. It's the *sender* that is parsing the PaymentRequest and following the instructions. By definition the sender must be online. A computer that is switched off cannot sign a transaction at all. If you have negotiated HD wallet details, you can use a new address every time, as mentioned. Yes, and an extension to BIP 70 to allow for this (or stealth addresses or whatever) has been discussed several times. This thread started by proposing (I think) an expiry time for addresses. BIP70 satisfies this use case, I think we all agree on that. Now for cases where someone can't use BIP70 for whatever reason, or it's suboptimal, absolutely we should design extensions to fix that. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
I was part of adding in that test vector, and I think it's a good test vector since it is an extreme edge-case of the current definition: If the BIP38 proposal allows any password that can be in UTF-8, NFC normalized form, those characters cover the various edge cases (combining characters, null character, astral range) that if your implementation doesn't handle, then it can't really be said to be BIP38-compatible/compliant, right? The passphrase in the test vector is NOT in NFC form; that's the point. Whatever implementation gets designed has to assume the input is not already NFC-normalized and needs to handle/sanitize that input before further processing. To test your implementation for compliance, you should not be inputting the NFC-normalized bytestring as the password input, you should be entering the original passphrase as the test. My original pull request for this change (https://github.com/bitcoin/bips/pull/29) shows a Python and a NodeJS way to input that test vector password as intended. Some input devices may already handle the input as NFC, which is great, but per the BIP38 proposal, that shouldn't be assumed, so various implementations are cross-compatible. If one implementation assumes the input is already NFC, they may encode/decode the password incorrectly, and lock a user out of their wallet. Android allows different user keyboards to be used, so I'm guessing there's one somewhere that allows manual entry of unicode codepoints that could be used to enter a null character, and with the next version of iOS, Apple devices will also get custom keyboard options, too, so even if the default Apple keyboard does NFC-form properly, other developers' keyboards may not. So while it is an extreme edge case, that is not very likely to be used as a real password by any user, that's what test vectors are for: to test for the edge case that you might not have expected and handled in your implementation. Brooks On Tue, Jul 15, 2014 at 8:07 AM, Eric Winer enwi...@gmail.com wrote: I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase include special characters. I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn m...@plan99.net wrote: [+cc aaron] We recently added an implementation of BIP 38 (password protected private keys) to bitcoinj. It came to my attention that the third test vector may be broken. It gives a hex version of what the NFC normalised version of the input string should be, but this does not match the results of the Java unicode normaliser, and in fact I can't even get Python to print the names of the characters past the embedded null. I'm curious where this normalised version came from. Given that pile of poo is not a character I think any sane user would put into a passphrase, I question the value of this test vector. NFC form is intended to collapse things like umlaut control characters onto their prior code point, but here we're feeding the algorithm what is basically garbage so I'm not totally surprised that different implementations appear to disagree on the outcome. Proposed action: we remove this test vector as it does not represent any real world usage of the spec, or if we desperately need to verify NFC normalisation I suggest using a different, more realistic test string, like Zürich, or something written in Thai. Test 3: - Passphrase ϓ␀Ѐ (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK UPSILON WITH HOOK http://codepoints.net/U+03D2, COMBINING ACUTE ACCENT http://codepoints.net/U+0301, NULL http://codepoints.net/U+, DESERET CAPITAL LETTER LONG I http://codepoints.net/U+10400, PILE OF POO http://codepoints.net/U+1F4A9) - Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF - Unencrypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 - *Note:* The non-standard UTF-8 characters in this passphrase should be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before further processing -- Want fast and easy access to all the code in your
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
Yes, we know, Andreas' code is indeed doing normalisation. However it appears the output bytes end up being different. What I get back is: cf9300*01*303430300166346139 vs cf9300*f0*909080f09f92a9 from the spec. I'm not sure why. It appears this is due to the character from the astral planes. Java is old and uses 16 bit characters internally - it wouldn't surprise me if there's some weirdness that means it doesn't/won't support this kind of thing. I recommend instead that any implementation that wishes to be compatible with JVM based wallets (I suspect Android is the same) just refuse any passphrase that includes characters outside the BMP. At least unless someone can find a fix. I somehow doubt this will really hurt anyone. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP 38 NFC normalisation issue
If the user creates a password on an iOS device with an astral character and then can't enter that password on a JVM wallet, that sucks. If JVMs really can't support unicode NFC then that's a strong case to limit the spec to the subset of unicode that all popular platforms can support, but it sounds like it might just be a JVM string library bug that could hopefully be reported and fixed. I get the same result as in the test case using apple's CFStringNormalize(passphrase, kCFStringNormalizationFormC); Aaron Voisine breadwallet.com On Tue, Jul 15, 2014 at 11:20 AM, Mike Hearn m...@plan99.net wrote: Yes, we know, Andreas' code is indeed doing normalisation. However it appears the output bytes end up being different. What I get back is: cf930001303430300166346139 vs cf9300f0909080f09f92a9 from the spec. I'm not sure why. It appears this is due to the character from the astral planes. Java is old and uses 16 bit characters internally - it wouldn't surprise me if there's some weirdness that means it doesn't/won't support this kind of thing. I recommend instead that any implementation that wishes to be compatible with JVM based wallets (I suspect Android is the same) just refuse any passphrase that includes characters outside the BMP. At least unless someone can find a fix. I somehow doubt this will really hurt anyone. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
