Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gregory Maxwell 
On Thu, Jul 17, 2014 at 2:35 PM, Kaz Wesley  wrote:
> A node should be able to forget invs it has seen without invalidating what
> peers
> know about its known txes. To allow for this, a node assembles a bloom
> filter of

Another option would be to just guarantee to keep at least the last N
sent in each direction to bound memory usage. N could be negotiated.

Going more complex than that may not have wins enough to justify it...
would be good to measure it.


(If you're not aware of it, check out—
https://en.bitcoin.it/wiki/User:Gmaxwell/block_network_coding for a
more complex idea)

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Kaz Wesley 
I'm moving this design document to a gist so that I can integrate
changes as they come up:
https://gist.github.com/kazcw/43c97d3924326beca87d
One thing that I think is an important improvement over my initial
idea is that the bloom filters don't need to be kept around and built
up, they can just be one-shot and clear any matching entries from the
set of known-knowns upon arrival -- provided a node is careful to
ensure the txes it wants to forget are known-known-known (which isn't
as bad as it sounds) to the peer it's telling it's forgetting them
when the forget-filter arrives.

On Thu, Jul 17, 2014 at 3:46 PM, Gavin Andresen  wrote:
>
> A couple of half-baked thoughts:
>
> On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley  wrote:
>>
>> If there's support for this proposal, I can begin working on the specific
>> implementation details, such as the bloom filters, message format, and
>> capability advertisment, and draft a BIP once I have a concrete proposal for
>> what those would look like and a corresponding precise cost/benefit analysis.
>
>
> I'd encourage you to code up a prototype first (or at the same time), in 
> whatever programming language / networking library you're most familiar with.
>
> Maybe not even using the existing p2p protocol; there could be a mining-only 
> very-fast-block-propagation network separate from the existing p2p network.
>
> Combining your optimizations with "broadcast as many near-miss blocks as 
> bandwidth will allow" on a mining backbone network should allow insanely fast 
> propagation of most newly solved blocks.
>
> --
> --
> Gavin Andresen

Thanks Gavin, I am planning on working out the design details as I
work on a prototype. I have the beginnings of a previous shot at
implementing this in bitcoind to start from but my new design has some
important improvements to add to that.

-kaz

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gavin Andresen 
A couple of half-baked thoughts:

On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley  wrote:

> If there's support for this proposal, I can begin working on the specific
> implementation details, such as the bloom filters, message format, and
> capability advertisment, and draft a BIP once I have a concrete proposal
> for
> what those would look like and a corresponding precise cost/benefit
> analysis.
>

I'd encourage you to code up a prototype first (or at the same time), in
whatever programming language / networking library you're most familiar
with.

Maybe not even using the existing p2p protocol; there could be a
mining-only very-fast-block-propagation network separate from the existing
p2p network.

Combining your optimizations with "broadcast as many near-miss blocks as
bandwidth will allow" on a mining backbone network should allow insanely
fast propagation of most newly solved blocks.

-- 
--
Gavin Andresen
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Kaz Wesley 
OVERVIEW

To improve block propagation, add a new block message that doesn't include
transactions the peer is known to have. The message must never require an
additional round trip due to any transactions the peer doesn't have, but
should
be compatible with peers sometimes forgetting transactions they have known.

APPROACH

For peers advertising support for squashed blocks: a node tracks what txes
it
knows each peer has seen (inv received, tx sent, tx appeared in competing
block
known to peer). Nodes push block contents as txes-not-already-known +
txids-known.

A node should be able to forget invs it has seen without invalidating what
peers
know about its known txes. To allow for this, a node assembles a bloom
filter of
a set of txes it is going to forget, and sends it to peers. The node can
erase
the txes as soon as no blocks requested before the filter was pushed are in
flight (relying on the assumption that messages can be expected to be
processed
in order).

When a node receives a forgotten-filter, it ORs it into its
forgotten-filter for
that peer. Any transactions matching the forgotten-filter are always
included in
full with a block. If the filter is getting full, the node can just clear it
along with peer.setTxKnown.

COSTS

Bloom filtering:
Since the bloom filter is likely to grow slowly and can be dropped when it
is
becoming full, a cheap set of hash functions and element size can be used to
keep overhead more restricted than the bloom filtering done for spv. It's
important for testing txes against the filter to be fast so that it doesn't
delay pushing the block more than the squashing helps.
Nodes currently forget txes rarely, so the bloom filters would only need to
be
used at all under conditions that are not currently common -- but I think
they're important to include to allow for different node behavior in this
regard
in the future.

Tracking txes known to peers:
A multimap of txid->peerId would obviate the current setCurrentlyKnown, and
would not take much more space since each additional peer adds about 1
peerId
per txid (setCurrentlyKnown keeps a uint256 per peer per txid, although it
tracks somewhat fewer txid per node).

Potential vulnerabilities:
- Since the bloom filters will have lower maximum overhead than the current
SPV
  filters and can be dropped at will, this shouldn't enable any resource
  exhaustion attacks that aren't already possible.
- A squashed block with bogus or missing data would be easily detected not
to
  produce the correct merkle root for its BlockHeader.

BENEFITS

Assuming a fairly typical 500 tx block with transaction sizes averaging 300b
(both on the low side), for a 150kb block:

% pruned | block size reduction | relative size reduction
 |  | ---
100  | 134 kB   | 89%
50   | 67 kB| 45%
25   | 33.5 kB  | 17%

I've been doing some logging, and when my node pushes a block to a peer it
seems
to typically know that a peer has seen most of the txes in the block. Even
in
the case of a small block with only 25% known-known transactions, total
network
bandwidth saved is greater than the bloom filters transmitted unless a node
is
forgetting transactions so rapidly that it pushes new maximum-size
forget-filters every block.

So this is a net gain even in total bandwidth usage, but most importantly
it's
an improvement in block propagation rate and in how block propagation rate
scales with additional transactions.

IMPLEMENTATION QUESTIONS

How should block squashing capability be advertised -- new service bit?

Bloom filters:
- How fast to test against could a suitable bloom filter be made?
- How much memory would each filter need to take, at maximum?
- Can the inputs all being 32 byte hashes be used to optimize filter hash
  calculations?

ROADMAP

If there's support for this proposal, I can begin working on the specific
implementation details, such as the bloom filters, message format, and
capability advertisment, and draft a BIP once I have a concrete proposal for
what those would look like and a corresponding precise cost/benefit
analysis.

--kaz
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Pay to MultiScript hash:

2014-07-17 Thread Gregory Maxwell 
On Wed, Jul 16, 2014 at 10:56 AM, Jeremy  wrote:
> Hey all,
> I had an idea for a new transaction type. The base idea is that it is
> matching on script hashes much like pay to script hash, but checks for one
> of N scripts.

This seems strictly less flexible and efficient than the Merkelized
Abstract Syntax Tree construction, though perhaps slightly easier to
implement it wouldn't be any easier to deploy.

Something like this was very recently proposed on this list (by Tier
Nolan), you might want to see the "Selector Script" thread.

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Pay to MultiScript hash:

2014-07-17 Thread Jeremy 
* the general cost of any network-wide change, versus P2SH which is
already analyzed by devs, rolled out and working
* the cost of updating everybody to relay this new transaction type,
whereas P2SH Just Works already
fair -- I think that there may be a big benefit realizable with this kind
of system.

* cost of increasing rate of UTXO growth versus P2SH
This operation is similar in cost to multisig? Although I suppose there is
the proposal to make all multisigs p2sh

* the cost of P2SH output is predictable, versus less predictable outputs
 * "default public", versus P2SH's "default private"
-- Can you elaborate on these?

I think part of the problem is that there is low incentive for
development/cataloging  of these useful types of script because there isn't
a horizon on getting them broadcastable by nodes other than testnet? Even
with pay to script hash it is still currently relegated to a subset of
script types iirc (I think I'm wrong on this one maybe (hopefully) -- if
so, let's get writing!)?



Hmm... another idea... what about doing a p2sh with a switch statement, ie:

OP_HASH160

Re: [Bitcoin-development] Decentralizing ming

2014-07-17 Thread slush 
On Thu, Jul 17, 2014 at 6:14 PM, Jeff Garzik  wrote:

> Historical note:  On one hand, Satoshi seemed to dislike the early
> emergence of GPU mining pools quite a bit.
>

To my knowledge, Satoshi left the project before mining pools got a
traction.

slush
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Mining Hashrate Caps

2014-07-17 Thread David A. Harding 
On Thu, Jul 17, 2014 at 09:35:20AM -0400, Mark Friedenbach wrote:
> Can someone explain to these guys and the public why promising to limit
> yourselves to *only* a 50% chance of successfully double-spending a 6
> confirm transaction is still not acceptable?

Hi, Mark.

We were asked on the bitcoin-documentation mailing list about a month
ago to work on something like this and we're getting close to a pull
request for the Bitcoin.org website.  There's a preview here:

http://dg0.dtrt.org/en/mining#lucky-attack

(Remember, it's a preview and still being actively written/edited.)

Discussion about that doc belongs on the bitcoin-documentation mailing
list. Here's the particular thread:

https://groups.google.com/forum/#!topic/bitcoin-documentation/PKwBcroWGGg

Thanks,

-Dave
-- 
David A. Harding

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Decentralizing ming

2014-07-17 Thread Jeff Garzik 
Define acceptable.  The 40% thing is marketing and a temporary
solution.  And people come down on both sides of whether or not
marketing "40%" is a good idea.

I think it is a baby step that is moving in the right direction.  You
want the numbers and sentiment moving in that direction (down, versus
"own the market! ").

The more critical piece is fleshing out the various proposals and
technical solutions for decentralized transaction selection and other
aspects of SPOF-proofing mining.

Historical note:  On one hand, Satoshi seemed to dislike the early
emergence of GPU mining pools quite a bit.  On the other hand, Satoshi
noted that the network would probably devolve down to a few big
players if we ever reached VISA/MC transaction levels.  Satoshi
clearly never figured this part out :)

Today, there is consensus on the need for a "keep bitcoin free and
open" technical solution, but it remains to be seen how much we
engineers can really do to make life fair.  Making transaction
selection a bit more independent from hashpower seems one step.  There
are several other proposals floating about.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.  https://bitpay.com/

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Mining Hashrate Caps

2014-07-17 Thread Mark Friedenbach 
Can someone explain to these guys and the public why promising to limit
yourselves to *only* a 50% chance of successfully double-spending a 6
confirm transaction is still not acceptable?

q=0.4
z=0 P=1
z=1 P=0.828861
z=2 P=0.736403
z=3 P=0.664168
z=4 P=0.603401
z=5 P=0.550625
z=6 P=0.50398
z=7 P=0.462301
z=8 P=0.424782
z=9 P=0.390828
z=10P=0.359976
z=11P=0.331858
z=12P=0.306167
z=13P=0.282649
z=14P=0.261083
z=15P=0.24128
z=16P=0.223076
z=17P=0.206324
z=18P=0.190896
z=19P=0.176676


On 07/17/2014 06:59 AM, Melvin Carvalho wrote:
> I noticed this article today. 
> 
> GHash Commits to 40% Hashrate Cap at Bitcoin Mining Summit
> 
> http://www.coindesk.com/ghash-commits-40-hashrate-cap-bitcoin-mining-summit/
> 
> Here's a quote from Satoshi when the mining arms race began:
> 
> "We should have a gentleman’s agreement to postpone the GPU arms race as
> long as we can for the good of the network. It’s much easer to get new
> users up to speed if they don’t have to worry about GPU drivers and
> compatibility. It’s nice how anyone with just a CPU can compete fairly
> equally right now."
> 
> Maybe outdated now, but I thought it was interesting.
> 
> 
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> 
> 
> 
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

2014-07-17 Thread Andreas Schildbach 
Here is a good article that helped me with what's going wrong:

http://www.oracle.com/technetwork/articles/javase/supplementary-142654.html

Basically, Java is stuck at 16 bits per char due to legacy reasons. They
admit that for a new language, they would probably use 32 (or 24?) bits
per char.

\u literals express UTF-16 encoding, so you have to use 16 bits. I
learned that for codepoint 0x010400, I could write "\uD801\uDC00", which
is the UTF-16 encoding of that codepoint.

Other languages have literals for codepoints. E.g. Python can use
u"\U00010400" or HTML has 𐐀  Unfortunately, Java is missing such
a construct (at least in Java6).


On 07/17/2014 12:59 PM, Mike Hearn wrote:
> Glad we got to the bottom of that. That's quite a nasty
> compiler/language bug I must say. Not even a warning. Still, python
> crashes when trying to print the name of a null character. It wouldn't
> surprise me if there are other weird issues lurking. Would definitely
> sleep better with a more restricted character set.
> 
> On 17 Jul 2014 00:04, "Andreas Schildbach"  > wrote:
> 
> Please excuse me. I had a more thorough look at the original problem and
> found that the only problem with the original test case was that you
> cannot specify codepoints from the SMP using \u in Java. I always tried
> \u010400 but that doesn't work.
> 
> Here is a fix for bitcoinj. The test now passes.
> 
> https://github.com/bitcoinj/bitcoinj/pull/143
> 
> We can (and probably should) still need to filter control chars, I'll
> have a look at that now again.
> 
> 
> On 07/16/2014 11:06 PM, Aaron Voisine wrote:
> > If I first remove \u, so the non-normalized passphrase is
> > "\u03D2\u0301\U00010400\U0001F4A9", and then NFC normalize it, it
> > becomes "\u03D3\U00010400\U0001F4A9"
> >
> > UTF-8 encoded this is: 0xcf93f0909080f09f92a9 (not the same as what
> > you got, Andreas!)
> >
> > Encoding private key:
> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> > with this passphrase, I get a BIP38 key of:
> > 6PRW5o9FMb4hAYRQPmgcvVDTyDtr6R17VMXGLmvKjKVpGkYhBJ4uYuR9wZ
> >
> > I recommend rather than simply removing control characters from the
> > password that instead the spec require that passwords containing
> > control characters are invalid. We don't want people trying to be
> > clever and putting them in thinking they are adding to the password
> > entropy.
> >
> > Also for UI compatibility across many platforms, I'm also in favor
> > disallowing any character below U+0020 (space)
> >
> > I can submit a PR once we figure out why Andreas's passphrase was
> > different than what I got.
> >
> > Aaron Voisine
> > breadwallet.com 
> >
> >
> > On Wed, Jul 16, 2014 at 4:04 AM, Andreas Schildbach
> > mailto:andr...@schildbach.de>> wrote:
> >> Damn, I just realized that I implement only the decoding side of
> BIP38.
> >> So I cannot propose a complete test vector. Here is what I have:
> >>
> >>
> >> Passphrase: ϓ␀𐐀💩 (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK
> >> UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL
> LETTER
> >> LONG I, PILE OF POO)
> >>
> >> Passphrase bytes after removing ISO control characters and NFC
> >> normalization: 0xcf933034303066346139
> >>
> >> Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
> >>
> >> Unencrypted private key (WIF):
> >> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> >>
> >>
> >> Can someone calculate the encrypted key from it (using whatever
> >> implementation) and I will verify it decodes properly in bitcoinj?
> >>
> >>
> >>
> >> On 07/16/2014 12:46 PM, Andreas Schildbach wrote:
> >>> I will change the bitcoinj implementation and propose a new test
> vector.
> >>>
> >>>
> >>>
> >>> On 07/16/2014 11:29 AM, Mike Hearn wrote:
>  Yes sorry, you're right, the issue starts with the null code point.
>  Python seems to have problems starting there too. It might work
> if we
>  took that out.
> 
> 
>  On Wed, Jul 16, 2014 at 11:17 AM, Andreas Schildbach
>  mailto:andr...@schildbach.de>
> >> wrote:
> 
>  Guys, you are always talking about the Unicode astral
> plane, but in fact
>  its a plain old (ASCII) control character where this
> problem starts and
>  likely ends: \u.
> 
>  Let's ban/filter ISO control characters and be done with
> it. Most
>  control characters will never be enterable by any keyboard
> into a
>  password field. Of course I assume that
> Character.isISOControl() works
>  consistently across

[Bitcoin-development] Mining Hashrate Caps

2014-07-17 Thread Melvin Carvalho 
I noticed this article today.

GHash Commits to 40% Hashrate Cap at Bitcoin Mining Summit

http://www.coindesk.com/ghash-commits-40-hashrate-cap-bitcoin-mining-summit/

Here's a quote from Satoshi when the mining arms race began:

"We should have a gentleman’s agreement to postpone the GPU arms race as
long as we can for the good of the network. It’s much easer to get new
users up to speed if they don’t have to worry about GPU drivers and
compatibility. It’s nice how anyone with just a CPU can compete fairly
equally right now."

Maybe outdated now, but I thought it was interesting.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

2014-07-17 Thread Mike Hearn 
Glad we got to the bottom of that. That's quite a nasty compiler/language
bug I must say. Not even a warning. Still, python crashes when trying to
print the name of a null character. It wouldn't surprise me if there are
other weird issues lurking. Would definitely sleep better with a more
restricted character set.
On 17 Jul 2014 00:04, "Andreas Schildbach"  wrote:

> Please excuse me. I had a more thorough look at the original problem and
> found that the only problem with the original test case was that you
> cannot specify codepoints from the SMP using \u in Java. I always tried
> \u010400 but that doesn't work.
>
> Here is a fix for bitcoinj. The test now passes.
>
> https://github.com/bitcoinj/bitcoinj/pull/143
>
> We can (and probably should) still need to filter control chars, I'll
> have a look at that now again.
>
>
> On 07/16/2014 11:06 PM, Aaron Voisine wrote:
> > If I first remove \u, so the non-normalized passphrase is
> > "\u03D2\u0301\U00010400\U0001F4A9", and then NFC normalize it, it
> > becomes "\u03D3\U00010400\U0001F4A9"
> >
> > UTF-8 encoded this is: 0xcf93f0909080f09f92a9 (not the same as what
> > you got, Andreas!)
> >
> > Encoding private key: 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> > with this passphrase, I get a BIP38 key of:
> > 6PRW5o9FMb4hAYRQPmgcvVDTyDtr6R17VMXGLmvKjKVpGkYhBJ4uYuR9wZ
> >
> > I recommend rather than simply removing control characters from the
> > password that instead the spec require that passwords containing
> > control characters are invalid. We don't want people trying to be
> > clever and putting them in thinking they are adding to the password
> > entropy.
> >
> > Also for UI compatibility across many platforms, I'm also in favor
> > disallowing any character below U+0020 (space)
> >
> > I can submit a PR once we figure out why Andreas's passphrase was
> > different than what I got.
> >
> > Aaron Voisine
> > breadwallet.com
> >
> >
> > On Wed, Jul 16, 2014 at 4:04 AM, Andreas Schildbach
> >  wrote:
> >> Damn, I just realized that I implement only the decoding side of BIP38.
> >> So I cannot propose a complete test vector. Here is what I have:
> >>
> >>
> >> Passphrase: ϓ␀𐐀💩 (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK
> >> UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER
> >> LONG I, PILE OF POO)
> >>
> >> Passphrase bytes after removing ISO control characters and NFC
> >> normalization: 0xcf933034303066346139
> >>
> >> Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
> >>
> >> Unencrypted private key (WIF):
> >> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> >>
> >>
> >> Can someone calculate the encrypted key from it (using whatever
> >> implementation) and I will verify it decodes properly in bitcoinj?
> >>
> >>
> >>
> >> On 07/16/2014 12:46 PM, Andreas Schildbach wrote:
> >>> I will change the bitcoinj implementation and propose a new test
> vector.
> >>>
> >>>
> >>>
> >>> On 07/16/2014 11:29 AM, Mike Hearn wrote:
>  Yes sorry, you're right, the issue starts with the null code point.
>  Python seems to have problems starting there too. It might work if we
>  took that out.
> 
> 
>  On Wed, Jul 16, 2014 at 11:17 AM, Andreas Schildbach
>  mailto:andr...@schildbach.de>> wrote:
> 
>  Guys, you are always talking about the Unicode astral plane, but
> in fact
>  its a plain old (ASCII) control character where this problem
> starts and
>  likely ends: \u.
> 
>  Let's ban/filter ISO control characters and be done with it. Most
>  control characters will never be enterable by any keyboard into a
>  password field. Of course I assume that Character.isISOControl()
> works
>  consistently across platforms.
> 
> 
> http://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl%28char%29
> 
> 
>  On 07/16/2014 12:23 AM, Aaron Voisine wrote:
>  > If the user creates a password on an iOS device with an astral
>  > character and then can't enter that password on a JVM wallet,
> that
>  > sucks. If JVMs really can't support unicode NFC then that's a
> strong
>  > case to limit the spec to the subset of unicode that all popular
>  > platforms can support, but it sounds like it might just be a JVM
>  > string library bug that could hopefully be reported and fixed.
> I get
>  > the same result as in the test case using apple's
>  > CFStringNormalize(passphrase, kCFStringNormalizationFormC);
>  >
>  > Aaron Voisine
>  > breadwallet.com 
>  >
>  >
>  > On Tue, Jul 15, 2014 at 11:20 AM, Mike Hearn   > wrote:
>  >> Yes, we know, Andreas' code is indeed doing normalisation.
>  >>
>  >> However it appears the output bytes end up being different.
> What
>  I get back
>  >> is:
>  >