Re: [Bitcoin-development] Proposal for extra nonce in block header

[Mike Hearn]

Although I agree 32 bits for a version is overkill, I really don't like the
idea of you simply ignoring the protocol spec to try and reduce your own
costs. Especially because in future we should make unknown versions a
validation rule, so we can easily trigger hard forks.

If this change was introduced through a proper process and software was
properly upgraded to understand the new header format, that'd be one thing.
Arbitrarily exploiting what is IMHO a missing rule in the rule set to shave
a bit more profit is something else.


On Sun, May 4, 2014 at 5:14 PM, Timo Hanke  wrote:

> > If changing the structure of the block header, wouldnt you also need to
> > increment the version number to 3?
>
> No, in this case I don't think so. Incrementing the version number has
> two purposes:
>
> 1. inform old clients that something new is going on
> 2. be able to phase out old version numbers and block them once the new
> version number becomes a supermajority.
>
> None of these two is necessary here. Old clients already recognize the
> new block headers as something new because they look like very high
> version numbers to them. And there is no reason to ever phase out blocks
> that have zero in the MSBs of the version.
>
> On Sun, Apr 27, 2014 at 10:17:11AM +0200, Melvin Carvalho wrote:
> > On 27 April 2014 09:07, Timo Hanke  wrote:
> >
> > I'd like to put the following draft of a BIP up for discussion.
> >
> > Timo
> >
> > # Abstract
> > There are incentives for miners to find cheap, non-standard ways to
> > generate new work, which are not necessarily in the best interest of
> the
> > protocol.
> > In order to reduce these incentives this proposal re-assigns 2 bytes
> from
> > the version field of the block header to a new extra nonce field.
> > # Copyright
> > # Specification
> > The block version number field in the block header is reduced in
> size from
> > 4 to 2 bytes.
> > The third and fourth byte in the block header are assigned to the
> new extra
> > nonce field inside the block header.
> > # Motivation
> > The motivation of this proposal is to provide miners with a cheap
> > constant-complexity method to create new work that does not require
> > altering the transaction tree.
> >
> > Furthermore, the motivation is to protect the version and timestamp
> fields
> > in the block header from abuse.
> > # Rationale
> > Traditionally, the extra nonce is part of the coinbase field of the
> > generation transaction, which is always the very first transaction
> of a
> > block.
> > After incrementing the extra nonce the minimum amount of work a
> miner has
> > to do to re-calculate the block header is a) to hash the coinbase
> > transaction and b) to re-calculate the left-most branch of the
> merkle tree
> > all the way to the merkle root.
> > This is necessary overhead a miner has to do besides hashing the
> block
> > header itself.
> > We shall call the process that leads to a new block header from the
> same
> > transaction set the _pre-hashing_.
> >
> > First it should be noted that the relative cost of pre-hashing in its
> > traditional form depends
> > on the block size, which may create an unwanted incentive for miners
> > to keep the block size small. However, this is not the main
> motivation for
> > the current proposal.
> >
> > While the block header is hashed by ASICs, pre-hashing typically
> happens on
> > a CPU because of the greater flexibility required.
> > Consequently, as ASIC cost per hash performance drops the relative
> cost of
> > pre-hashing increases.
> >
> > This creates an incentive for miners to find cheaper ways to create
> new
> > work than by means of pre-hashing.
> > An example of this currently happening is the on-device rolling of
> the
> > timestamp into the future.
> > These ways of creating new work are unlikely to be in the best
> interest of
> > the protocol.
> > For example, rolling the timestamp faster than the real time is
> unwanted
> > (more so on faster blockchains).
> >
> > The version number in the block header is a possible target for
> alteration
> > with the goal of cheaply creating new work.
> > Currently, blocks with arbitrarily large version numbers get relayed
> and
> > accepted by the network.
> > As this is unwanted behaviour, there should not exist any incentive
> for a
> > miner to abuse the version number in this way.
> >
> > The solution is to reduce the range of version numbers from 2^32 to
> 2^16
> > and to declare the third and forth bytes of the block header as
> legitimate
> > space for an extra nonce.
> > This will reduce the incentive for a miner to abuse the shortened
> version
> > number by a factor in the order of 2^16.
> >
> > As a side effect, this proposal greatly reduces the bandwidth
> requirements

Re: [Bitcoin-development] Announcing the Statoshi fork

[Mike Hearn]

Really nice! We definitely need to put together a team who really cares
about the operations side of the network and this is a fantastic start.

It'd be nice if you didn't assume knowledge of what statsd is out of the
box. Given the name I'd assumed it was a small UNIX daemon but it seems
it's actually a Javascript thingy?

It looks like putting together a monitored bitcoind setup can be quite a
lot of work. I wonder if there are ways to simplify it. For example, would
it make sense for someone to run a community statsd and graphite instance,
so we can get aggregate statistics across many nodes and the node operators
don't have to set everything up themselves? Does that make any sense?
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Announcing the Statoshi fork

[Mike Hearn]

It looks like the packet format statsd expects is rather simple - it should
be easy to experiment with.

Perhaps a good next step would be to improve your patch so that someone can
get a feed of the stats packets via TCP by e.g. ssh tunnelling to their
host. Once it's easy to get a feed of simple stats packets, people can
easily experiment with different kinds of aggregation and monitoring
software from their desktop.
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Announcing the Statoshi fork

[Mike Hearn]

I think there a few different possible ways to go here.

One is to try and simplify the setup of all the components so it all gets
installed together. That might be feasible in some quite restricted setups
but the installation instructions for Graphite look kind of terrifying.

Another is to export stats over regular TCP and make them public so
literally anyone can listen to the stats feed for any node. Then people who
dig stats and graphs could work on stats aggregators that give global
network visibility independently, effectively crawling the p2p network for
data. It'd have the advantage of having zero setup for the node operators
and not require much in the way of resources.

For what it's worth, although the environment is a bit different inside
Google the latter approach is used. Monitoring servers locate servers of
interest via a discovery service, connect to them and start streaming stats
data into a database service that can then be queried later to get graphs.

The stats are also run through various rules to obtain alerts about
problematic conditions. For example, if a subset of the network splits it
might be hard to notice that if the node operators aren't paying attention
and Matt's fork alert/emailing code isn't set up. But if there was a site
crawling nodes and aggregating chain heights by version, that could trigger
an alert to people who *are* paying attention.

I know from practical experience that monitoring and analysis tends to
appeal more to certain types of people than others. So I quite like the
"let anyone monitor" approach. However, it may not be appropriate in a P2P
network, I did not think about it much.

Obviously I'm assuming none of the stats expose privacy sensitive data.



On Wed, May 7, 2014 at 10:18 PM, Wladimir  wrote:

> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp 
> wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > I agree that it would be awesome to offer these types of stats with the
> installer; unfortunately the route I've taken has dependencies on several
> other other pieces of software to do all the heavy lifting of stats
> aggregation and chart rendering. I'm assuming that you would not want to
> build any of that processing into Bitcoin Core itself; would you be opposed
> to packaging other software along with the installer?
>
> Depends on just how much stuff it is. The idea is primarily to have an
> installer for running a (wallet-less) node as an OS background
> service.
>
> Having some statistics available would be worth some extra download
> size, otherwise it would be pretty much invisible.
>
> We'd already decided that we would need something like Python for the
> stats service. Implementing things like web services in C++ is just
> not realistic given the time constraints and the great already-written
> code that is out there. As an optional tool it should be external, not
> part of bitcoind itself.
>
> I suppose the chart rendering happens client-side? In that case the
> web service just has to collect and provide the data, and serve static
> html/js files.
>
> Wladimir
>
>
> --
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Announcing the Statoshi fork

[Mike Hearn]

>
> In a way it looks similar to how the Bitcoin DNS seeds work, trying to
> find good and stable nodes, although more extensive.


Yeah, it's somewhat similar, except that Tor directory authorities are
authenticated (the public keys are in the source code), whereas DNS seeds
aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor
does. For Tor having a P2P network is just something that's needed in order
to build an anonymity network, but it's not actually the goal. For us,
decentralisation is pretty much the end goal.
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] ECDH in the payment protocol

[Mike Hearn]

I wrote an article about an ECDH extension for BIP 70:

   https://medium.com/p/cb2f81962c1b

The article is meant for people who don't follow bitcoin-development so
I'll summarise it here:

   - The notion of being able to publish a piece of data once and use it to
   receive lots of payments without any coordination is really useful. This is
   the idea behind the stealth address proposal.

   - Stealth addresses don't fit with the payment protocol, because they're
   a new kind of address (obviously).

   - Stealth addresses are not backwards compatible. If you give someone a
   stealth address and their wallet doesn't support it, they can't pay you,
   not even with worse privacy. Sometimes people may optionally want that
   behaviour but stealth addresses have it all the time.

   - The proposed stealth address design makes huge sacrifices to try and
   keep everything within the block chain. It bloats the chain with OP_RETURN
   stuff that isn't a part of the validation. But more seriously, the only way
   to make it efficient enough for lightweight clients is to reduce the
   "stealthyness". The more efficient you make your address the less private
   it becomes. This is somewhat similar to the dilemma we have with Bloom
   filtering, except Bloom filters are transient and can only be used to link
   addresses by someone who observes them on the wire. Stealth addresses
   record the relationship in the block chain forever.

   - The design makes these sacrifices to avoid moving data around outside
   the block chain. But with BIP 70 that's the direction we're heading in
   anyway. So by adding ECDH to the payment protocol and putting our effort
   into making BIP 70 work really well for everyone, we end up killing
   multiple birds with one stone. The same work that resolves the privacy
   problems inherent in the stealth address design also allows us to attach
   messages to payments and other commonly requested features.

There's a straw man in the article that I recreate here:

message Output {
   optional uint64 amount = 1 [default = 0];
   optional bytes script = 2;
   *optional boolean accept_ecdh = 3;  // Requires script to be a
pay-to-pubkey output.*
}

message Payment {
   optional bytes merchant_data = 1;
   repeated bytes transactions = 2;
   repeated Output refund_to = 3;
   optional string memo = 4;
   *repeated bytes ecdh_nonces = 5;*
}

The way the nonces are combined to arrive at the address could be the same
as in the current stealth address spec. A wallet that doesn't understand
ECDH but does understand raw BIP 70 would deliver the money to the base
address, which receiving wallets would look for too - so it's backwards
compatible. The nonces stay out of the block chain. The transactions are
delivered directly to the recipient so there's no problems with trying to
make it fit with Bloom/prefix filtering.

To make this work there obviously has to be a backchannel from payer to
payee. BIP 70 is mostly used by web shops today so that back channel is
just HTTPS to the website itself, but shops benefit less from ECDH than
others do. So we need a simple email-like store and forward network where
HTTP POSTs to a server get queued up and delivered later (or possibly
forwarded to another store-and-forward network like the Android push
network). Most of the article discusses how best to build such a thing.

The justification for the original stealth address design can be summed up
as "it's easier to [ab]use the Bitcoin network for delivery of short
messages than use a different system". But there are just so many features
we may want to add into the Payment message in future it seems better to
crack the SaF problem earlier rather than continue trying to jam a square
peg into a round hole. There are lots of very reliable SAF networks around
(email, instant messaging, etc) so it doesn't seem infeasible.

Thoughts welcome!
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ECDH in the payment protocol

[Mike Hearn]

>
> Of course we quickly rejected the idea of depending solely on a
> communications backchannel to retrieve funds. Any communications medium
> that isn't the blockchain makes the payment non-atomic


Yes, I know you rejected this design, which is why I'm now proposing it
instead. I think you made the wrong design call, but at any rate, it's
something reasonable people can disagree on.

Payment messages are sent directly to the merchant, who takes
responsibility for broadcast. Once you delivered transactions to the
merchant successfully, from your perspective the payment is made. A good
store and forward network doesn't allow messages to go missing - email is
an example of that (ignoring spam filters that explicitly want messages to
go missing). It either gets delivered or it doesn't. So I'm not worried
about atomicity.
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ECDH in the payment protocol

[Mike Hearn]

>
> Ah, you're still misunderstanding my point: You can get atomicity in the
> worst-case where the communications medium fails *and* stealth payments
> that use up no extra space in the blockchain. This gives you the best of
> both worlds.


Sounds great! How does a lightweight client identify such transactions
without any markers?

Regardless, there are lots of other useful features that require BIP70 to
work well person to person, like messages, refund addresses, etc. So
extending it with ECDH makes sense in the end anyway no matter what.
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ECDH in the payment protocol

[Mike Hearn]

>
> The exact same way you're proposing: via the payment protocol.
>

Ah, I see, that's what I was missing. So rather than have an explicit
repeated field for nonces, have an algorithm for extracting randomness from
one of the scriptSigs. I guess that makes sense.
--
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Allow cross-site requests of payment requests

[Mike Hearn]

>
> Would it be a terrible idea to amend BIP 70 to suggest implementors
> include a "Access-Control-Allow-Origin: *" response header for their
> payment request responses? I don't think this opens up any useful attack
> vectors.
>

It sounds OK to me, although we should all sleep on it for a bit. The
reason this header exists is exactly because mobile code fetching random
web resources can result in surprising security holes.

For this to be useful, someone would have to actually want to fully
implement the payment protocol (with its own root cert store, ASN.1
parsing, RSA etc) in browser-sandboxed Javascript rather than just
providing a real app for people to download.

Is that really going to be popular, though? I think it's unclear.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ECDH in the payment protocol

[Mike Hearn]

On Tue, May 13, 2014 at 12:40 AM, Chris Pacia  wrote:

> Just a thought. Using the payment protocol for stealth would mean we
> would likely have to return to backing up wallets all the time would it
> not?
>

I think you are right. Awkward.

Wallets could auto-respend transactions to a plain (private) HD derived key
to make them findable again. But that gets us back to using block space
inefficiently.

Over time I think wallet backups will get more valuable anyway, as they
will start containing more and more essential data that isn't in the block
chain: receipts, messages, exchange rate records for tax purposes etc. But
being able to get access to your money with just the 12 words (+a date for
SPV wallets) is a pretty desirable safety feature.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Regtest Address Version Change Proposal

[Mike Hearn]

Yes, bitcoinj supports and uses regtest mode. It would also have to be
changed.

You didn't provide a rationale for this. What's the cost of having them be
the same?


On Tue, May 13, 2014 at 11:45 AM, Warren Togami Jr. wrote:

> Hi folks,
>
> I propose changing all of the address versions in -regtest mode to be
> unique so they are no longer identical to testnet.
>
> https://en.bitcoin.it/wiki/List_of_address_prefixes
> For example, regtest pubkey hash addresses could begin with r or R.
>
> We need to know if any existing tools would need to be modified to support
> this change to regtest.  Do existing tools outside of pull tester expect
> regtest to have testnet addresses?  If the quantity of tools that currently
> handle regtest is small then we can modify them to the new address versions.
>
> Warren Togami
>
>
>
> --
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform
> available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Regtest Address Version Change Proposal

[Mike Hearn]

And they can't just do

  NetworkParams.TESTNET = NetworkParams.REGTEST

at the start of a program that is connecting to regtest?

It's not like changing the address code is a huge problem or anything, but
it would disrupt a bunch of people and seems kind of annoying. Surely
there's a simpler way to work around this issue on their side? I mean their
code already has to know what network is *expected*, right, otherwise what
stops you accidentally trying to send coins cross chain?



On Tue, May 13, 2014 at 1:02 PM, Warren Togami Jr. wrote:

> bitcore guesses the network from the address version in several places in
> its code.  They don't want to change that.  Perhaps it wasn't the wisest
> approach for them to use.  I thought it might be simple to change the
> address version since its still relatively new and it isn't a real network.
>  Would it be too much work to change?
>
>
> On Tue, May 13, 2014 at 12:30 AM, Mike Hearn  wrote:
>
>> Yes, bitcoinj supports and uses regtest mode. It would also have to be
>> changed.
>>
>> You didn't provide a rationale for this. What's the cost of having them
>> be the same?
>>
>>
>> On Tue, May 13, 2014 at 11:45 AM, Warren Togami Jr. wrote:
>>
>>> Hi folks,
>>>
>>> I propose changing all of the address versions in -regtest mode to be
>>> unique so they are no longer identical to testnet.
>>>
>>> https://en.bitcoin.it/wiki/List_of_address_prefixes
>>> For example, regtest pubkey hash addresses could begin with r or R.
>>>
>>> We need to know if any existing tools would need to be modified to
>>> support this change to regtest.  Do existing tools outside of pull tester
>>> expect regtest to have testnet addresses?  If the quantity of tools that
>>> currently handle regtest is small then we can modify them to the new
>>> address versions.
>>>
>>> Warren Togami
>>>
>>>
>>>
>>> --
>>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
>>> Instantly run your Selenium tests across 300+ browser/OS combos.
>>> Get unparalleled scalability from the best Selenium testing platform
>>> available
>>> Simple to use. Nothing to install. Get started now for free."
>>> http://p.sf.net/sfu/SauceLabs
>>> ___
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>>
>>
>
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] DNS seeds unstable

[Mike Hearn]

My android wallet is working OK. Yes it isn't great when seeds have
temporary availability problems but things are still working.

There's a couple of pull reqs outstanding to include hard coded seed peers
and getaddr sourced IPs. Once those are finished and merged in there'll be
more backup paths.
On 16 May 2014 11:16, "Andreas Schildbach"  wrote:

> On 05/15/2014 07:48 PM, Gregory Maxwell wrote:
> > On Thu, May 15, 2014 at 4:50 AM, Andreas Schildbach
> >  wrote:
> >> I'm bringing this issue up again. The current Bitcoin DNS seed
> >> infrastructure is unstable. I assume this is because of we're using a
> >> custom DNS implementation which is not 100% compatible. There have been
> >> bugs in the past, like a case sensitive match for the domain name.
> >
> > If software is using the DNS seeds in a way where one or two being
> > unavailable is problematic, then the software may be using them
> > poorly.
> >
> > Generally DNS seeds should only be used as fast connectivity hints,
> > primarily for initial connectivity. Relying on them exclusively
> > increases isolation vulnerabilities (e.g. because the dns seed
> > operators or any ISP or network attacker on the path between you and
> > the seeds can replace the results with ones that isolate you on a
> > bogus network).
>
> I just used "nslookup", after seeing the issues in bitcoinj.
>
> I agree that clients should be robust regarding DNS lookups (and
> bitcoinj isn't), but still I think the first step needs to be
> maintaining a quality infrastructure.
>
>
>
>
> --
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform
> available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] About the small number of bitcoin nodes

[Mike Hearn]

>
> As an interested party not intimately familiar with the bitcoin codebase
> who also spent some time setting up a node a while ago, I would like to
> add one thing to the above list - network rate limiting.


The problem is that this is easier said than done. Bitcoin Core won't
notice a remote peer is working but slow and switch to a faster one, and
even if it did, it'd just mean throttling your connection would cause all
remote nodes to give up and hit the other unthrottled peers even more.

The best way to implement this is to do chain pruning, so your node will
still try and shovel bytes as fast as possible, but it's limited by how
many bytes it has to shovel. Remote nodes that are pulling down the block
chain can then switch between nodes depending on what they have available
in order to try and avoid hitting one node too hard. Nodes that were
offline for a while and just catching up would prefer nodes that have less
of the chain.

It'd be great if someone could experiment with this. The first step is
extending the p2p protocol so addr broadcasts and version messages include
how much of the chain (counting blocks from the head?) the peer is willing
to serve, and then updating the downloading code so it tries to be smarter
about peer selection. Unfortunately all this work is sort of backed up
waiting for sipa to finish merging in headers-first downloading.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Paper Currency

[Mike Hearn]

> Submitted with humility and some fear of getting laughed out of here...
>

Off topic aside, a bunch of us have lately started to think about the
atmosphere on this list and how to improve it. Nobody should have to fear
getting flamed or laughed at for proposing ideas, even if they turn out to
be silly ones. Gavin talked about this in his Bitcoin 2014 keynote and
asked for someone to solve the forum trolling problem.

I don't know if there are any silver bullets per se, but:

1) Please do keep ideas coming. It's easy to mute threads in any good mail
client for people who don't care. If anyone gets too aggressive, the rest
of us will remind them that this is unacceptable.

2) If you're willing to become a list moderator, please get in touch. Gavin
and I are looking for neutral people who are willing to keep up with this
list and help ensure the debate is civilised. Ideally moderation is not
necessary, but that's what we tried so far and we keep getting consistent
feedback from lots of people that it's not working.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] About the small number of bitcoin nodes

[Mike Hearn]

>
> Does this mean that you can currently actively hurt the network by
> adding a node with a very slow upstream / downstream?


Well, I guess "hurting" the network is perhaps a bit dramatic. There are
already lots of ways the download process can go wrong and take days. Using
the torrent is much faster. But my understanding is that this will slow
down the bootstrap process for some people yes. Remote peers won't try and
download in parallel or anything like that.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] About the small number of bitcoin nodes

[Mike Hearn]

>
> (sure - there are tricks to limit rates anyway, like the script in
> contrib/qos, but to have it generally available the block download
> needs to be more robust first)


One thing we could consider as a short term solution (if headers
first+parallel downloading will take a while, which seems plausible) is to
add a service bit that says "I have chain data and am willing to Bloom
filter it for you, but I won't serve full block data", and then just
exclude all of those from the chain download logic. It should not be a deep
change to the code headers first is impacting, and would allow home users
who may have no tolerance for block chain uploads at all to still take part
and offer useful services to the network.

I know Pieter likes the idea of an "archival node" service bit, or
something like that. I'd been thinking that the stored chain height value
would be better, but perhaps we need to divorce "I have CPU and can filter"
from "I have bandwidth and can serve" more vigorously.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] patents...

[Mike Hearn]

IMO this list is fine for discussing such topics.

Here are some thoughts. I had to deal with patents at Google (my name is on
a few, not my choice unfortunately). Many aspects of patent law are deeply
unintuitive, so here's the crash course as I was given it.



The first rule of patents is *you do not go looking for patents*. US law is
written in a really stupid way, such that if you knowingly infringe,
damages triple. Because America uses the patent office as a revenue source,
basically everything you can possibly imagine is covered by some ridiculous
patent so if you go looking you will always find applicable patents on
every idea and then you end up potentially much worse off.

Most companies (Google certainly included) have therefore banned their
staff from reading patents, thus ensuring that the whole point of them, the
sharing of knowledge, doesn't actually function! And it's much better I
think if we follow the same policy. So *please do not ever mention that
suchandsuch is patented on this list*! When it comes to patent law,
ignorance is bliss. Patents are written in a heavily obfuscated manner such
that actually trying to learn from them is hard work anyway.


One reason I wrote up the contracts stuff when I did is to get it out there
into the public domain, so people couldn't patent the basics of the Bitcoin
protocol. It'll be much better for everyone if new ideas are just put right
out into the public domain. *Please do not patent Bitcoin related research
you do*, even if you think it's for the best:

1) Defensive patenting doesn't work. The whole idea was mutually assured
destruction, you hit me I'll hit you type of logic, but the prevalence of
shell/troll companies killed off that idea. Plus it turns out that big
companies are quite willing to sue each other into oblivion anyway. Once a
patent exists, it'll be used as a weapon by someone eventually, and
attempting to "fight back" is probably not a workable strategy. Far better
to ensure the material is simply unpatentable by anyone.

2) Patenting with the intention to sue people using Bitcoin in the same
way: well, if you plan to do this, there's not much to talk about  you
won't make any friends this way.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] patents...

[Mike Hearn]

>
> Meh. The world is much bigger than the USA. Secondly that rule makes it
> difficult to educate people about why patents are as bad as they are.
>

You can easily find examples that are not relevant to Bitcoin if you want
to discuss the patent system in general.


> Feel free to continue censoring your own discussion within closed
> corporate environments. But to say keeping patent discussion off mailing
> lists is appropriate or wise when the tech news is full of such discussion
> is silly.


It is both appropriate and wise. Please keep discussion of Bitcoin-relevant
patents elsewhere.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] patents...

[Mike Hearn]

>
> Avoiding willfull infringement no longer requires paying off a
> patent attorney to get a freedom to operate review.  This isn't to say
> that reading patents is always productive


That case raised the bar a bit, but the core problem remains - if you learn
about a patent you definitely violate (and there is very likely to be at
least one and possibly many), via whatever means, then by continuing
business you become a wilful violator. Which makes sense: how could it be
any other way?

It still never makes sense to read patents. You can only lose.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Working on social contracts (was: Paper Currency)

[Mike Hearn]

Sorry. I will never agree to the concept of a relevant idea so dangerous it
cannot be discussed. That's medieval thinking. If you would like to create
a parallel development forum where people have to swear an oath not to
think bad thoughts, go right ahead and do so.

But I'm glad to see you correctly identified yourself as one of the people
causing problems on this list. Your vicious attacks are one of the reasons
we're now seeing threads that start with "I hope I don't get flamed or
laughed at for this idea but " which is totally unacceptable. I would
prefer you just unsubscribe, in the hope we get a second chance from some
of the potential developers we've lost.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Why are we bleeding nodes?

[Mike Hearn]

Yeah I'm expecting port 8333 to go away in China at some point. Actually I
was expecting that years ago and was kind of surprised that the suppression
was being done via banks. Guess the GFW operators were just slow to catch
up.
On 20 May 2014 10:16,  wrote:

> Recently China has updated its firewall blocking bitcoin sites and pools.
> Whether this is simple blacklist or more sophisticated packet targeting
> is uncertain, however this update did spefically target VPN handshakes.
>
>  *Sent:* Monday, April 07, 2014 at 1:07 PM
> *From:* Drak 
> *To:* "Mike Hearn" 
> *Cc:* "Bitcoin Dev" 
> *Subject:* Re: [Bitcoin-development] Why are we bleeding nodes?
>  For what it's worth, the number of nodes rose dramatically during the
> China bullrun (I recall 45k in China alone) and dropped as dramatically as
> the price after the first PBOC announcement designed to cool down bitcoin
> trading in China.
>
> On 7 April 2014 12:34, Mike Hearn  wrote:
>>
>> At the start of February we had 10,000 bitcoin nodes. Now we have 8,500
>> and still falling:
>>
>>http://getaddr.bitnodes.io/dashboard/chart/?days=60
>>
>> I know all the reasons why people *might* stop running a node (uses too
>> much disk space, bandwidth, lost interest etc). But does anyone have any
>> idea how we might get more insight into what's really going on? It'd be
>> convenient if the subVer contained the operating system, as then we could
>> tell if the bleed was mostly from desktops/laptops (Windows/Mac), which
>> would be expected, or from virtual servers (Linux), which would be more
>> concerning.
>>
>> When you set up a Tor node, you can add your email address to the config
>> file and the Tor project sends you emails from time to time about things
>> you should know about. If we did the same, we could have a little exit
>> survey: if your node disappears for long enough, we could email the
>> operator and ask why they stopped.
>>
>>
>> --
>> Put Bad Developers to Shame
>> Dominate Development with Jenkins Continuous Integration
>> Continuously Automate Build, Test & Deployment
>> Start a new project now. Try Jenkins in the cloud.
>> http://p.sf.net/sfu/13600_Cloudbees_APR
>> ___
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>  
> --
> Put Bad Developers to Shame Dominate Development with Jenkins Continuous
> Integration Continuously Automate Build, Test & Deployment Start a new
> project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees___Bitcoin-development
>  mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Cut-through propagation of blocks

[Mike Hearn]

>
> There
> is no need to have Bitcoin transport all using a single protocol, and
> we can get better robustness and feature velocity if there are a
> couple protocols in use (you could just run a block-transport-protocol
> daemon that connects to your local node via the classic protocol).


Although this is a somewhat appealing notion, would it really improve
feature velocity? I don't think the current p2p protocol is holding
anything back, and having to implement features twice in two protocols
would slow things down quite a bit.

Probably the lowest hanging fruit now is fixing the 100msec sleep and just
generally having tools to measure latency and queuing inside the code.
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Cut-through propagation of blocks

[Mike Hearn]

>
> it: I'd be much more likely to run an experimental gateway in another
> process on a node than experimental p2p code inside my production
> bitcoinds themselves.
>

Yes, it's certainly better to do that during the development phase. However
if it does turn out to be good and valuable then it'd eventually need to be
integrated or rewritten into Core anyway, lest we accidentally increase the
setup cost of running a node and end up with a two-tier network. And if the
code will eventually want to be merged into Core anyway, it might as well
be implemented into it directly, perhaps behind a switch that can disable
those codepaths if something goes wrong.

So I think the tradeoffs here are rather complicated and subtle.
--
The best possible search technologies are now affordable for all companies.
Download your FREE open source Enterprise Search Engine today!
Our experts will assist you in its installation for $59/mo, no commitment.
Test it for FREE on our Cloud platform anytime!
http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] # error "Bitcoin cannot be compiled without assertions." <<<

[Mike Hearn]

Hi Ron,

FYI your mail is being spamfoldered due to Yahoo's DMARC policy and the
brokenness of the SF.net mailing list software. I would not expect to get
replies reliably whilst this is the case. I think we should move away from
SF.net for hosting mailing lists personally, because it's this list that's
at fault not Yahoo, but until then you may wish to send to the list with a
different email address.

As to your question,

assert() should have *no* side effects, that is the problem.
>
> See
>
> http://books.google.com/books?id=L5ZbzVnpkXAC&pg=PA72&lpg=PA72&dq=Gotcha+%2328+Side+Effects&source=bl&ots=Rn15TlPmje&sig=tymHqta0aSANwaM2GaXC-1Di_tk&hl=en&sa=X&ei=uVKNU47fCcvTsAT6goHIBA&ved=0CCAQ6AEwAA#v=onepage&q=Gotcha%20%2328%20Side%20Effects&f=false
>
> a great book, BTW.  Everyone who thinks they know what they are doing when
> they write C++ should read this book!  They will realize that they don't
> know Jack [image: Roll Eyes]
>
> Why weren't these and all the other examples of amateur, i.e.,
> non-professional, software fixed way back in version 0.3.0 in 2010, before
> any more releases were done?  And why were these and other sub-standard
> coding practices continued and expanded in later releases, right up until
> the present?
>

Back in 2010 most code was still being written by Satoshi so perhaps you
should ask him. Regardless, it's very common for professional codebases to
require assertions be enabled. For example the entire Google C++ codebase
uses always-on assertions that have side effects liberally: it's convenient
and safe, when you have the guarantee the code will always run, and the
performance benefits of compiling out assertions are usually non-existent.

So for this reason I think Bitcoin Core currently will fail to build if
assertions are disabled, and that seems OK to me.
--
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] # error "Bitcoin cannot be compiled without assertions." <<<

[Mike Hearn]

>
> As a matter of procedure we do not use assertions with side effects— the
> codebase did at one point, but have cleaned them up.  In an abundance of
> caution we also made it refuse to compile without assertions enabled: A
> decision who's wisdom was clearly demonstrated when not long after, some
> additional side-effect having assert was contributed. In the real world
> errors happen here and there, and making robust software involves defense
> in depth.
>

I think this class of errors could be removed entirely by just saying it's
OK for assertions to have side effects and requiring them to be enabled, as
is currently done.

The glog library:

http://google-glog.googlecode.com/svn/trunk/doc/glog.html

provides CHECK macros that print stack traces when they fail. Using them
would also be good.
--
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] # error "Bitcoin cannot be compiled without assertions." <<<

[Mike Hearn]

Currently expensive checks are guarded with command line flags. It'd be
nice if there could be one unified command line flag -expensivechecks that
subsumes -checkmempool and so on.


On Wed, Jun 4, 2014 at 6:42 PM, Jannis Froese  wrote:

>  There are reasons to have assertions enabled by default in software like
> Bitcoin Core, where incorrect behaviour can be costly. But this comes at a
> prize: our assertions have to satisfy certain performance requirements.
> It's no longer possible to do expensive, redundant checks in performance
> critical code, which is one of the main advantages of asserts. Imho,
> asserts are not intended for small range checks etc, but are meant for
> checks that a hash hasn't changed, that a tree structure is still a tree,
> that data is still sorted, or that data structures are in sync.
>
> I think most concerns about the current use of asserts would be resolved
> if the currently used asserts would be changed to a nicer definition which
> is independent of NDEBUG, and a second class of debugging asserts would be
> introduced, which is exclusively for expensive, redundant checks and is
> disabled by NDEBUG.
>
>
>
> Am 2014-06-04 12:15, schrieb Gregory Maxwell:
>
> On Wed, Jun 4, 2014 at 2:51 AM, Mike Hearn  wrote:
>
>> Hi Ron,
>>
>>  FYI your mail is being spamfoldered due to Yahoo's DMARC policy and the
>> brokenness of the SF.net mailing list software. I would not expect to get
>> replies reliably whilst this is the case. I think we should move away from
>> SF.net for hosting mailing lists personally, because it's this list that's
>> at fault not Yahoo, but until then you may wish to send to the list with a
>> different email address.
>>
>>  As to your question,
>>
>> assert() should have *no* side effects, that is the problem.
>>>
>>> See
>>>
>>> http://books.google.com/books?id=L5ZbzVnpkXAC&pg=PA72&lpg=PA72&dq=Gotcha+%2328+Side+Effects&source=bl&ots=Rn15TlPmje&sig=tymHqta0aSANwaM2GaXC-1Di_tk&hl=en&sa=X&ei=uVKNU47fCcvTsAT6goHIBA&ved=0CCAQ6AEwAA#v=onepage&q=Gotcha%20%2328%20Side%20Effects&f=false
>>>
>>> a great book, BTW.  Everyone who thinks they know what they are doing
>>> when they write C++ should read this book!  They will realize that they
>>> don't know Jack [image: Roll Eyes]
>>>
>>>  Why weren't these and all the other examples of amateur, i.e.,
>>> non-professional, software fixed way back in version 0.3.0 in 2010, before
>>> any more releases were done?  And why were these and other sub-standard
>>> coding practices continued and expanded in later releases, right up until
>>> the present?
>>>
>>
>>  Back in 2010 most code was still being written by Satoshi so perhaps
>> you should ask him. Regardless, it's very common for professional codebases
>> to require assertions be enabled. For example the entire Google C++
>> codebase uses always-on assertions that have side effects liberally: it's
>> convenient and safe, when you have the guarantee the code will always run,
>> and the performance benefits of compiling out assertions are usually
>> non-existent.
>>
>>  So for this reason I think Bitcoin Core currently will fail to build if
>> assertions are disabled, and that seems OK to me.
>>
>
>  As a matter of procedure we do not use assertions with side effects— the
> codebase did at one point, but have cleaned them up.  In an abundance of
> caution we also made it refuse to compile without assertions enabled: A
> decision who's wisdom was clearly demonstrated when not long after, some
> additional side-effect having assert was contributed. In the real world
> errors happen here and there, and making robust software involves defense
> in depth.
>
>  Considering the normal criticality of the software it should always be
> with the assertions. Without them is an untested configuration.  It would
> probably be superior to use our own assertion macros (for one, they can
> give some better reporting…) that don't have the baggage ordinary
> assertions have, but as a the codebase is a production thing, making larger
> changes all at once to satisfy aesthetics would be unwise... simply
> refusing to compile in that untested, unsupported configuration is prudent,
> for the time being.
>
>
>
> --
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed lea

Re: [Bitcoin-development] Future Feature Proposal - getgist

[Mike Hearn]

Why do you want to optimise this? getheaders is used by SPV clients not
full nodes. SPV clients like bitcoinj can and do simply ship with gist
files in them, then getheaders from the last "checkpoint"   (I wish I
hadn't reused terminology like that but this is what bitcoinj calls them).

In practice when I look at performance issues with current SPV clients,
getheaders speed is not on my radar.
--
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bloom bait

[Mike Hearn]

You can send different bloom filters to different peers too, so I'm not
sure why you're listing subsetting as a unique advantage of prefix filters.

The main advantage of prefix filters seems to be faster lookups if the node
is calculating a sorted index for each block, and the utxo commitment
stuff, both of those would be cool but involve imposing extra costs on
nodes. We lack models that let us understand the tradeoffs involved in
various indexing schemes, I feel.
--
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bloom bait

[Mike Hearn]

>
> As I explained in the email you're replying to and didn't quote, bloom
> filters has O(n) cost per query, so sending different bloom filters to
> different peers for privacy reasons costs the network significant disk
> IO resources. If I were to actually implement it it'd look like a DoS
> attack on the network.
>

DoS attack? Nice try.

Performance is subtle, disk iops especially so. I suspect you'd find - if
you implemented it - that for the kinds of loads Bitcoin is processing both
today and tomorrow prefix filtering either doesn't save any disk seeks or
actively makes it worse.

Consider a client that is syncing the last 24 hours of chain. bitcoind
pre-allocates space for blocks in large chunks, so most blocks are laid out
sequentially on disk. Almost all the cost of a disk read is rotational
latency. Once the head is in place data can be read in very fast and modern
kernels will attempt to adaptively read ahead in order to exploit this,
especially if a program seems to be working through a disk file
sequentially. The work of Bloom filtering parts of the chain for this
client boils down to a handful of disk seeks at best and the rest of the
work is all CPU/memory bound as the block is parsed into objects and tested
against the filter. A smarter filtering implementation than ours could do
SAX-style parsing of the block and avoid the overhead of turning it all
into objects.

Now consider a prefix filtering implementation. You need to calculate a
sorted list of all the data elements and tx hashes in the block, that maps
to the location in the block where the tx data can be found. These
per-block indexes take up extra disk space and, realistically, would likely
be implemented using LevelDB as that's a tool which is designed for
creating and using these kinds of tables, so then you're both loading the
block data itself (blocks are sized about right currently to always fit in
the default kernel readahead window) AND also seeking through the indexes,
and building them too. A smart implementation might try and pack the index
next to each block so it's possible to load both at once with a single
seek, but that would probably be more work, as it'd force building of the
index to be synchronous with saving the block to disk thus slowing down
block relay. In contrast a LevelDB based index would do the bulk of the
index-building work on a separate core.

At *some* block size and client load the additional data storage and
increased pressure on the page cache would probably make it worthwhile. But
I find it unlikely to be true at current traffic levels, or double or
triple today's levels. So I'd rather we spend our very limited collective
time on finding ways to increase usage rather than worrying about resources
which are not presently scarce.

(as an aside, some of the above analysis would be invalidated if most nodes
end up running on SSDs, but I doubt most are. It'd be neat to export
storage tech via some kind of stats message - LevelDB is designed for HDDs
not SSDs so at some point a new storage subsystem might make sense if the
network switched over).
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bloom bait

[Mike Hearn]

>
>  A NODE_BLOOM service bit is a very reasonable
> and simple way to do exactly that, and is defacto what implementations
> that don't support bloom filters do anyway.
>

BTW, I find it curious that any nodes have code to disconnect peers that
send Bloom filters. It shouldn't be necessary. Bitcoinj is the only large
scale user of filtering and it will disconnect itself if a peer advertises
support for a version lower than 7. If a node advertises support for
this version or higher then it is supposed to implement BIP37.

It sounds like some node authors decided to advertise support for a
protocol version they didn't bother implementing, which would be a bug.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bloom bait

[Mike Hearn]

>
> Is this any different from my bloom filter IO attack code? Nope.
>

It's obviously different; a thin client trying to obtain more privacy is
not attempting to deny service to anyone. You can't simply state that a
feature which uses resources for a legitimate reason is a DoS attack,
that's a spurious definition that would reclassify innocuous things like
web browser prefetching as DoS attacks too.

With respect to the work you're proposing, trying to retroactively make
Bloom filtering an optional part of the protocol is just wasting people's
time at this point:

   - There is no evidence there's an actual problem today.
   - There is no evidence there will be a problem any time soon, given the
   meagre levels of traffic growth we have.
   - It involves a complicated rollout strategy that would create work for
   many people.
   - Gavin, Wladimir and myself have all concluded it's not worth the cost.
   - The only justification you have provided is that two node
   implementations hardly anyone uses can't be bothered to implement Bloom
   filtering, but want to advertise support in their ver message anyway. They
   can simply lower the number they advertise in their ver message.

That said, if you want to implement better support for archival nodes then
that'd be great. The way to do it would be either to implement block ranges
in the addr announcements as has been discussed many times before, or
perhaps introduce a new protocol optimised for serving the chain. Nodes
that are CPU limited won't want to take part in the rest of the P2P
protocol anyway, it's not just Bloom filtering that uses CPU time but also
block and tx relay.

But until you have done these things, please stop attempting to reclassify
any feature you can imagine a more efficient version of as an "attack".
It's just silly.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Incentivizing the running of full nodes

[Mike Hearn]

Hi Odinn,

I think trying to incentivise nodes with money is tricky: it makes
intuitive sense but right now the market is flooded with supply relative to
demand. Yes, we worry about the falling number of nodes, but that's for
reasons that aren't really economic: the more nodes we have, the bigger and
more grassroots the project seems to the outside world, plus the cheaper it
gets for everyone as the biggest cost (chain upload bandwidth) is spread
over multiple people.

Also there's research showing that when you have people volunteering,
introducing money can ruin the motivation of the volunteers, so the
transition to a pay-for-node-services world could be quite painful and
difficult.

Right now rather than microdonations to all nodes, IMO the lowest hanging
fruit is to move chain upload onto specialised "archival nodes" which can
potentially charge for their services. I prototyped this here

https://github.com/mikehearn/PayFile

but never finished it.


On Mon, Jun 16, 2014 at 10:12 AM, Odinn Cyberguerrilla <
odinn.cyberguerri...@riseup.net> wrote:

> I have been noticing for some time the problem which Mike H. identified as
> how we are bleeding nodes ~ losing nodes over time.
>
> This link was referenced in the coindesk article of May 9, 2014:
>
>
> http://sourceforge.net/p/bitcoin/mailman/bitcoin-development/thread/CANEZrP2rgiQHpekEpFviJ22QsiV%2Bs-F2pqosaZOA5WrRtJx5pg%40mail.gmail.com/#msg32196023
>
> (coindesk article for reference:
> http://www.coindesk.com/bitcoin-nodes-need/)
>
> The proposed solution is noted here as a portion of an issue at:
>  https://github.com/bitcoin/bitcoin/issues/4079
>
> Essentially that part which has to do with helping reduce
> the loss of nodes is as follows:
>
> "a feature similar to that suggested by @gmaxwell that would process small
> change and tiny txouts to user specified donation targets, in an
> incentivized process. Those running full nodes (Bitcoin Core all the
> time), processing their change and txouts through Core, would be provided
> incentives in the form of a 'decentralizing lottery' such that all
> participants who are running nodes and donating no matter how infrequently
> (and no matter who they donate to) will be entered in the 'decentralizing
> lottery,' the 'award amounts' (which would be distinct from 'block
> rewards' for any mining) would vary from small to large bitcoin amounts
> depending on how many participants are involved in the donations process.
> This would help incentivize individuals to run full nodes as well as
> encouraging giving and microdonations. The option could be expressed in
> the transactions area to contribute to help bitcoin core development for
> those that are setting up change and txouts for donations, regarding the
> microdonation portion (which has also has been expressed conceptually at
> abis.io"
>
> This addresses the issue of how to incentivize more
> interested individuals to run full nodes (Bitcoin Core).  The lottery
> concept (which would be applicable to anyone running the full node
> regardless of whether or not they are mining) is attractive from the point
> of view that it will complement the block reward concept already in place
> which serves those who mine, but more attractive to the individual who
> doesn't feel the urge to mine, but would like to have the chance of being
> compensated for the effort they put into the system.
>
> I hope that this leads to additional development discussion on these
> concepts regarding incentivizing giving. This may also involve a process
> BIP.  I look forward to your remarks.
>
> Respect,
>
> Odinn
>
>
>
> --
> HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
> Find What Matters Most in Your Big Data with HPCC Systems
> Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
> Leverages Graph Analysis for Fast Processing & Easy Data Exploration
> http://p.sf.net/sfu/hpccsystems
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

Looking good! I think this is much better than the original draft. Agree
with Andreas that supports_instant is simply equal to
(supported_instant_providers.size() > 1) which makes it redundant.

Daniel is right that putting every possible provider in the Payment message
might not scale in a world where there are huge numbers of
instant-confirmation providers, but I'm hoping that we never have to scale
to that size, because if we did that'd rather imply that Bitcoin has become
useless for in-person payments without trusted third parties and avoiding
that is rather the whole point of the project :) So I'm OK with some
theoretical lack of scalability for now.

A more scalable approach would be for the user to send the name and
signature of their "instant provider" every time and the merchant just
chooses whether to ignore it or not, but as Lawrence points out, this is
incompatible with the provider charging extra fees for "instantness". But
should we care? I'm trying to imagine what the purchasing experience is
like with optional paid-for third party anti-double-spend protection.
Ultimately it's the merchant who cares about this, not me, so why would I
ever pay? It makes no sense for me to pay for double spend protection for
the merchant: after all, I'm honest. This is why it doesn't make sense for
me to pay miners fees either, it's the *receiver* who cares about
confirmations, not the sender.

So I wonder if a smarter design is that the user always submits the details
of their instantness provider and we just don't allow for optional
selection of instantness. I'm not sure that works, UX wise, so is having a
less scalable design to support it worthwhile?
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

Oh yes the other thing we need to decide is how to extend BIP70.

Protocol buffers have an extend keyword. But I'm not sure it's what we
really want. IMHO a simpler solution is to have a single "living" version
of the protobuf (where? in a new git repo?) which has all the fields
defined by all the accepted BIPs in a single place. Otherwise the build
process and so on for wallet implementors would get unnecessarily
complicated for no real reason. Also if you wanted to pick a new number for
fields, you'd end up having to read *all* BIPs to figure out what is
available. Simpler seems better.

If we do that then I suggest just using field number 8 or 9 or whatever
rather than 1000. IMHO fields 1000+ should be for private extensions that
are unlikely to collide with other users.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> I don't see more than a bunch of accepted payment methods anywhere I
> ever been in my life, I don't see merchants trusting more than a handful of
> third parties.
>

Sure. I buy this. Although the credit card market is a great example of
what we *don't* want: a stagnant duopoly of trusted third parties who
rampantly abuse their position. So I'd hope we see either (a) nobody really
caring about this BIP because Bitcoin gives good enough double spend
protection or (b) lots of anti-double-spend providers (hundreds seems fine).


> Ultimately you care because the alternative is to wait.
>

No, I will never wait. Neither me nor the merchant wants to me to be
pointlessly hanging around for an hour. The alternative is to pay by credit
card or cash. Outside of experiments there is no such thing as a shop that
only accepts only Bitcoin and will require me to wait for a block because I
didn't use a TTP to guarantee anti-double spends.

So this seems like a fundamental problem to me: having the ability to say,
"here is a proof I won't double spend" is fine, but it doesn't achieve
anything if the merchant would have sold me the goods in return for a
normal Bitcoin tx anyway, which in practice they always will because this
system starts out from zero users and would have to work upwards. I
*especially* will never use this system if I have to pay for it - I'd much
rather just put my money into a wallet that can't generate these proofs and
pay the sticker price.

Maybe what this BIP needs is an extra field that lets the merchant say, I
will give you a discount of X satoshis if you give me a no-double-spends
proof. In other words invert it: the sticker price is what normal Bitcoin
transactions cost, and then your wallet shows you the regular BIP70 price
minus the discount plus the third parties fee as what you finally pay. I
compare it to the sticker price the merchant is asking and if it's lower
I'm happy, and if it's higher my wallet would automatically avoid using the
TTP because I don't want to ever pay more, only less.

The market would then figure out if the fees the TTP charges are worth the
lower losses due to double spending fraud.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> Mike Hearn, why don't we just have all nodes report attempted double
> spends through the node network.
>

Please see https://github.com/bitcoin/bitcoin/pull/3883 which implements
this exact scheme. It can solve some kinds of double spends (probably), but
others - like ones done by corrupt miners (see bitundo) - can't be solved
this way.

Lawrence's motivation for this BIP is essentially to act as a backup in
case the Bitcoin native double spending protections end up being too weak
to be useful. It reintroduces a notion of centralised trust as a layer on
top of the Bitcoin protocol, but only for cases where the seller/recipient
feels it'd be useful. In this way it gives us slack: if someone is able to
reliably double spend and the merchants losses due to payment fraud go up,
we can fall back to TTPs for a while until someone finds a solution for
Bitcoin, or we just give up on the Bitcoin experiment, but hey - at least
we now have a better intermediary protocol than SWIFT :-)

In practice of course this is something payment processors like Bitpay and
Coinbase will think about. Individual cafes etc who are just using mobile
wallets won't be able to deal with this complexity: if we can't make native
Bitcoin work well enough there, we're most likely to just lose that market
or watch it become entirely centralised around a handful of payment
processing companies.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> Come to think of it, is the payment protocol really the place to put this
> instant provider signature
>

Yes it's the right place. The original attempt at this concept was in fact
called *green addresses* and the idea was you could identify a spend from a
trusted wallet by checking which keys were being used to sign. But the
problem is, lack of privacy. Everyone can see what wallet provider you use.

Also it'd be inefficient to have in the chain. There's no reason for the
extra signatures to be there: double spend risk is something only the
recipient cares about.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> I read the comments on the PR. I mean no disrespect but this patch can't
> prevent double spends minutes apart and a solution is as good as it's
> weakest link.
>

Actually Tom is running a page where he shows double spends detected by his
node or relayed by mine (there are only two nodes in this little detection
network currently), and it does show double spends that occur seconds,
minutes or even days apart.

Regardless, whether that approach helps or not is off topic for this
thread. Let's all hope it does and discuss the details in some other
thread, or on the pull request.


> instant confirmation between exchanges can create huge arbitrage
> opportunities and as such
> liquidity.
>

Yes indeed, if you want to do high frequency trading then every millisecond
counts and you probably don't want to rely on watching transactions
propagate across the block chain. For inter-exchange traffic this BIP would
probably be useful. I've been talking about the consumer case.


> What do you expect for e-commerce and escrow to happen? Don't you think the
> market will naturally converge to a handful of hubs that will helps with
> refunds and things like that?


No, I expect there to be many kinds of trades where dispute mediation is
unnecessary, e.g. when I buy a drink at Starbucks or a burger at McDonalds
the chances of me wanting to charge it back is basically zero. Same for
sending between people who know each other, large corporate transactions
where the threat of a lawsuit is more useful than mediation, etc.

But for transactions where third parties are needed for dispute mediation,
yes, I'd expect there to be a handful of well known trusted names for the
majority of such transactions, and then a long tail of specialists who only
mediate e.g. purchases of rare Aztec artifacts or other things where a
generic company might be easily fooled.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> I only meant that double spends minutes apart are possible and that by then
> the sole use of a monitor is too late even if it will tell you.
>

As long as miners stick to Satoshi's first seen rule, which is the default,
it's useful:

https://bitcointalk.org/index.php?topic=423.msg3819#msg3819

(this is the famous "snack machine" thread from 2010)

If they decide to change to something like highest-fee-always-wins, then
they (again) centralise things by forcing all instant transactions to pay
GreenAddress and its competitors money - much though I like your product
Lawrence, let's hope they don't collectively lemming us all off a cliff by
doing that ;)
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Incentivizing the running of full nodes

[Mike Hearn]

>
> This is a cool idea, but doesn't it generate some perverse incentives? If
> I'm running a full node and I want to pay CheapAir for some plane tickets,
> I'll want to pay in the greatest number of individual transactions possible


Peers can calculate rewards based on number of inputs or total kb used:
you're paying for kilobytes with either coin age or fees no matter what. So
I think in practice it's not a big deal.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

I think many of us feel it'd be better if this kind of thing were not
needed at all, however, the best way to ensure it doesn't end up being used
is to write code, not to try and block alternative approaches. If Bitcoin
is robust the market should sort it out. If it's robust for some
transactions and not others, that makes for a fun project for a future
generation of hackers to sort out.

OK, enough philosophy - let's try and keep this thread just for discussion
of the BIP itself from now on. If you'd like to continue debating the
Future of Bitcoin please change the subject line and break it out into a
new thread.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

On Mon, Jun 16, 2014 at 10:29 PM, Daniel Rice 
wrote:

> I'm trying to think through how to encourage the maximum number of instant
> signature providers and avoid the VISA monopoly. Ideal case would be that
> people can even be their own instant provider.
>

A provider does not have to be an interactive third party. One reason I
suggested using X.509 is so secure hardware devices like the TREZOR could
also be instant providers. The hardware would be tamperproof and assert
using a secret key embedded in it that the tx came from a genuine,
unflashed TREZOR. The the server can know the device won't double spend.

In this way you have decentralised anti-double spending. Of course, it's an
old solution. MintChip sort of worked a bit like this.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

On Mon, Jun 16, 2014 at 10:37 PM, Daniel Rice 
wrote:

> True, that would work, but still how are you going to bootstrap the trust?
> TREZOR is well known, but in a future where there could be 100 different
> companies trying to release a similar product to TREZOR it seems like one
> company could corner the market by being the only one that is an accepted
> instant provider at most vendors
>

It's no different to the CA problem. People can only mentally handle a few
trust anchors, so for SSL it goes:

   1 User -> 2-3 browser makers -> 100's of CAs -> millions of websites

The trust starts out narrowly funnelled and grows outwards as things get
outsourced.

For this it'd go

   1 merchant -> 4-5 payment processing engines -> dozens of hardware
manufacturers -> hundreds of thousands of devices
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

Yes that's true. Though it's off topic, check out
http://www.certificate-transparency.org/   it's a project to force CA's
to publish all certs they make publicly.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

Please, let's talk about other anti-double spend things on a separate
thread.

On Tue, Jun 17, 2014 at 5:58 PM, Isidor Zeuner  wrote:

> What prevents the following steps from happening:
>

I linked to Satoshi's post on this earlier, he explains why it works there,
assuming people follow the original protocol rules.

Your analysis holds as long as network abandons the original Bitcoin
design. Obviously, we hope people won't do that. If everyone decides not to
do things how Satoshi laid out then things will break, although whether we
have a failure of "Bitcoin" at that point is debatable.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> - allowing multiple signatures ?


I'm not sure this is actually important or useful; trusting someone not to
double spend is a pretty binary thing. I'm not sure saying "you need to get
three independent parties to sign off on this" is worth the hassle,
especially because the first signature is obvious (your risk analysis
provider or hardware) but the second and third are . who? Special
purpose services you have to sign up for? Seems like a hassle.

But it's up to you.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] instant confirmation via payment protocol backwards compatible proto buffer extension

[Mike Hearn]

>
> I think that's true if you assume that the instant provider list is based
> on a by hand created list of accepted instant providers. That's how VISA
> works now and that's why I was asking for an approach where the
> trusted_instant_providers list is scalable because that seems very
> dangerous.
>

Supporting it in the protocol is easy. Building such a thing: that's hard.
Decentralised automated reputation systems are complex and subtle.

I don't feel strongly about whether the field should be "optional" or
"repeated", 100% of implementations in the forseeable future would just
look at the first item and ignore the rest. But if later someone did crack
this problem it would lead to a simple upgrade path. So perhaps you're
right and the protobuf should allow multiple signatures. It means a new
sub-message to wrap the pki_type, pki_data and signature fields into one,
and then making that repeated.

Up to Lawrence.
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BlockPow: A Practical Proposal to prevent mining pools AND reduce payoff variance:

[Mike Hearn]

>
> The issue is centralized transaction selection policies, which is
> entirely orthogonal. And the solution already exists: getblocktemplate.


My (fresh!) understanding is that the reason we don't see people using
getblocktemplate to decentralise pools is because libblkmaker and other
implementations don't actually support connecting your own node to the
miners and choosing your own blocks, even though the protocol does.

I've written up a blog post that I hope will go out on the Foundation blog
soon with some low hanging fruity ideas for miner decentralisation.

Sergio, I'd love to give you intelligent feedback but unfortunately reading
it made my brain explode :) Sorry!
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

>
> So first bitcoind will support SPV mode then we separate the wallet?
> Are the core and the wallet share any code (say, the p2p messages via
> a sub-repo or something)?
>

bitcoind already supports SPV mode, that's how bitcoinj clients work.
However the current wallet code doesn't use it, it integrates directly with
the full mode main loop and doesn't talk P2P internally. Which is the fine
and obvious way to implement the wallet feature. I'm not totally convinced
it should become an SPV wallet given the complexity of doing that. But if
you did want to separate the wallet code from the full node then that'd be
the way to do it.

The question is; what does this buy us, and is it worth the potentially
huge amount of time it could take? My gut feeling is we have bigger fish to
fry. There's plenty of work to do just on the core consensus code, making
Bitcoin Core into a competitive wallet as well would be an additional
burden.

However I may be quite biased, as I am the maintainer of what is primarily
a wallet library :)


> I'm sorry, but I still don't know what Electrum has to do with all this.
>

People use Electrum as shorthand to mean "something a bit like the P2P
network, but with trusted remote servers which build additional databases
and thus support additional commands".
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

>
> From my experience the main thing people are missing with BitcoinJ is
> a quick and easy way to set up a wallet as a daemon, to use the
> functionality from non-java through RPC.


Yes. I'd love to have a mostly Core compatible JSON-RPC frontend. Most of
my current users are happy using it as a library though. A lot of popular
languages can run directly on the JVM these days. The big ones we miss are
C++ and PHP, I think. But you can use JavaScript, Python 2.7, Lisp, Ruby,
along with other less well known ones.

The other good reason to have JSON-RPC support would be to reuse the Core
regression tests.

Anyway, this is off topic :)
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

>
> Although Pieter and I disagree with regard to issue #4351, we agree on
> wanting to keep (or at least making) bitcoind as lean as possible.
> Maintaining extra indices for others doesn't fit in there - that's
> also why the address index patch was not merged. An 'index node' could
> be a different animal.


We definitely want to head in the direction of allowing a p2p node to be as
useful as possible within its resource constraints and optional advertising
of new (expensive) indexes is the way to go.

Sometimes I wonder if we should have an RPC or new socket based method
where additional programs could run along side Bitcoin Core and opt to
handle a subset of p2p commands. But then I think, that seems like a lot of
complexity for people who just want to help out the system, which I guess
is the bulk of our network now. Keeping their lives simple should have a
high priority. So a single unified program that just figures it out
automatically rather than expecting users to assemble a bag of parts seems
a goal worth striving for.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Proposed BIP 70 extension

[Mike Hearn]

Coinbase have started allowing merchants to set discounts for purchasing
with Bitcoin. Seeing an individual discount is not very motivating as they
tend to be small. Seeing them stack up over time can be more motivating
because it feels like free money. Many businesses exploit this effect with
loyalty points, etc. Bitcoin should do this too - show the user how much
they're saving by using Bitcoin instead of credit cards.

I suggested to Charlie Lee (who pushed this through at Coinbase) and
Stephen Pair the following minor BIP 70 extension:


message PaymentDetails {
// Size in satoshis of any discount provided by the merchant ONLY
// because the user chose to pay using Bitcoin or other similar
// digital currency. Other kinds of discounts, loyalty bonuses and
// so on should not be recorded here, rather they could be mentioned
// in the memo field. This field exists so wallets can show the user
// a running total of how much money they have saved by avoiding
// credit cards and bank payments; the goal is to encourage people to
// use Bitcoin. Putting other kinds of discounts here would make the
// running total calculated meaningless; so don't do it!
optional uint64 currency_usage_discount_size = 8;
}

Wallets would then be able to persist this data to disk and compete on cool
visualisations for how much money you saved over time.

We haven't formalised how to extend BIP 70 yet, that's my fault. We should
do that. In the meantime, what do people think of this proposal?
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Proposed BIP 70 extension

[Mike Hearn]

>
> It also seems like it would be subject to instant inflation, as it's
> unprovable


The user knows the price that is on the website or menu, they know the
price they actually paid ... if the numbers don't add up that would seem to
be pretty easily detectable. But sure it's only for marketing.  I think the
comment makes it clear it's just for fun.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Proposed BIP 70 extension

[Mike Hearn]

>
> Sounds like marketing bullshit to me. It does not have even statistical
> meaning; well, you can "save" a lot of satoshis, but nobody tell you that
> the merchant cut you on BTC/USD exchange rate in tens of %.
>

Your own wallet can look up the exchange rate and compare it to what you're
getting (and in fact, wallets do!).

Besides, assuming the customer is *always* being scammed seems extreme.
There are plenty of merchants that genuinely care about their reputation
and genuinely want people to pay with Bitcoin so they can avoid card fees.


> Payment protocol should not contain these fictional data
>

Well, I think the protocol should contain whatever is useful.

I'll probably draft a BIP for this next week or so.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Proposed BIP 70 extension

[Mike Hearn]

>
> I think it should be made more clear what's the reference price for the
> discount.
>

That might be useful for merchants that already provide a series of
price-differentiated payment methods, yes. Will think about it.


> Also, currently PR are created by the payment processors afaik. How can
> they know what other payment option the merchant provides and what's the
> conditions?
>

Currently Coinbase let merchants specify the size of their discount (I
guess in percentage terms, I should ask). So the merchants tell the payment
processor. I don't think this is a worry at the moment.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Proposed BIP 70 extension

[Mike Hearn]

>
> I agree. It would be even sillier to start specifying container formats
> for random one-off "that would be kind of nice, I guess" features.
>

No, it'd be sensible.

Here's a list I drew up a long time ago of features I imagined adding to
the payment protocol:

https://bitcointalk.org/index.php?topic=270055.msg2890147#msg2890147

The protocol is there to contain features! There is zero benefit to
slavishly following some religious notion of purity or minimalism here. The
shared resource in question is just varint encoded integers. So, we should
be guided by what will help our users and what will help adoption.

Anyway, Gavin asked me to start handling more BIP 70 stuff a few weeks ago.
I want to use something simple to set up the extensions process more
formally. IMO we need a "living document" version of the payment protocol
with all the different extensions out there folded into it, to simplify
programming tasks and ensure field numbers don't collide.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bill Request Message - (another) Proposed BIP 70 extension

[Mike Hearn]

I'm not convinced this inversion is really a problem, but as this is quite
a complex proposal (e.g. new barcode types) the best way to move it forward
at this stage is to implement it in some existing wallets.

FWIW NFC is a lot more common than you might think. For the drive-thru case
you could also consider using wifi hotspots with a special name or
Bluetooth LE tags. So I suspect before trying to write a specification it'd
be better to explore different technologies and see what works best in
practice.



On Tue, Jun 24, 2014 at 9:44 PM, Paul Goldstein  wrote:

> Here's an idea for a BIP 70 extension to let wallets be scanned by
> merchant bar code readers to start off a payment request flow instead of
> the other way around (wallet scanning the merchant QR). Useful for brick
> and mortar merchants and mobile wallet apps.
>
>
> Motivation:
> A mechanism is needed for mobile wallets to request a bill, so that a
> payment protocol flow can be initiated. Current mechanisms for initiating
> BIP70 payment flows generally require wallets to either scan a merchant
> barcode (not optimal, see below), or click on a specially formatted url
> link (only suitable while browsing and purchasing on a merchant website).
>
> Successful non-bitcoin mobile wallet apps to date (a popular coffee shop
> app comes to mind) allow for the wallet app to be scanned by the merchant
> and not the other way around (as is commonly done in bitcoin wallets
> today). For broad bitcoin adoption we need a mechanism for wallets to be
> scanned by merchant bar code readers at brick and mortar shops. This will
> also greatly ease checkouts at drive throughs and allows merchants to
> leverage existing hardware (barcode readers).
>
> Other technologies like NFC may obviate the need for this extension,
> however, those technologies remain somewhat uncommon and may not be
> suitable for smaller wearables hitting the market.
>
> message BillRequest {
>  required uint64 time = 1;
>  optional uint64 expires = 2;
>  required string bill_request_uri = 3;
> }
>
>
> time - Unix timestamp (seconds since 1-Jan-1970 UTC) when the BillRequest
> was created.
> expires - Unix timestamp (UTC) after which the BillRequest should be
> considered invalid (wallets may only be monitoring for messages for a short
> time).
> bill_req_addr - Typically a URL where a BIP70 payment request can be sent
> that is being monitored by the wallet. However this could also support URIs
> like "sms:860-555-1212" or "mailto:a...@gmail.com"; allowing for a variety
> of connection options.
>
> Recommendations: it is recommended that wallet apps display a non-QR
> barcode like a PDF417 barcode to initiate the Bill Request flow. This will
> avoid confusion with existing QR code usage in wallet apps.
>
> 
> Paul G.
>
>
> --
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bill Request Message - (another) Proposed BIP 70 extension

[Mike Hearn]

Alright. I still tend to think it's not a big deal, but there's no reason
both or all mechanisms can't co-exist.

BTW: a QR code next to a cash register can be fixed i.e. printed on paper
when using BIP70. The PoS would upload payment details to the server and
the URL for that particular PoS unit would then serve it when the user
scans the QR code. Alternatively, Andreas' work on Bluetooth may be more
appropriate: the QR code can contain the BT MAC of the device and the
payment request is downloaded that way. That's already implemented! I still
feel that if a seller can scan a users phone, the users phone can certainly
scan some rectangle that's physically near by the sales counter.

The other nice thing about that approach is the QRcode can also be an NFC
tag i.e. have the tag behind it with a little icon in the middle of the QR
code to indicate that touching works as well as scanning.

One project I keep wanting to play with is making these little NFC-QRcode
hybrids and a simple PoS app to go with them. But no time, alas 
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Payment Protocol for Face-to-face Payments

[Mike Hearn]

>
> ​However it's not ideal at the moment. Basically the main problem is that
> in the BIP72 there is no way to provide a fallback alternative URI for
> payment request fetch if client wallet supports BIP70 but doesn't not
> support fetching over bluetooth or bluetooth connection fails for any
> reason.
>

So the idea here is that the recipient wallet both uploads to the internet
and exposes the payment request over Bluetooth simultaneously, then let's
the sending wallet pick whatever radio layer works best in its current
conditions?

I think having multiple r= params is reasonable, but the Bluetooth support
is not specced in any BIP anyway. And if it were to be, people would point
out the lack of link-layer encryption.

So this is a bit tricky, overall. Right now I'd say things are kinda half
baked: not only is bluetooth not standardised nor encrypted (my fault, I
prototyped this code during a hackathon), but Bitcoin Wallet doesn't
properly implement BIP 72 either. To push this work forward I think we need
to sit down and do some more spec and implementation work :/
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Payment Protocol for Face-to-face Payments

[Mike Hearn]

Nope, r1/r2 sounds good to me. BTW we should update the spec to reflect
that escaping is largely unnecessary in many cases.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ASIC-proof mining

[Mike Hearn]

Yup, no need to apologise. If nothing else the conversations get archived
where other people can use them to get up to speed faster. A lot of these
discussions get spread across forums, lists and IRC so it can be hard to
know what the current state of the art thinking is.

Recall the second prong of my opening argument - if you could beat ASICs,
you'd end up with botnets. I prefer having the chain be dominated by a
single pool for a while than having one with a major botnet presence, given
their history of doing things like mining empty blocks and giving random
people enormous electricity bills.

I think we can make good head way if we just optimise a lot and finish
things off, to be honest. I'm not sure we need an algorithmic silver
bullet. Remember you can always outsource mining by just not having any
hardware at all, CEX style, so trying to prevent outsourcing using clever
hacks seems ultimately doomed.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] ASIC-proof mining

[Mike Hearn]

>
> Is it possible instead to allocate a portion of the reward to " a # of
> runner up(s)" even though the runner-up(s) block will be orphaned?
>

There's really no concept of a "runner up" because hashing is progress
free. It's unintuitive and often trips people up. There's no concept that
everyone is 95% of the way to finding a solution and then someone pips you
to the post. It's more like playing the lottery over and over again.
Doesn't matter how many times you did it before, the next time your chances
are the same.

A better concept is of rewarding "near miss" solutions which is what we
already do of course, via pools, which pay you for shares which don't quite
meet the difficulty target but almost do. So the question is how can we
implement pools which have this reward structure (which obviously works
well) without miners simultaneously giving up their right to block creation
either due to technical problems or sheer lazyness.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Protocol Specification

[Mike Hearn]

On Tue, Jul 8, 2014 at 10:04 PM, Matt Whitlock 
wrote:

> Is anyone working on a similar specification document for Satoshi's P2P
> protocol?  I know how blocks and transactions are structured and verified,
> but I'm interested in knowing how they're communicated over the network.


There's a high level guide here:

https://bitcoin.org/en/developer-guide#p2p-network

It's not a protocol specification however, more a lay-of-the-land type
description. The protocol is not very complicated however. The wiki page
describes it adequately. It's basically just a framing mechanism with
excessively aggressive checksumming, a version handshake, an inventory
mechanism to cut down on bandwidth usage, the alerts, and that's about it.
--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Draft BIP for geutxos message

[Mike Hearn]

I opened up a pull req for a draft BIP for getutxo.

   https://github.com/bitcoin/bips/pull/88

I include a rendering below for your reading convenience. If you'd like to
comment on design/security/etc then please first familiarise yourself with
the long discussions that were already had here:

   https://github.com/bitcoin/bitcoin/pull/4351


  BIP: 45
  Title: getutxo message
  Author: Mike Hearn 
  Status: Draft
  Type: Standards Track
  Created: 2014-06-10

Table of Contents

   - Abstract
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Abstract>
  - Motivation
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Motivation>
  - Specification
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Specification>
  - Backward compatibility
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Backward_compatibility>
  - Authentication
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Authentication>
  - Implementation
  
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Implementation>

<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#abstract>
Abstract

This document describes a small P2P protocol extension that performs UTXO
lookups given a set of outpoints.
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#motivation>
Motivation

All full Bitcoin nodes maintain a database called the unspent transaction
output set. This set is how double spending is checked for: to be valid a
transaction must identify unspent outputs in this set using an identifier
called an "outpoint", which is merely the hash of the output's containing
transaction plus an index.

The ability to query this can sometimes be useful for a lightweight/SPV
client which does not have the full UTXO set at hand. For example, it can
be useful in applications implementing assurance contracts to do a quick
check when a new pledge becomes visible to test whether that pledge was
already revoked via a double spend. Although this message is not strictly
necessary because e.g. such an app could be implemented by fully
downloading and storing the block chain, it is useful for obtaining
acceptable performance and resolving various UI cases.

Another example of when this data can be useful is for performing floating
fee calculations in an SPV wallet. This use case requires some other
changes to the Bitcoin protocol however, so we will not dwell on it here.
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#specification>
Specification

Two new messages are defined. The "getutxos" message has the following
structure:

Field SizeDescriptionData typeComments1check mempoolboolWhether to apply
mempool transactions during the calculation, thus exposing their UTXOs and
removing outputs that they spend.?outpointsvectorThe list of outpoints to
be queried. Each outpoint is serialized in the same way it is in a tx
message.

The response message "utxos" has the following structure:

Field SizeDescriptionData typeComments4chain heightuint32The height of the
chain at the moment the result was calculated.32chain tip hashuint256Block
hash of the top of the chain at the moment the result was calculated.?hit
bitmapbyte[]An array of bytes encoding one bit for each outpoint queried.
Each bit indicates whether the queried outpoint was found in the UTXO set
or not.?result utxosresult[]A list of result objects (defined below), one
for each outpoint that is unspent (i.e. has a bit set in the bitmap).

The result object is defined as:

Field SizeDescriptionData typeComments4tx versionuint32The version number
of the transaction the UTXO was found in.4heightuint256The height of the
block containing the defining transaction, or 0x7FFF if the tx is in
the mempool.?outputCTxOutThe output itself, serialized in the same way as
in a tx message.
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#backward-compatibility>Backward
compatibility

Nodes indicate support by advertising a protocol version above 70003 and by
setting a new NODE_GETUTXO flag in their nServices field, which has a value
of 2 (1
<https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#authentication>
Authentication

The UTXO set is not currently authenticated by anything. There are
proposals to resolve this by introducing a new consensus rule that commits
to a root hash of the UTXO set in blocks, howev

Re: [Bitcoin-development] Draft BIP for geutxos message

[Mike Hearn]

I took the number out, it is now just "the getutxo bip" until a number is
assigned.


On Thu, Jul 10, 2014 at 4:29 PM, Mike Hearn  wrote:

> I opened up a pull req for a draft BIP for getutxo.
>
>https://github.com/bitcoin/bips/pull/88
>
> I include a rendering below for your reading convenience. If you'd like to
> comment on design/security/etc then please first familiarise yourself with
> the long discussions that were already had here:
>
>https://github.com/bitcoin/bitcoin/pull/4351
>
>
>   BIP: 45
>   Title: getutxo message
>   Author: Mike Hearn 
>   Status: Draft
>   Type: Standards Track
>   Created: 2014-06-10
>
>  Table of Contents
>
>- Abstract
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Abstract>
>   - Motivation
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Motivation>
>   - Specification
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Specification>
>   - Backward compatibility
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Backward_compatibility>
>   - Authentication
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Authentication>
>   - Implementation
>   
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#Implementation>
>
>
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#abstract>
> Abstract
>
> This document describes a small P2P protocol extension that performs UTXO
> lookups given a set of outpoints.
>
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#motivation>
> Motivation
>
> All full Bitcoin nodes maintain a database called the unspent transaction
> output set. This set is how double spending is checked for: to be valid a
> transaction must identify unspent outputs in this set using an identifier
> called an "outpoint", which is merely the hash of the output's containing
> transaction plus an index.
>
> The ability to query this can sometimes be useful for a lightweight/SPV
> client which does not have the full UTXO set at hand. For example, it can
> be useful in applications implementing assurance contracts to do a quick
> check when a new pledge becomes visible to test whether that pledge was
> already revoked via a double spend. Although this message is not strictly
> necessary because e.g. such an app could be implemented by fully
> downloading and storing the block chain, it is useful for obtaining
> acceptable performance and resolving various UI cases.
>
> Another example of when this data can be useful is for performing floating
> fee calculations in an SPV wallet. This use case requires some other
> changes to the Bitcoin protocol however, so we will not dwell on it here.
>
> <https://github.com/mikehearn/bips/commit/6058b92f5d9804ee4104649f53afc2fa53248c81?short_path=35c7795#specification>
> Specification
>
> Two new messages are defined. The "getutxos" message has the following
> structure:
>
>  Field Size DescriptionData typeComments 1check mempoolbool Whether to
> apply mempool transactions during the calculation, thus exposing their
> UTXOs and removing outputs that they spend. ?outpointsvector The list of
> outpoints to be queried. Each outpoint is serialized in the same way it is
> in a tx message.
>
> The response message "utxos" has the following structure:
>
>  Field Size DescriptionData typeComments 4chain heightuint32 The height
> of the chain at the moment the result was calculated. 32chain tip hash
> uint256 Block hash of the top of the chain at the moment the result was
> calculated. ?hit bitmapbyte[] An array of bytes encoding one bit for each
> outpoint queried. Each bit indicates whether the queried outpoint was found
> in the UTXO set or not. ?result utxosresult[] A list of result objects
> (defined below), one for each outpoint that is unspent (i.e. has a bit set
> in the bitmap).
>
> The result object is defined as:
>
>  Field Size DescriptionData typeComments 4tx versionuint32 The version
> number of the transaction the UTXO was found in. 4heightuint256 The
> height of the block containing the defining transaction, or 0x7FFF if
> the tx is in the mempool. ?outputCTxOut The output itself, serialized in
> 

Re: [Bitcoin-development] Self-dependency transaction question...

[Mike Hearn]

Conceptually all transactions in the block chain lie on a single timeline.
The fact that we quantise that timeline into blocks is in many ways neither
here nor there - it's still a strict line.

What *can* happen and you must be aware of is duplicated transactions.
Satoshi sort of assumed this could never happen because everything is hash
based, but forgot that duplicating coinbases is possible and at one point
this did happen. It was banned by a rule change afterwards but you still
must be able to process the older parts of the chain that have this. There
is a BIP that covers the new rule.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Protocol Specification

[Mike Hearn]

Nice work, but please don't call it the "Bitcoin protocol spec". Your
document is not a spec. It is an attempt to describe in English the Bitcoin
protocol, but anyone who implemented it based on your description would get
it wrong. For example you didn't mention the SIGHASH_SINGLE bug and many
other important areas like the difficulty transitions are also left
unspecified.

As a loose description of the protocol for newbies it's an invaluable
resource and perhaps we should link to it from the developer guide. As
something that claims to be a specification it is quite possibly dangerous
- the only spec that matters is the C++ original.


On Mon, Jul 14, 2014 at 11:54 AM, Krzysztof Okupski <
k.okup...@googlemail.com> wrote:

>  Dear all,
>
> thank you for your invaluable feedback. As requested, the spec
> will from now on be under version control. It can be found under:
>
> https://github.com/minium/Bitcoin-Spec
>
> The old link to the PDF will be, just in case, kept updated as well.
>
>
> Warm greetings,
> Krzysztof Okupski
>
>
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck®
> Code Sight™ - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Protocol Specification

[Mike Hearn]

Ah, that's great. Still, it would be good to be careful with the word
"specification".


On Mon, Jul 14, 2014 at 1:41 PM, Wladimir  wrote:

> > As a loose description of the protocol for newbies it's an invaluable
> > resource and perhaps we should link to it from the developer guide.
>
> It has already been linked from the developer guide for quite a while,
> under Additional Resources.
>
> Wladimir
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Protocol Specification

[Mike Hearn]

>
> just out of curiosity, do you think it will be possible to create any
> other proper protocol specifications rather than the C++  original?


Well it's a finite code base so yes, it should be possible.

The only problem is  so far everyone who tried it, didn't succeed :)
Heck even people who tried to reimplement it by reading the code keep
getting subtle details wrong.

So it should definitely be possible one day, assuming Bitcoin doesn't
become radically more complex, but it's a minefield.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin address TTL & key expiration?

[Mike Hearn]

>
> The request:   It would be useful to limit the lifetime of a bitcoin
> address.


Not only useful but essential! Otherwise mobile clients can run out of RAM
and have to cycle around and reuse addresses.

Which is indeed why BIP70 has this feature. It was thought about quite some
time ago. Addresses are an evolutionary dead end, they will never do
everything we need them to do. If there's somewhere that's using addresses,
that's somewhere we will eventually need to upgrade to use BIP70 instead.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

[+cc aaron]

We recently added an implementation of BIP 38 (password protected private
keys) to bitcoinj. It came to my attention that the third test vector may
be broken. It gives a hex version of what the NFC normalised version of the
input string should be, but this does not match the results of the Java
unicode normaliser, and in fact I can't even get Python to print the names
of the characters past the embedded null. I'm curious where this normalised
version came from.

Given that "pile of poo" is not a character I think any sane user would put
into a passphrase, I question the value of this test vector. NFC form is
intended to collapse things like umlaut control characters onto their prior
code point, but here we're feeding the algorithm what is basically garbage
so I'm not totally surprised that different implementations appear to
disagree on the outcome.

Proposed action: we remove this test vector as it does not represent any
real world usage of the spec, or if we desperately need to verify NFC
normalisation I suggest using a different, more realistic test string, like
Zürich, or something written in Thai.



Test 3:

   - Passphrase ϓ␀𐐀💩 (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK
   UPSILON WITH HOOK , COMBINING ACUTE ACCENT
   , NULL , DESERET
   CAPITAL LETTER LONG I , PILE OF POO
   )
   - Encrypted key:
   6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn
   - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
   - Unencrypted private key (WIF):
   5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
   - *Note:* The non-standard UTF-8 characters in this passphrase should be
   NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before
   further processing
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin address TTL & key expiration?

[Mike Hearn]

On Tue, Jul 15, 2014 at 4:02 PM, Jeff Garzik  wrote:

> BIP70 does not work well for unknown number of future payments of
> unknown, unpredictable value.


You can specify zero as an output value, in which case it's the same as "no
value specified". You can then just reuse the PaymentRequest until it
expires. So I think it provides the same functionality already.

Now sure, you'll get address reuse in this scenario, but that's no worse
than with an extended textual address.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin address TTL & key expiration?

[Mike Hearn]

>
> Payment protocol just doesn't well the use cases of,
> * an on-going payment stream from, e.g. Eligius to coinbase
> * deposit addresses and deposit situations


This seems to be the key point of disagreement here. Wladimir and I think
it satisfies your requirement just fine. You disagree. Let's get to the
bottom of that.

A PaymentRequest with a zero valued pay-to-address output and an expiration
time, base58 encoded, would look very much like your extended address form.
I don't suggest anyone actually represents PaymentRequest's using base58
but it helps to see the conceptual analogue. There'd be a bit more stuff in
there like some varint and wiretype codes but we're talking a handful of
bytes. Functionally, it'd be identical.

Places like protocols or APIs that require a piece of text and cannot
handle a piece of binary data could be retrofitted into the new world by
accepting base58 encoded PaymentRequest's. This would be kind of silly
because it's fundamentally binary data, but we already do this so it's at
least consistently silly :)
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

>
> Unicode guarantees that null-terminated strings still work.


UTF-8 guarantees that. Other encodings do not, you can have null bytes in
UTF-16 strings for example. Indeed most languages that use pascal-style
encodings internally allow null characters in strings, it's just not a good
idea to exploit that fact ...
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin address TTL & key expiration?

[Mike Hearn]

>
> Actually, and this is key, there _are_ reasons why deposits might not
> be able to use PaymentRequests.  Payments happen even when
> wallets/computers are offline.
>

I don't understand this point. It's the *sender* that is parsing the
PaymentRequest and following the instructions. By definition the sender
must be online. A computer that is switched off cannot sign a transaction
at all.


> If you have negotiated HD wallet details, you can use a new address
> every time, as mentioned.


Yes, and an extension to BIP 70 to allow for this (or stealth addresses or
whatever) has been discussed several times.

This thread started by proposing (I think) an expiry time for addresses.
BIP70 satisfies this use case, I think we all agree on that. Now for cases
where someone can't use BIP70 for whatever reason, or it's suboptimal,
absolutely we should design extensions to fix that.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

Yes, we know, Andreas' code is indeed doing normalisation.

However it appears the output bytes end up being different. What I get back
is:

cf9300*01*303430300166346139

vs

cf9300*f0*909080f09f92a9

from the spec.

I'm not sure why. It appears this is due to the character from the astral
planes. Java is old and uses 16 bit characters internally - it wouldn't
surprise me if there's some weirdness that means it doesn't/won't support
this kind of thing.

I recommend instead that any implementation that wishes to be compatible
with JVM based wallets (I suspect Android is the same) just refuse any
passphrase that includes characters outside the BMP. At least unless
someone can find a fix. I somehow doubt this will really hurt anyone.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

I'm all for fixing bugs, but I know from bitter experience that outside the
BMP dragons lurk. Browsers don't even expose Unicode APIs at all. You end
up needing to ship an entire pure-js implementation, which can be too large
for some use cases (too much time sunk on that issue in my last job).

I'm hoping BIP 38 doesn't get widely used anyway, to be frank. People
moving private keys around by hand has caused quite a few problems in the
past, sometimes people lost money. It's better to work at the level of a
wallet and ideally ask people to move money using regular transactions. Way
less potential for errors.

Regardless, I'll file a JVM bug and see what the outcome is.


On Wed, Jul 16, 2014 at 12:23 AM, Aaron Voisine  wrote:

> If the user creates a password on an iOS device with an astral
> character and then can't enter that password on a JVM wallet, that
> sucks. If JVMs really can't support unicode NFC then that's a strong
> case to limit the spec to the subset of unicode that all popular
> platforms can support, but it sounds like it might just be a JVM
> string library bug that could hopefully be reported and fixed. I get
> the same result as in the test case using apple's
> CFStringNormalize(passphrase, kCFStringNormalizationFormC);
>
> Aaron Voisine
> breadwallet.com
>
>
> On Tue, Jul 15, 2014 at 11:20 AM, Mike Hearn  wrote:
> > Yes, we know, Andreas' code is indeed doing normalisation.
> >
> > However it appears the output bytes end up being different. What I get
> back
> > is:
> >
> > cf930001303430300166346139
> >
> > vs
> >
> > cf9300f0909080f09f92a9
> >
> > from the spec.
> >
> > I'm not sure why. It appears this is due to the character from the astral
> > planes. Java is old and uses 16 bit characters internally - it wouldn't
> > surprise me if there's some weirdness that means it doesn't/won't support
> > this kind of thing.
> >
> > I recommend instead that any implementation that wishes to be compatible
> > with JVM based wallets (I suspect Android is the same) just refuse any
> > passphrase that includes characters outside the BMP. At least unless
> someone
> > can find a fix. I somehow doubt this will really hurt anyone.
> >
> >
> --
> > Want fast and easy access to all the code in your enterprise? Index and
> > search up to 200,000 lines of code with a free copy of Black Duck
> > Code Sight - the same software that powers the world's largest code
> > search on Ohloh, the Black Duck Open Hub! Try it now.
> > http://p.sf.net/sfu/bds
> > ___
> > Bitcoin-development mailing list
> > Bitcoin-development@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

Yes sorry, you're right, the issue starts with the null code point. Python
seems to have problems starting there too. It might work if we took that
out.


On Wed, Jul 16, 2014 at 11:17 AM, Andreas Schildbach 
wrote:

> Guys, you are always talking about the Unicode astral plane, but in fact
> its a plain old (ASCII) control character where this problem starts and
> likely ends: \u.
>
> Let's ban/filter ISO control characters and be done with it. Most
> control characters will never be enterable by any keyboard into a
> password field. Of course I assume that Character.isISOControl() works
> consistently across platforms.
>
>
> http://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl%28char%29
>
>
> On 07/16/2014 12:23 AM, Aaron Voisine wrote:
> > If the user creates a password on an iOS device with an astral
> > character and then can't enter that password on a JVM wallet, that
> > sucks. If JVMs really can't support unicode NFC then that's a strong
> > case to limit the spec to the subset of unicode that all popular
> > platforms can support, but it sounds like it might just be a JVM
> > string library bug that could hopefully be reported and fixed. I get
> > the same result as in the test case using apple's
> > CFStringNormalize(passphrase, kCFStringNormalizationFormC);
> >
> > Aaron Voisine
> > breadwallet.com
> >
> >
> > On Tue, Jul 15, 2014 at 11:20 AM, Mike Hearn  wrote:
> >> Yes, we know, Andreas' code is indeed doing normalisation.
> >>
> >> However it appears the output bytes end up being different. What I get
> back
> >> is:
> >>
> >> cf930001303430300166346139
> >>
> >> vs
> >>
> >> cf9300f0909080f09f92a9
> >>
> >> from the spec.
> >>
> >> I'm not sure why. It appears this is due to the character from the
> astral
> >> planes. Java is old and uses 16 bit characters internally - it wouldn't
> >> surprise me if there's some weirdness that means it doesn't/won't
> support
> >> this kind of thing.
> >>
> >> I recommend instead that any implementation that wishes to be compatible
> >> with JVM based wallets (I suspect Android is the same) just refuse any
> >> passphrase that includes characters outside the BMP. At least unless
> someone
> >> can find a fix. I somehow doubt this will really hurt anyone.
> >>
> >>
> --
> >> Want fast and easy access to all the code in your enterprise? Index and
> >> search up to 200,000 lines of code with a free copy of Black Duck
> >> Code Sight - the same software that powers the world's largest code
> >> search on Ohloh, the Black Duck Open Hub! Try it now.
> >> http://p.sf.net/sfu/bds
> >> ___
> >> Bitcoin-development mailing list
> >> Bitcoin-development@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >>
> >
> >
> --
> > Want fast and easy access to all the code in your enterprise? Index and
> > search up to 200,000 lines of code with a free copy of Black Duck
> > Code Sight - the same software that powers the world's largest code
> > search on Ohloh, the Black Duck Open Hub! Try it now.
> > http://p.sf.net/sfu/bds
> >
>
>
>
>
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Draft BIP for geutxos message

[Mike Hearn]

Thanks Jeff.

I do feel like a lot of this is covered in the writeup I attached to the
implementation pull request, and I went over it again in the ensuing
discussion, and also in the BIP.

The discussion of how to make it secure is covered in the "Upgrade" section
of the writeup and in the "Authentication" section of the BIP. Please do
let me know if these sections are missing something. The ideas discussed
there are not implemented in this pull request because outside of some
special cases, it is a very large project that involves a chain fork. You
can see the start of a solution here:

https://github.com/bitcoin/bitcoin/pull/3977


> If one implements your BIP in a naive manner -- simply find a node, and
> issue a single query -- they are dangerously exposed to malicious
> information.  The BIP should describe this major security issue, and
> describe at least one method of solving it (ditto implementation, if
> lighthouse has not already solved this).
>

The BIP already does discuss this, in the authentication section.
Suggestions for how to make it better are welcome.


> Comparison between this and BIP 35 (mempool command) are not apt, as
> miners and full nodes treat "mempool" returned data just like any other
> randomly solicited "tx" command on the network.  Unlike "mempool" cmd, this
> "getutxos" cmd proffers post-verification trusted data.
>

I don't think it does proffer that, but if a part of the BIP could be read
as doing so, let me know which part and I'll fix it.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Draft BIP for geutxos message

[Mike Hearn]

>
> On the specific issue I raised, the BIP only says "Querying multiple
> nodes and combining their answers can be a partial solution to this"
> which is not very helpful advice.  That's a partial answer to my
> question #2 with zero response for question #3.
>

I'm sorry you think it's unhelpful. It is nonetheless the best that can be
done within the constraints of the current Bitcoin protocol.


> This sort of thing really needs a warning label like "use only if you
> don't have a trusted solution" and discussion of that choice is
> completely absent (question #1).
>

It's absent for the same reason it's absent for all the other protocol
BIPs: the ability to use a trusted third party is always present and a
possible answer for any problem in Bitcoin. So I figured it didn't need
stating.

How about adding the following sentence:

"If the above constraints are insufficient for your use case, you can
alternatively query a block explorer or other trusted third party to obtain
the same information".

Would that make the BIP clearer?
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Draft BIP for geutxos message

[Mike Hearn]

>
> In particular, can this document
> specifically call out that a local network attacker can MITM all the
> peers.


It already does, last sentence of the authentication section is:

Querying multiple nodes and combining their answers can be a partial
solution to this, although as nothing authenticates the Bitcoin P2P network
a man in the middle could still yield incorrect results



> (If Mike would prefer, I can send a diff with proposed changes)
>

Yes please.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 38 NFC normalisation issue

[Mike Hearn]

Glad we got to the bottom of that. That's quite a nasty compiler/language
bug I must say. Not even a warning. Still, python crashes when trying to
print the name of a null character. It wouldn't surprise me if there are
other weird issues lurking. Would definitely sleep better with a more
restricted character set.
On 17 Jul 2014 00:04, "Andreas Schildbach"  wrote:

> Please excuse me. I had a more thorough look at the original problem and
> found that the only problem with the original test case was that you
> cannot specify codepoints from the SMP using \u in Java. I always tried
> \u010400 but that doesn't work.
>
> Here is a fix for bitcoinj. The test now passes.
>
> https://github.com/bitcoinj/bitcoinj/pull/143
>
> We can (and probably should) still need to filter control chars, I'll
> have a look at that now again.
>
>
> On 07/16/2014 11:06 PM, Aaron Voisine wrote:
> > If I first remove \u, so the non-normalized passphrase is
> > "\u03D2\u0301\U00010400\U0001F4A9", and then NFC normalize it, it
> > becomes "\u03D3\U00010400\U0001F4A9"
> >
> > UTF-8 encoded this is: 0xcf93f0909080f09f92a9 (not the same as what
> > you got, Andreas!)
> >
> > Encoding private key: 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> > with this passphrase, I get a BIP38 key of:
> > 6PRW5o9FMb4hAYRQPmgcvVDTyDtr6R17VMXGLmvKjKVpGkYhBJ4uYuR9wZ
> >
> > I recommend rather than simply removing control characters from the
> > password that instead the spec require that passwords containing
> > control characters are invalid. We don't want people trying to be
> > clever and putting them in thinking they are adding to the password
> > entropy.
> >
> > Also for UI compatibility across many platforms, I'm also in favor
> > disallowing any character below U+0020 (space)
> >
> > I can submit a PR once we figure out why Andreas's passphrase was
> > different than what I got.
> >
> > Aaron Voisine
> > breadwallet.com
> >
> >
> > On Wed, Jul 16, 2014 at 4:04 AM, Andreas Schildbach
> >  wrote:
> >> Damn, I just realized that I implement only the decoding side of BIP38.
> >> So I cannot propose a complete test vector. Here is what I have:
> >>
> >>
> >> Passphrase: ϓ␀𐐀💩 (\u03D2\u0301\u\U00010400\U0001F4A9; GREEK
> >> UPSILON WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER
> >> LONG I, PILE OF POO)
> >>
> >> Passphrase bytes after removing ISO control characters and NFC
> >> normalization: 0xcf933034303066346139
> >>
> >> Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
> >>
> >> Unencrypted private key (WIF):
> >> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
> >>
> >>
> >> Can someone calculate the encrypted key from it (using whatever
> >> implementation) and I will verify it decodes properly in bitcoinj?
> >>
> >>
> >>
> >> On 07/16/2014 12:46 PM, Andreas Schildbach wrote:
> >>> I will change the bitcoinj implementation and propose a new test
> vector.
> >>>
> >>>
> >>>
> >>> On 07/16/2014 11:29 AM, Mike Hearn wrote:
> >>>> Yes sorry, you're right, the issue starts with the null code point.
> >>>> Python seems to have problems starting there too. It might work if we
> >>>> took that out.
> >>>>
> >>>>
> >>>> On Wed, Jul 16, 2014 at 11:17 AM, Andreas Schildbach
> >>>> mailto:andr...@schildbach.de>> wrote:
> >>>>
> >>>> Guys, you are always talking about the Unicode astral plane, but
> in fact
> >>>> its a plain old (ASCII) control character where this problem
> starts and
> >>>> likely ends: \u.
> >>>>
> >>>> Let's ban/filter ISO control characters and be done with it. Most
> >>>> control characters will never be enterable by any keyboard into a
> >>>> password field. Of course I assume that Character.isISOControl()
> works
> >>>> consistently across platforms.
> >>>>
> >>>>
> http://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl%28char%29
> >>>>
> >>>>
> >>>> On 07/16/2014 12:23 AM, Aaron Voisine wrote:
> >>>> > If the user creates a password on an iOS device with an astral
> >>>> > character and then can

Re: [Bitcoin-development] Decentralizing ming

[Mike Hearn]

Jeff, I think the message you're replying to got clipped.

Satoshi's only comment AFAIK on the topic of GPU mining was to wish for a
gentlemen's agreement to postpone it as long as possible, to help make sure
the distribution of coins was as even as possible. Indeed this predated
pooled mining.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Decentralizing ming

[Mike Hearn]

Oops, sorry, I see the subject line changed. This is what I get for working
down the thread list top to bottom :)

I think the best path forward now is to finish off getblocktemplate support
in the various tools so it's possible to pool for payout purposes without
giving up control of block creation modulo the coinbase. Combined with the
recent sipa performance enhancing goodness, it would hopefully persuade
some non-trivial chunk of hashpower to go back to running their own node
and start the process of turning pools merely into payout trackers rather
than block selectors.


On Fri, Jul 18, 2014 at 12:41 PM, Mike Hearn  wrote:

> Jeff, I think the message you're replying to got clipped.
>
> Satoshi's only comment AFAIK on the topic of GPU mining was to wish for a
> gentlemen's agreement to postpone it as long as possible, to help make sure
> the distribution of coins was as even as possible. Indeed this predated
> pooled mining.
>
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Small update to BIP 62

[Mike Hearn]

The rationale doesn't seem to apply to rule #4, what's so special about
that one?

Although I agree not having to support all of DER is nice, in practice I
think all implementations do and libraries to parse DER are widespread.
Given that the last time we modified tx rules without bumping version
numbers we managed to break the only functioning iPhone client, I've become
a big fan of backwards compatibility: seems the default choice should be to
preserve compatibility over technical niceness until the old versions have
been fully phased out.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Trickle and transaction propogation

[Mike Hearn]

No comment on the proposal itself, which sounds reasonable but I didn't
think about it much yet. Instead, just an observation that by now most
users are not using the Core wallet anymore, but rather use either SPV
wallets or more centralised blockchain/coinbase style gateways to the
network.

SPV wallets don't relay thus you know any tx sent from them must be
originated by that wallet. Centralised services accept tx submissions via
SSL and can easily improve their users privacy by sending transactions out
via a node that isn't listening.

So IMHO we should be optimising the network for the common use case rather
than stuff that only helps Core wallet users, and actively slows down
everyone else. If your proposed techniques let us have our cake and eat it,
fantastic, otherwise I still think we should remove tx trickling.



On Sun, Jul 20, 2014 at 11:01 PM, Kaz Wesley  wrote:

> The inv trickling mechanism currently serves two purposes:
> - protect casual users' privacy by slightly obscuring a tx's originating
> node
> - reduce invs unnecessarily sent both directions for a connection
> It has some drawbacks:
> - it slows transaction propagation
> - it delays knowledge between two nodes of what txes are mutually known
> These drawbacks will be especially costly once optimizations based on
> mutually-known transactions are available (in progress, see "sparse
> blocks" thread).
>
> Both of the benefits of trickling can be achieved more efficiently and
> without the costs to transaction propagation and mutual transaction
> knowledge.
>
> Privacy: trickling helps hide the origin of 3/4 of the transactions a
> node is pushing by preventing most of the node's neighbors from seeing
> the transactions from that node right away; by the time a peer becomes
> the trickle node, it may have received the same inv from another of
> its peers.
> This staggering of introduction of new invs to the network could be
> made more effective by scheduling staggered pushes of wallet
> transactions to each peer in a structure similar to mapAskFor.
> This does have the drawback that someone who has established multiple
> connections to a node can observe that some invs are pushed at
> different times, suggesting they are in the stagger set. I don't see
> any straightforward way to remedy this, but trickling is also
> vulnerable to sybil attacks, and floods 1/4 of its transactions
> immediately anyway -- so I think staggered push would be an overall
> privacy improvement.
> Likelihood of a partial sybil obtaining inv origin information could
> be reduced by a policy of ending staggering and pushing to all peers
> once another peer has received the tx from elsewhere and inved the
> transaction back to the original node; if the staggering is
> sufficiently slow, only one or two nodes would receive the initial
> push to the network and after that the inv would be treated
> indistinguishably from if it originated externally.
>
> Redundant invs: without trickling, when two nodes receive transactions
> at around the same time they may each send each other an inv before
> receiving the other's. Trickling reduces this by giving all
> non-trickleSend nodes a chance to send first. Thus just eliminating
> trickling would at most double inv traffic. Although invs are small
> they are numerous, being the only common message potentially sent from
> every node to all its neighbors.
> A more efficient solution to the who-sends-first problem would be for
> connections to have directional parity:
> - a node initiating a connection would announce its parity (even or odd)
> - an inv is sent right away to peers with matching parity, but only
> sent against parity if a certain timeout has elapsed without the inv
> being received
> In order to allow for nodes with few peers (i.e. -connect) or nodes on
> local connections that might as well flood everything to each other,
> parity could be specified as a mask (fEven << 1 & fOdd). Peers from
> pre-directional-parity versions can be treated as having the mask
> fully set.
>
> Both push staggering and directional parity admit simple
> implementations. The specific staggering delay distribution would need
> some thought; it could be set slower than the typical trickle cycle
> period for better than current privacy, since general transaction
> propagation would not impeded by heavy staggering. What do you think
> of this approach? Any gotchas/improvements/alternatives?
>
>
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/

Re: [Bitcoin-development] Question on creating test cases for block.CheckBlock()

[Mike Hearn]

There is no infrastructure for writing block chain unit tests
unfortunately. Last time I tried to fix this I ended up going down a rabbit
hole - Bitcoin wasn't written to be a testable codebase and as a result
reinitialising it from scratch is rather difficult (there are lots of
global variables that have to be reset to particular states and no real
list of where they are).

Instead what I ended up doing is extending the pull tester. This is a
bitcoinj based app (BitcoindComparisonTool in the codebase) which builds a
regtest chain and submits it to a local regtest node. It tests things like
reorgs and various rules. It speaks to the node only via P2P so how easy it
is to verify your BIP works will depend on that. Also the code needs
cleaning up, there's a lot of copy/paste coding going on in there.




On Mon, Jul 21, 2014 at 7:30 PM, Sergio Lerner 
wrote:

> I'm working on a BIP which needs to modify the block acceptance rules. I
> have two ways of testing:
>
> - Mining blocks on the testnet
> - Creating test cases for Bitcoin Core.
>
> I want to create those test cases for block.CheckBlock(), which involves
> verifying 100 dynamically generated blocks.
> What is the state of the blockchain when a test case is executed ? Is is
> configured for the regtest, testnet3 or mainnet? What blocks are in the
> blockchain? Only the genesis block?
>
> checkblock_tests.cpp seems to be the only test case for CheckBlock() and
> it assumes the mainnet is configured.
> I need to use the regtest so I can create blocks of difficulty 1.
>
> Best regards and thank you in advance,
>
>
>
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Time

[Mike Hearn]

>
> Ok... 'time' on the blockchain could be 'gamed' ... but with great
> difficulty.


Unfortunately not: miners have in the past routinely gamed the timestamp in
order to use it as an extra nonce and squeeze some more gigahashes out of
their hardware/pool.

Also remember that currently the chain could be dominated by a coalition of
just two pools.


> An application presented with a fake blockchain can use
> quite a few heuristics to test the 'validity' of the block chain.
>

The app cannot tell if it was given a truncated chain. You could keep such
an app stuck in the past forever. This is often a problem.


> Reliable 'time' has been impossible up until now - because you need to
> trust the time source, and that can always be faked.  Using the
> blockchain as an approximate time source gives you a world wide
> consensus without direct trust of any player.
>

Much though I hate to be a party pooper, you could currently get
Bitcoin-level trusted time by just polling at least two or three
independent servers e.g. google.com, baidu.cn, yandex.ru(they all serve
time via HTTPS headers).

If we crack the mining decentralisation problem then this argument becomes
a lot stronger, but for now ..


> So if this presumption is correct, then we can now build time capsule
> applications that can not be tricked into exposing their contents too
> early by running them in a virtual environment with the wrong system time.


If you have a tamper resistant execution environment (TXT, SGX, Flicker
etc) then yes. However trusted execution environments sometimes have tamper
resistant clocks as well for exactly this reason. So whether this technique
makes sense depends a lot on the details of your configuration, I think.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Time

[Mike Hearn]

Given that the speed at which the block chain advances is kind of
unpredictable, I'd think it might be better to just record the time to disk
when a PIN attempt is made and if you observe time going backwards, refuse
to allow more attempts until it's advanced past the previous attempt.


On Fri, Jul 25, 2014 at 7:56 AM, Aaron Voisine  wrote:

> It's based on the block height, not the block's timestamp. If you have
> access to the device and the phone itself is not pin locked, then you
> can jailbreak it and get access to the wallet seed that way. A pin
> locked device however is reasonably secure as the filesystem is
> hardware aes encrypted to a combination of pin+uuid. This was just an
> easy way to prevent multiple pin guesses by changing system time in
> settings, so that isn't the weakest part of the security model.
>
> Aaron Voisine
> breadwallet.com
>
>
> On Thu, Jul 24, 2014 at 8:21 PM, William Yager 
> wrote:
> > On Thu, Jul 24, 2014 at 10:39 PM, Gregory Maxwell 
> > wrote:
> >>
> >>
> >> Is breadwallet tamper resistant & zero on tamper hardware? otherwise
> >> this sounds like security theater I attach a debugger to the
> >> process (or modify the program) and ignore the block sourced time.
> >>
> >
> > It's an iOS application. I would imagine it is substantially more
> difficult
> > to attach to a process (which, at the very least, requires root, and
> perhaps
> > other things on iOS) than to convince the device to change its system
> time.
> >
> > That said, the security benefits might not be too substantial.
> >
> >
> --
> > Want fast and easy access to all the code in your enterprise? Index and
> > search up to 200,000 lines of code with a free copy of Black Duck
> > Code Sight - the same software that powers the world's largest code
> > search on Ohloh, the Black Duck Open Hub! Try it now.
> > http://p.sf.net/sfu/bds
> > ___
> > Bitcoin-development mailing list
> > Bitcoin-development@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >
>
>
> --
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Time

[Mike Hearn]

Sorry, you're right. I'd have hoped a delay that doubles on failure each
time up to some max would be good enough, relying on the p2p network to
unlock a PIN feels weird, but I can't really quantify why or what's wrong
with it so I guess it's just me :-)


On Fri, Jul 25, 2014 at 4:45 PM, Aaron Voisine  wrote:

> The problem is if someone moves system time forward between app launches.
> The lockout period doesn't have to be all that precise, it just makes you
> wait for the next block, then 5, then 25, and so on. Using a well
> known time server over https would also be a good option, but the wallet
> app already has the chain height anyway.
>
>
> On Friday, July 25, 2014, Mike Hearn  wrote:
>
>> Given that the speed at which the block chain advances is kind of
>> unpredictable, I'd think it might be better to just record the time to disk
>> when a PIN attempt is made and if you observe time going backwards, refuse
>> to allow more attempts until it's advanced past the previous attempt.
>>
>>
>> On Fri, Jul 25, 2014 at 7:56 AM, Aaron Voisine  wrote:
>>
>>> It's based on the block height, not the block's timestamp. If you have
>>> access to the device and the phone itself is not pin locked, then you
>>> can jailbreak it and get access to the wallet seed that way. A pin
>>> locked device however is reasonably secure as the filesystem is
>>> hardware aes encrypted to a combination of pin+uuid. This was just an
>>> easy way to prevent multiple pin guesses by changing system time in
>>> settings, so that isn't the weakest part of the security model.
>>>
>>> Aaron Voisine
>>> breadwallet.com
>>>
>>>
>>> On Thu, Jul 24, 2014 at 8:21 PM, William Yager 
>>> wrote:
>>> > On Thu, Jul 24, 2014 at 10:39 PM, Gregory Maxwell 
>>> > wrote:
>>> >>
>>> >>
>>> >> Is breadwallet tamper resistant & zero on tamper hardware? otherwise
>>> >> this sounds like security theater I attach a debugger to the
>>> >> process (or modify the program) and ignore the block sourced time.
>>> >>
>>> >
>>> > It's an iOS application. I would imagine it is substantially more
>>> difficult
>>> > to attach to a process (which, at the very least, requires root, and
>>> perhaps
>>> > other things on iOS) than to convince the device to change its system
>>> time.
>>> >
>>> > That said, the security benefits might not be too substantial.
>>> >
>>> >
>>> --
>>> > Want fast and easy access to all the code in your enterprise? Index and
>>> > search up to 200,000 lines of code with a free copy of Black Duck
>>> > Code Sight - the same software that powers the world's largest code
>>> > search on Ohloh, the Black Duck Open Hub! Try it now.
>>> > http://p.sf.net/sfu/bds
>>> > ___
>>> > Bitcoin-development mailing list
>>> > Bitcoin-development@lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>> >
>>>
>>>
>>> --
>>> Want fast and easy access to all the code in your enterprise? Index and
>>> search up to 200,000 lines of code with a free copy of Black Duck
>>> Code Sight - the same software that powers the world's largest code
>>> search on Ohloh, the Black Duck Open Hub! Try it now.
>>> http://p.sf.net/sfu/bds
>>> ___
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>
>>
>
> --
>
> Aaron Voisine
> breadwallet.com
>
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] "On behalf of" BIP 70 extension proposal

[Mike Hearn]

Hi Mark,

This is very similar to a proposal I made some time ago:


https://www.mail-archive.com/bitcoin-development%40lists.sourceforge.net/msg04053.html

I think the outlines of a design are clear - my proposal and yours don't I
think differ substantially. Someone needs to make it happen though.
--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Abnormally Large Tor node accepting only Bitcoin traffic

[Mike Hearn]

> As I pointed out above, — it isn't really.  Without the exit flag, I
> believe no tor node will select it to exit 8333 unless manually
> configured. (someone following tor more closely than I could correct
> if I'm wrong here)
>

The "exit" flag doesn't mean what you would expect it to mean. The reason
such a node won't get much traffic is that Tor speculatively builds
circuits at startup on the assumption they'll be used for web browsing.
Thus if you don't exit web traffic you won't get much in the way of traffic
at least not until bitcoinj based wallets start shipping Tor mode.

There's a perfectly reasonable explanation for why someone would run such a
node. In fact I run a Tor exit that only allows port 8333 too: it's a way
to contribute exit bandwidth without much risk of getting raided by the
cops.

Occam's razor and all 
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] "On behalf of" BIP 70 extension proposal

[Mike Hearn]

On Mon, Jul 28, 2014 at 11:01 AM, Mark van Cuijk  wrote:

> Good to see that it has been discussed, but I see the idea has been
> postponed.
>

I'm not sure postponed is the right word. It wasn't in v1, but many useful
things weren't. It's more like, a bunch of people have to do work to
upgrade this and at the moment they're all busy with other things.


> I do like the idea coined by Mike that a PP can issue non-SSL certificates
> for the purpose of merchant identification, as long as a customer is free
> to determine whether he trusts the PP for this purpose.
>

I don't think I proposed this exactly? It's the other way around - a
merchant issues an extension cert to allow the PP to act on their behalf.


> Regarding the choice of how to authenticate the PP, I’m a bit
> undetermined. Disregarding backward compatibility, I think the extended
> certificate system proposed by Mike is cleaner. However, I don’t like the
> concept of requiring two separate signatures for old and new clients.
> Taking backward compatibility in mind, I tend to prefer my proposal.
>

I'm not sure I understand. Your proposal also has two signatures. Indeed it
must because delegation of authority requires a signature, but old clients
won't understand it.
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] "On behalf of" BIP 70 extension proposal

[Mike Hearn]

>
> I referred to your idea in
> https://www.mail-archive.com/bitcoin-development%40lists.sourceforge.net/msg04076.html
> 
>  which
> is indeed not the proposal itself.
>

Right, gotcha. Had forgotten about that.

Indeed there is another signature, which is to authenticate the payment
> delegation. If you take it into account in the signature count, then your
> proposal has three signatures.
>

Yes, I see now, you are right. A mandate type system is probably simpler
indeed.

So what now? To be honest my next priority with BIP70 was to formalise the
extensions process, I've been dragging my feet over that because I'm
working on other things. And then after that to knock some heads together
over at BitPay/Coinbase and get them to put useful text in the memo field
instead of random numbers. Baby steps 
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] "On behalf of" BIP 70 extension proposal

[Mike Hearn]

That would definitely be a new BIP.

But firstly it'd make sense to implement it and make sure that the payment
processors intend to use it. Like I said, I wasn't very successful so far
in getting them to make useful memo fields. I'm hoping that once wallets
start actually recording and displaying the memo in their transactions list
that will change.


On Wed, Jul 30, 2014 at 9:54 AM, Mark van Cuijk  wrote:

> On 28 Jul 2014, at 15:32 , Mike Hearn  wrote:
>
> > So what now? To be honest my next priority with BIP70 was to formalise
> the extensions process, I've been dragging my feet over that because I'm
> working on other things. And then after that to knock some heads together
> over at BitPay/Coinbase and get them to put useful text in the memo field
> instead of random numbers. Baby steps 
>
> I can probably pick up writing the proposal.
>
> However, I’m not sure what process to follow. Should I format the proposal
> as a new BIP or should it become part of BIP.70? How does the extensions
> process you’re working on going to describe the process?
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Abusive and broken bitcoin seeders

[Mike Hearn]

>
> I suspect it is something that is going to have to be dealt with in the
> future (I just don't know how yet).
>

The web has managed to survive despite constant fast crawls being the norm
for the past 10 years or so. I wouldn't worry too much about this unless
you can prove that a big chunk of your nodes resources are going to
answering ver queries.
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] How to create a pull tester JAR

[Mike Hearn]

I just checked in a change to bitcoinj git master that makes it much easier
to create a pull tester jar. Here are instructions for how to do it.

You will need:

   - A Java Development Kit (JDK), version 6 or up should work. As Java 6
   was released eight years ago, this should not be a challenging requirement.
   If you have a Mac just running "java" from the command line should give you
   a GUI prompt to install it automatically. Otherwise apt-get or fetch the
   latest from the interwebs.

   - Apache Maven. This is a rough equivalent of autotools, except it does
   dependency resolution for you. Grab it from
   http://maven.apache.org/download.cgi then unzip it and make sure the bin
   directory is in your PATH. You may need to set the JAVA_HOME environment
   variable if you installed Java to an odd place.

   - git

Make sure you can run "javac" from the command line, then make sure you can
run "mvn", it should complain it can't find a POM (this is a build config
file) and not, say, that it can't find Java.

Now grab bitcoinj from git master:

git clone https://github.com/bitcoinj/bitcoinj.git

... and build 

cd bitcoinj
mvn -DskipTests package

It will go off and download the libraries needed, compile, and create a
bundled executable JAR called core/target/pull-tests.jar. This is sort of
analogous to static linking in the Java world. It should be fast - expect a
full build plus downloads to take less than a minute. You can use it either
with the QA scripts in the bitcoin core qa/pull-tester directory or just
run things directly:

./bitcoind -regtest -connect=0.0.0.0 -listen -whitelist=127.0.0.1
-datadir=/tmp/pulltester
java -jar core/target/pull-tests.jar

It should go ahead and print lots of debug spew, then at the end say it's
happy.

Let me know if you encounter any problems with this.

Java JARs (which are just zip files renamed) are easily reproduced if you
use the same version of javac and the same bitcoinj version. The ZIP
container has timestamps, but unzipping them and simply diffing the files
between two builds should reveal no differences. I am happy to provide a
pull-tests.jar from my local machine if anyone would like to do this.
--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] How to create a pull tester JAR

[Mike Hearn]

Oh, I forgot to mention something important. Ridiculously, the default
package repository Maven uses was not protected by SSL up until a few days
ago.  They made it available via SSL now, but you have to tell Maven about
the new URL. I guess they'll do a new release where SSL is the default
soon. But for now before you run mvn save the following magic incantation
to the path ~/.m2/settings.xml:

(side note: yes maven's love of XML is widely ridiculed and more modern
build tools have much better config languages, but we didn't upgrade yet)


  

securecentral
  
  

  securecentral
  
  

  central
  https://repo1.maven.org/maven2
  
true
  

  
  

  central
  https://repo1.maven.org/maven2
  
true
  

  

  

--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development