[Bitcoin-development] Even simpler minimum fee calculation formula: f bounty*fork_rate/average_blocksize

2013-11-13 Thread Michael Gronager
Last week I posted a writeup: On the optimal block size and why
transaction fees are 8 times too low (or transactions 8 times too big).

Peter Todd made some nice additions to it including different pool sizes
into the numbers.

However, it occurred to me that things can in fact be calculated even
simpler: The measured fork rate will mean out all the different pool
sizes and network latencies and will as such provide a simple number we
can use to estimate the minimum fee. Key assumption is that the latency
will depend on block size (# txns) and the fork rate will depend on latency.

Using the formulas from last week:

P_fork = t_propagate/t_blocks

and:

t_propagate = t_0 + alpha*S ~= alpha*S

We get a measure for alpha as a function of the average fork rate and
average block size:

alpha = P_fork*t_block/S

Further, take the formula for the minimum fee:

f  alpha*E_bounty/t_block

And insert the formula for alpha:

f  P_fork*E_bounty/S_average

Luckily the fork frequency and the average block size are easily
measurable. blockchain.info keeps historical graphs of number of
orphaned blocks pr day - average over the last year is 1.5. Average
number of blocks per day over the last year is 169, which yields a
P_fork of ~1/113. Average block size in the same time is 134kBytes,
which yields a minimum fee:

f  0.00165XBT/kb or 0.00037XBT/txn

So the 0.0001 is only 4 times too small. Further, let us look at the
trend over the last 12 months. Pieter Wuille claimed that there has been
several improvements over the last half year that would bring down the
latency, there has also been speculations regarding direct connections
between the major pools etc - lets see if this is indeed true.

If you look instead of 360 days, only at the last 90 days the average
block size has been 131kBytes, and the fork rate has been ~1/118, which
results in a minimum fee of:

f  0.00162XBT/kb or 0.00037XBT/txn

So a small improvement but not statistically important...

Last question, recalling that optimal revenue block size is a function
of the txn-fee (from the last writeup) - lets see what fee it takes to
support a block size of 131kBytes:

S = 1/2 * (t_block/alpha - E_bounty/f)

S = 1/2 * (S/P_fork - E_bounty/f)

f = E_bounty/[(1/P_fork-2)*S] = 0.00165XBT/kB

So a 4 times increase is still sufficient for the current load.

Anyway - the all important number is alpha, the network latency which we
expect to be dependent of various things such as interconnectivity,
bandwidths, software quality etc, where mainly the latter is within our
hands to bring down the fee. And you can actually setup the standard
client to choose a better fee, as all the parameters in the formula are
easily measured!

--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Even simpler minimum fee calculation formula: f bounty*fork_rate/average_blocksize

2013-11-13 Thread Michael Gronager
Just a quick comment on the actual fees (checked at blockchain.info) the
average fee over the last 90 days is actually ~0.0003BTC/txn - so not
too far behind the theoretical minimum of 0.00037BTC/txn.

I suppose, though, that it has more to do with old clients and fee
settings (0.0005) than network wisdom ;)

On 13/11/13, 12:52 , Michael Gronager wrote:
 Last week I posted a writeup: On the optimal block size and why
 transaction fees are 8 times too low (or transactions 8 times too big).
 
 Peter Todd made some nice additions to it including different pool sizes
 into the numbers.
 
 However, it occurred to me that things can in fact be calculated even
 simpler: The measured fork rate will mean out all the different pool
 sizes and network latencies and will as such provide a simple number we
 can use to estimate the minimum fee. Key assumption is that the latency
 will depend on block size (# txns) and the fork rate will depend on latency.
 
 Using the formulas from last week:
 
 P_fork = t_propagate/t_blocks
 
 and:
 
 t_propagate = t_0 + alpha*S ~= alpha*S
 
 We get a measure for alpha as a function of the average fork rate and
 average block size:
 
 alpha = P_fork*t_block/S
 
 Further, take the formula for the minimum fee:
 
 f  alpha*E_bounty/t_block
 
 And insert the formula for alpha:
 
 f  P_fork*E_bounty/S_average
 
 Luckily the fork frequency and the average block size are easily
 measurable. blockchain.info keeps historical graphs of number of
 orphaned blocks pr day - average over the last year is 1.5. Average
 number of blocks per day over the last year is 169, which yields a
 P_fork of ~1/113. Average block size in the same time is 134kBytes,
 which yields a minimum fee:
 
 f  0.00165XBT/kb or 0.00037XBT/txn
 
 So the 0.0001 is only 4 times too small. Further, let us look at the
 trend over the last 12 months. Pieter Wuille claimed that there has been
 several improvements over the last half year that would bring down the
 latency, there has also been speculations regarding direct connections
 between the major pools etc - lets see if this is indeed true.
 
 If you look instead of 360 days, only at the last 90 days the average
 block size has been 131kBytes, and the fork rate has been ~1/118, which
 results in a minimum fee of:
 
 f  0.00162XBT/kb or 0.00037XBT/txn
 
 So a small improvement but not statistically important...
 
 Last question, recalling that optimal revenue block size is a function
 of the txn-fee (from the last writeup) - lets see what fee it takes to
 support a block size of 131kBytes:
 
 S = 1/2 * (t_block/alpha - E_bounty/f)
 
 S = 1/2 * (S/P_fork - E_bounty/f)
 
 f = E_bounty/[(1/P_fork-2)*S] = 0.00165XBT/kB
 
 So a 4 times increase is still sufficient for the current load.
 
 Anyway - the all important number is alpha, the network latency which we
 expect to be dependent of various things such as interconnectivity,
 bandwidths, software quality etc, where mainly the latter is within our
 hands to bring down the fee. And you can actually setup the standard
 client to choose a better fee, as all the parameters in the formula are
 easily measured!
 
 --
 DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
 OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
 Free app hosting. Or install the open source package on any LAMP server.
 Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
 http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
 ___
 Bitcoin-development mailing list
 Bitcoin-development@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/bitcoin-development
 


--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Even simpler minimum fee calculation formula: f bounty*fork_rate/average_blocksize

2013-11-13 Thread John Dillon
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

 Last week I posted a writeup: On the optimal block size and why
 transaction fees are 8 times too low (or transactions 8 times too big).

 Peter Todd made some nice additions to it including different pool sizes
 into the numbers.

Peter claims on IRC that he is writing a paper of some kind on this topic. I
suggest he submit it to that crypto-currency thing the foundation is
sponsoring. Given the Nov 24th deadline, I also suggest at least making part of
it public ASAP so some peer review can be done. It would be a shame for a
simple math error to cause embarassment later.


 However, it occurred to me that things can in fact be calculated even
 simpler: The measured fork rate will mean out all the different pool
 sizes and network latencies and will as such provide a simple number we
 can use to estimate the minimum fee.

Are you sure about that? You are assuming linearity where none may exist.


 Luckily the fork frequency and the average block size are easily
 measurable. blockchain.info keeps historical graphs of number of
 orphaned blocks pr day

Are those stats accurate? Have any pool operators at least confirmed that the
orphaned blocks that blockchain.info reports match their own records?

My gut feeling is to relay all orphaned blocks. We know that with a high
investment and sybil attack as blockchain.info has done you can have better
awareness of orphaned blocks than someone without those resources. If having
that awareness is ever a profitable thing we have both created an incentive to
sybil attack the network and we have linked profitability to high up-front
capital investments.

On those grounds alone I will argue that we should relay all orphans to even
the playing field. If there is a circumstance where we do not want the attacker
to have that knowledge we have failed anyway, as blockchain.info's sybil attack
on the network clearly shows.


 Anyway - the all important number is alpha, the network latency which we
 expect to be dependent of various things such as interconnectivity,
 bandwidths, software quality etc, where mainly the latter is within our
 hands to bring down the fee. And you can actually setup the standard
 client to choose a better fee, as all the parameters in the formula are
 easily measured!

With relayed orphans you could even have P2Pool enforce an optimal tx inclusion
policy based on a statistical model by including proof of those orphans into
the P2Pool share chain. P2Pool needs to take fees into account soon, but simply
asking for blocks with the highest total fees or even highest fee/kb appears to
be incomplete according to what your and Peter's analysis is suggesting.

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBCAAGBQJSg9pfAAoJEEWCsU4mNhiP5mcH/jKd2Rpl9gEJ7WhTndS5gYJ9
Ep151NyD/iKpAA4E/d9QVYalo8595LCqnrXnV6wuvuiifB6EJD5WBJq3MAMyaJLA
agl920ygY98slhDmFhnwlU9lkJVim5FoUkZgE7lQ5dr0MIhvoLQiF2Ywky49Izf0
IqL+nyW83AQweSalvktA+XGkDfGDV/EnJN7SdNqKDNtE7E9NeMl61NNOWNndsYy6
uT4PF2YB7rh8wGyHXMTC4Z192pfW4S4s60ZAflG/sTtWCcEwWi+5V/RIu0o5Hmog
RFpEPvc6d6ykdqtPfTRADMGkT2wC1yXsgeos9oFFVVuVSj8EqHb2db0B+psHRBk=
=76Qs
-END PGP SIGNATURE-

--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] Even simpler minimum fee calculation formula: f bounty*fork_rate/average_blocksize

2013-11-13 Thread Michael Gronager
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi John,

Thanks for the feedback - comments below:

 However, it occurred to me that things can in fact be calculated even
 simpler: The measured fork rate will mean out all the different pool
 sizes and network latencies and will as such provide a simple number we
 can use to estimate the minimum fee.
 
 Are you sure about that? You are assuming linearity where none may exist.

Well, my work from last week and now is a model. A model enabling you to
easily calculate the minimum fee and as a miner which transaction to
include to not shoot yourselves in the foot risking to create an
orphaned block.

The assumption that there is a linearity between block size and latency
is shown pretty well in the paper by Decker et. al (see last weeks
post). What I add this week is mainly more up to date numbers and a
formula dependent only of data that is easy to measure. (fork rate and
block size).

 
 Are those stats accurate? Have any pool operators at least confirmed that the
 orphaned blocks that blockchain.info reports match their own records?

Probably not - but the are at least a minimum - in case they are higher,
the fee should go up further.

 
 My gut feeling is to relay all orphaned blocks. We know that with a high
 investment and sybil attack as blockchain.info has done you can have better
 awareness of orphaned blocks than someone without those resources. If having
 that awareness is ever a profitable thing we have both created an incentive to
 sybil attack the network and we have linked profitability to high up-front
 capital investments.

Another way to measure latency is to setup a node that only listens but
do not relay data. By measuring the propagation of blocks of different
size as well as transactions, you can get a propagation distribution and
from that an average. However, the relevant propagation time is the one
between the pools/(single miners). Which you cannot assess using this
scheme - however, it would be nice to compare it to the orphan block scheme.

 
 With relayed orphans you could even have P2Pool enforce an optimal tx 
 inclusion
 policy based on a statistical model by including proof of those orphans into
 the P2Pool share chain. P2Pool needs to take fees into account soon, but 
 simply
 asking for blocks with the highest total fees or even highest fee/kb appears 
 to
 be incomplete according to what your and Peter's analysis is suggesting.

Indeed, and nice... But note that it is never of benefit for the miner
to include a transaction with a fee of less than ~0.0004BTC - unless it
is linked to another transaction that pay an extra fee.

There have been a lot of assumptions on the fee size and generally it
has been linked to the bitcoin exchange rate. This analysis shows that
this is wrong. Also it shows that the scalability of bitcoin is directly
linked to the network and node latency (with the current latency it will
never beneficial for miners to include more than ~30k transactions in a
block or ~70 pr second resulting in ~10MB blocks).
However, halving the latency will double the capacity, down to the
minimum which is governed by the speed of light.

 
 
 --
 DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
 OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
 Free app hosting. Or install the open source package on any LAMP server.
 Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
 http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
 ___
 Bitcoin-development mailing list
 Bitcoin-development@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/bitcoin-development
 

-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSg+H7AAoJEKpww0VFxdGRn+gIAIgju90DED5r//USqKvkQsYI
JDj0tLBLMg9BPXOOt3eJ+NX4YE4lW+QkwqDd/swuJxLmj0l9BQKgt1lTb/f0P/cY
GdE14gh5EYlvNzY1h0TGKcMe8NTWXU0/tC+Clpy4sqBHPXW/eF/77sLQUnFRrLKi
sT48aHOOFUdBLdlyylUzzevh/FFVLidkKqV031tv52+BFHcTFd4kRPwZXgBSs9YH
U66MkJ4ytAqeOfJue9n7Qn4kJF9kNIhRpqTrtapqu8jglLfuYlJ3s5fwaw9FxQdR
+On4IWeXzURQ6tcVRCovCq/2lxRKIbYGlW7HGVASjRmm68/+8YUAfFsYFl6DIgA=
=9tbL
-END PGP SIGNATURE-

--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net

Re: [Bitcoin-development] Even simpler minimum fee calculation formula: f bounty*fork_rate/average_blocksize

2013-11-13 Thread Gavin Andresen
Couple of thoughts:

RE: the marvelous coincidence that the average fee these days is very close
to the modeled minimum orphan cost:

Engineers tend to underestimate the power of markets, even inefficient
markets, to arrive at the 'correct' price. It would not surprise me at all
if the messy, chaotic inefficient market with tens of thousands of
individual decisions (which mining pool should I join and how high
should my dice site set fees and how large should the minimum payout be
and should I make my blocks bigger or smaller) might arrive at the
'correct' price, even if NOBODY involved has any clue how or why it
happened.

Or it might just be a coincidence.

RE: orphan rate:

The network-wide orphan rate has been very steady apart from the March
blockchain fork. Kudos to Ben Reeves for keeping track of the data and
giving us a nice chart:
  http://blockchain.info/charts/n-orphaned-blocks

RE: new block latency:

We should be able to reduce the size of new block announcements by about a
factor of ten with very little additional effort (transmit/relay as
merkleblock with full bloom filters-- the factor of 10 is because a
transaction id hash is 32 bytes, average transaction size is a few hundred
bytes).

Mining revenue is a fixed-size pie, so if EVERYBODY agreed to accept
(somewhat) higher orphan rates for more transaction volume then, in the
long run, there is no difference.  Well, except that more transaction
volume means more utility for Bitcoin as a whole, so everybody should
benefit from a higher bitcoin price.

That's a classic free-rider problem, though-- a miner could defect to try
to get a lower orphan rate.

This is one of the reasons why I think relaying all blocks in a race is
probably the right thing to do; if a miner is mildly punished (by losing
the occasional block race) for creating blocks that don't include enough
already-relayed transactions, that is a strong incentive to go along with
whatever consensus has been established.

The same argument applies for a miner producing too-large blocks, or blocks
with lots of transactions that were never relayed across the network.

-- 
--
Gavin Andresen
--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] [ANN] High-speed Bitcoin Relay Network

2013-11-13 Thread Matt Corallo


On 11/08/13 06:46, Mike Hearn wrote:
 I took a brief look at the code - it's looking very reasonable. You can
 replace any construct like
 
 try {
   Thread.sleep(1000);
 } catch (InterruptedException e) {
   throw new RuntimeException(e);
 }
 
 which is quite verbose, just with
 Uninterruptibles.sleepUninterruptably(1000, TimeUnit.MILLISECONDS); (and
 of course static imports help too)

Thanks, fixed.


 
 I think for this concept to take off, you'd need a website and to
 recruit someone to help you market it. Pool operators won't reach out to
 you.

Yes, I've done some initial outreach and plan on doing another major
round now that the initial network is up and Im working on running some
relay time benchmarks. Finding someone to help push peering would be
nice, if you have any suggestions, Im all ears.

 
 I still find it perhaps more elegant to just boost the connectivity of
 the existing network with bitcoind changes, but this can help for now.

Agreed, improving relay times across the regular P2P network would be
nice, however I really dont see this as a part of the P2P network. Its
more of a backup relay network that just happens to follow the P2P
protocol (mostly, it doesnt do full block verification, so technically
it breaks spec). In this model, this is really a nice augment to the P2P
network no matter what improvements are made. Having more protocols/ways
blocks are relayed is always nice (anyone wanna launch a satellite?)

Matt

--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development


Re: [Bitcoin-development] [ANN] High-speed Bitcoin Relay Network

2013-11-13 Thread Matt Corallo
In the short-term, maybe. Keep in mind that the code for tx relay is
fairly different and the bandwidth for transaction relay on these
nodes is already lower than it is for blocks (by design). That said,
I'd like to look into doing tx-less block relays for transactions that
peers already have to limit block relay times even for large blocks,
in which case tx relay is very much required.

Matt

On 11/13/13 15:13, John Dillon wrote:
 You should split the block-only and block+tx not only by port
 number, but also by DNS address. DoS attack by flooding blocks is
 fundamentally more difficult than DoS attack by flooding
 transctions, so doing the split by IP address ensures that in the
 event of an attack the more important block relaying functionality
 is less likely to be damaged. In the meantime point both DNS 
 addresses to the same IP until it becomes an issue.
 
 

--
DreamFactory - Open Source REST  JSON Services for HTML5  Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development