Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
On Thu, Jul 31, 2014 at 4:18 PM, Gregory Maxwell wrote: > On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley wrote: >>> the FEC still lets you fill in the missing transactions without knowing in >>> advance all that will be missing. >> >> I don't see why we need to solve that problem, since the protoco

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley wrote: >> the FEC still lets you fill in the missing transactions without knowing in >> advance all that will be missing. > > I don't see why we need to solve that problem, since the protocol > already involves exchanging the information necessary to de

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
> the FEC still lets you fill in the missing transactions without knowing in > advance all that will be missing. I don't see why we need to solve that problem, since the protocol already involves exchanging the information necessary to determine (with some false positives) what a peer is missing,

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 2:41 PM, Kaz Wesley wrote: >> the need to have transmitted the transaction list [..] first > > 32 bits per transaction is at least double the communication overhead > of the simple approach, and only offers a bound on the probability of > needing a round trip. "(e.g. from

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
> the need to have transmitted the transaction list [..] first 32 bits per transaction is at least double the communication overhead of the simple approach, and only offers a bound on the probability of needing a round trip. On Thu, Jul 31, 2014 at 2:29 PM, Gregory Maxwell wrote: > On Thu, Jul 3

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 1:47 PM, Kaz Wesley wrote: > trip to request the missing tx; if we could somehow get the "What's > the Difference" approach to effectively operate on full transactions > instead I explain how to do this on the network block coding page. Given that you know the sizes and o

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
I don't see how set reconciliation alone would be practical for condensed block exchange -- if the keys are txids it'd require a round trip to request the missing tx; if we could somehow get the "What's the Difference" approach to effectively operate on full transactions instead, the log(keysize) f

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-19 Thread Wladimir
On Fri, Jul 18, 2014 at 4:53 PM, Gavin Andresen wrote: > Two more half-baked thoughts: > > We should be able to assume that the majority of transaction data (except > for coinbase) has already been propagated. As Jeff said, incentivizing nodes > to propagate transactions is a very good thing (the

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gregory Maxwell
On Fri, Jul 18, 2014 at 8:06 PM, Emin Gün Sirer wrote: > >> Most things I've seen working in this space are attempting to minimize >> the data transfered. At least for the miner-interested case the round >> complexity is much more important because a single RTT is enough to >> basically send the w

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Emin Gün Sirer
> Most things I've seen working in this space are attempting to minimize > the data transfered. At least for the miner-interested case the round > complexity is much more important because a single RTT is enough to > basically send the whole block on a lot of very relevant paths. Agreed. Yaron's s

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gregory Maxwell
On Fri, Jul 18, 2014 at 5:54 PM, Emin Gün Sirer wrote: > The problem being tackled here is very similar to "set reconciliation," > where > peer A thinks that the set of transactions that should be in the block is > S_A, Most things I've seen working in this space are attempting to minimize the da

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Emin Gün Sirer
I thought I'd chime in and point out some research results that might help. Even if they don't, there is a cool underlying technique that some of you might find interesting. The problem being tackled here is very similar to "set reconciliation," where peer A thinks that the set of transactions tha

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
Related: We must handle some legitimate miner-privately-mined cases, such as miner payout TXs (outside coinbase) or side chain conditional TXs[1]. [1] https://bitcointalk.org/index.php?topic=676703.msg7682680#msg7682680 On Fri, Jul 18, 2014 at 3:51 PM, Kaz Wesley wrote: > I've updated the gist,

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
I've updated the gist, and added an additional proposal that I think meshes well: https://gist.github.com/kazcw/43c97d3924326beca87d#ultra-fast-block-validation sparseblocks + UFBV would tighten the new-block process to this (when txes have been received in advance): - receive block (~2kB for 1000

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
That's true, but I think it can be balanced with the usefulness of knowing what messages a node has received. An invack would be sent if N invs have been received without any resulting getdata; since we're keeping track of peer inv order, one message can cover an arbitrarily large series of invs.

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On a flood-fill network, you don't want to create a storm of "I already have this" replies. On Fri, Jul 18, 2014 at 1:39 PM, Kaz Wesley wrote: > Peers exchanging mempool priority policies is great; that accomplishes > the flexibility in what txes to remember that I was going for with the > forget

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
Peers exchanging mempool priority policies is great; that accomplishes the flexibility in what txes to remember that I was going for with the forget-filters, but much more neatly, with less overhead and some side benefits. Here's what I'm picturing now: - exchange priority policies in peer introdu

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On Fri, Jul 18, 2014 at 10:53 AM, Gavin Andresen wrote: > But if there was some agreed-upon canonical ordering, then it should > theoretically be possible to take shortcuts in the "what order". > > You'd start with setof(transactions I think everybody knows about) > Select some subset, based on mi

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gavin Andresen
Two more half-baked thoughts: We should be able to assume that the majority of transaction data (except for coinbase) has already been propagated. As Jeff said, incentivizing nodes to propagate transactions is a very good thing (the signature cache already gives a small incentive to miners to prop

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On Thu, Jul 17, 2014 at 6:46 PM, Gavin Andresen wrote: > I'd encourage you to code up a prototype first (or at the same time), in > whatever programming language / networking library you're most familiar > with. +1 > Maybe not even using the existing p2p protocol; there could be a mining-only >

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gregory Maxwell
On Thu, Jul 17, 2014 at 2:35 PM, Kaz Wesley wrote: > A node should be able to forget invs it has seen without invalidating what > peers > know about its known txes. To allow for this, a node assembles a bloom > filter of Another option would be to just guarantee to keep at least the last N sent i

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Kaz Wesley
I'm moving this design document to a gist so that I can integrate changes as they come up: https://gist.github.com/kazcw/43c97d3924326beca87d One thing that I think is an important improvement over my initial idea is that the bloom filters don't need to be kept around and built up, they can just be

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gavin Andresen
A couple of half-baked thoughts: On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley wrote: > If there's support for this proposal, I can begin working on the specific > implementation details, such as the bloom filters, message format, and > capability advertisment, and draft a BIP once I have a concre

[Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Kaz Wesley
OVERVIEW To improve block propagation, add a new block message that doesn't include transactions the peer is known to have. The message must never require an additional round trip due to any transactions the peer doesn't have, but should be compatible with peers sometimes forgetting transactions t