### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

I don't see how set reconciliation alone would be practical for condensed block exchange -- if the keys are txids it'd require a round trip to request the missing tx; if we could somehow get the What's the Difference approach to effectively operate on full transactions instead, the log(keysize)

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 31, 2014 at 1:47 PM, Kaz Wesley kezi...@gmail.com wrote: trip to request the missing tx; if we could somehow get the What's the Difference approach to effectively operate on full transactions instead I explain how to do this on the network block coding page. Given that you know

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

the need to have transmitted the transaction list [..] first 32 bits per transaction is at least double the communication overhead of the simple approach, and only offers a bound on the probability of needing a round trip. On Thu, Jul 31, 2014 at 2:29 PM, Gregory Maxwell gmaxw...@gmail.com

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 31, 2014 at 2:41 PM, Kaz Wesley kezi...@gmail.com wrote: the need to have transmitted the transaction list [..] first 32 bits per transaction is at least double the communication overhead of the simple approach, and only offers a bound on the probability of needing a round trip.

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

the FEC still lets you fill in the missing transactions without knowing in advance all that will be missing. I don't see why we need to solve that problem, since the protocol already involves exchanging the information necessary to determine (with some false positives) what a peer is missing,

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley kezi...@gmail.com wrote: the FEC still lets you fill in the missing transactions without knowing in advance all that will be missing. I don't see why we need to solve that problem, since the protocol already involves exchanging the information

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 31, 2014 at 4:18 PM, Gregory Maxwell gmaxw...@gmail.com wrote: On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley kezi...@gmail.com wrote: the FEC still lets you fill in the missing transactions without knowing in advance all that will be missing. I don't see why we need to solve that

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Fri, Jul 18, 2014 at 8:06 PM, Emin Gün Sirer el33th4...@gmail.com wrote: Most things I've seen working in this space are attempting to minimize the data transfered. At least for the miner-interested case the round complexity is much more important because a single RTT is enough to

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Fri, Jul 18, 2014 at 4:53 PM, Gavin Andresen gavinandre...@gmail.com wrote: Two more half-baked thoughts: We should be able to assume that the majority of transaction data (except for coinbase) has already been propagated. As Jeff said, incentivizing nodes to propagate transactions is a

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 17, 2014 at 6:46 PM, Gavin Andresen gavinandre...@gmail.com wrote: I'd encourage you to code up a prototype first (or at the same time), in whatever programming language / networking library you're most familiar with. +1 Maybe not even using the existing p2p protocol; there could

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

Two more half-baked thoughts: We should be able to assume that the majority of transaction data (except for coinbase) has already been propagated. As Jeff said, incentivizing nodes to propagate transactions is a very good thing (the signature cache already gives a small incentive to miners to

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Fri, Jul 18, 2014 at 10:53 AM, Gavin Andresen gavinandre...@gmail.com wrote: But if there was some agreed-upon canonical ordering, then it should theoretically be possible to take shortcuts in the what order. You'd start with setof(transactions I think everybody knows about) Select some

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

Peers exchanging mempool priority policies is great; that accomplishes the flexibility in what txes to remember that I was going for with the forget-filters, but much more neatly, with less overhead and some side benefits. Here's what I'm picturing now: - exchange priority policies in peer

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On a flood-fill network, you don't want to create a storm of I already have this replies. On Fri, Jul 18, 2014 at 1:39 PM, Kaz Wesley kezi...@gmail.com wrote: Peers exchanging mempool priority policies is great; that accomplishes the flexibility in what txes to remember that I was going for

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

I thought I'd chime in and point out some research results that might help. Even if they don't, there is a cool underlying technique that some of you might find interesting. The problem being tackled here is very similar to set reconciliation, where peer A thinks that the set of transactions that

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Fri, Jul 18, 2014 at 5:54 PM, Emin Gün Sirer el33th4...@gmail.com wrote: The problem being tackled here is very similar to set reconciliation, where peer A thinks that the set of transactions that should be in the block is S_A, Most things I've seen working in this space are attempting to

### [Bitcoin-development] Squashing redundant tx data in blocks on the wire

OVERVIEW To improve block propagation, add a new block message that doesn't include transactions the peer is known to have. The message must never require an additional round trip due to any transactions the peer doesn't have, but should be compatible with peers sometimes forgetting transactions

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

A couple of half-baked thoughts: On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley kezi...@gmail.com wrote: If there's support for this proposal, I can begin working on the specific implementation details, such as the bloom filters, message format, and capability advertisment, and draft a BIP once

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

I'm moving this design document to a gist so that I can integrate changes as they come up: https://gist.github.com/kazcw/43c97d3924326beca87d One thing that I think is an important improvement over my initial idea is that the bloom filters don't need to be kept around and built up, they can just

### Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

On Thu, Jul 17, 2014 at 2:35 PM, Kaz Wesley kezi...@gmail.com wrote: A node should be able to forget invs it has seen without invalidating what peers know about its known txes. To allow for this, a node assembles a bloom filter of Another option would be to just guarantee to keep at least the