Re: [bitcoin-dev] Low-bandwidth transaction relay

2018-04-04 Thread Gleb Naumenko via bitcoin-dev 
Thanks for the links!

Blocksonly is definitely a relevant piece. However, I’m wondering what are the 
implications, especially at larger scale. For example, transactions processing 
will be not smooth anymore and will happen every 10 minutes at once. Another 
question is transaction propagation.

I think what I’ve proposed does not have those implications. Well, propagation 
is still a concern, but it’s not that extreme. One weakness of my idea is 
relative complexity comparing to blocksonly.

Another variation of the idea I described might work without INVs at all  (then 
N=1 and transactions are relayed through 1 link only, during the time between 
blocks) and it would have the same security assumptions as blocksonly.

Your IBLT and BCH-sets proposals sound very promising. I had something like 
that on mind, but I decided to start with a more conservative protocol.
It looks like sync-relay idea has a lot of interesting questions, I’m excited 
to follow that research.

On Apr 3, 2018, 12:04 PM -0700, Gregory Maxwell , wrote:
> On Mon, Apr 2, 2018 at 10:18 PM, Gleb Naumenko via bitcoin-dev
>  wrote:
> > Hi all,
> > I have a couple of ideas regarding transaction relay protocol and wanted to
> > share it with and probably get some feedback.
>
>
>
> https://bitcointalk.org/index.php?topic=1377345.0
>
> https://people.xiph.org/~greg/mempool_sync_relay.txt
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Low-bandwidth transaction relay

2018-04-03 Thread Gleb Naumenko via bitcoin-dev 
Yeah, sure.

> How much bandwidth is consumed by redundant tx INVs currently?
Currently, for an average public-IP node all INVs consume 0.05 Mbps or 540 
megabytes per day. This number is based on current ratio public-IP 
nodes:private-IP nodes and transaction rate. This number is a sum of both 
incoming and outgoing aspects. Thus redundant INV’s on average consume 0.044 
Mbps or 475 megabytes per day.

> What is this as a % of overall bandwidth usage?
This is hard to estimate because overall bandwidth includes helping other nodes 
to bootstrap from scratch. If we don’t consider this aspect, my very rough 
estimate, and a short experiment shows that INV’s are around 50% of overall 
bandwidth (it also depends on different factors like your hardware comparing to 
other public-IP nodes). I’m going to double-check this number soon.

> How would filtering txs through N=2 links affect network propagation?
Yes, network propagation for a new protocol definitely worth measuring. I’m 
going to look at it in the near future.

> Do you propose setting filters on inbound peers as well?
This is a good question.
I think some filter may be applied to inbound connections. Theoretically, a 
symmetrical filter does not make much sense — it might be eventually the same 
filter for all of the connections except first 8 outgoing ones, so it’s better 
to use independent filters.
However, I’m not entirely sure it is needed. Filters on inbound peers will 
reduce a download aspect. It might be much less critical than upload (if we 
assume that private-IP nodes hear about transactions later because those have 
much fewer connections). I think this question needs another experiment.

On Apr 3, 2018, 10:45 AM -0700, Jim Posen , wrote:
> Hey. This idea sounds quite interesting. It'd be helpful to see some more 
> numbers to evaluate it.
>
> - How much bandwidth is consumed by redundant tx INVs currently? What is this 
> as a % of overall bandwidth usage?
> - How would filtering txs through N=2 links affect network propagation? This 
> probably requires simulation to determine.
> - Do you propose setting filters on inbound peers as well?
>
> > On Mon, Apr 2, 2018 at 3:18 PM, Gleb Naumenko via bitcoin-dev 
> >  wrote:
> > > Hi all,
> > > I have a couple of ideas regarding transaction relay protocol and wanted 
> > > to share it with and probably get some feedback.
> > >
> > > I did some emulation and simulation and found out that around 90% of INV 
> > > messages sent by public-IP nodes are idle (duplicate), obviously because 
> > > each node creates 8 connections.  I also realized that sending INV 
> > > messages is a significant part of the overall bandwidth consumed by a 
> > > public-IP node. At a larger scale, this will result in people not able to 
> > > run a public-IP node.
> > >
> > > My idea is in some sense similar to BIP37 but applied to public-IP nodes. 
> > > Here I want to emphasize that all the nodes will still receive *all* of 
> > > the transactions. A new protocol should also keep the same zero-trust, 
> > > robustness, decentralization guarantees and latency.
> > >
> > > Idea: while joining the network, a new node agrees on some filter with 
> > > each of 8 nodes it connects to. So that NewNode <-> Node_A will be used 
> > > to relay only a subset of transactions, NewNode <-> Node_B for another 
> > > subset. This will significantly decrease the redundancy.
> > >
> > > To keep the guarantees, I would keep some redundancy (for example, each 
> > > transaction INV is sent over 2 links).
> > >
> > > To make it robust to attacks, I have 2 extensions in my mind:
> > > 1. Set reconciliation (for a subset of transactions) with *other* nodes. 
> > > Getting a bloom filter of a subset of the mempool transactions from 
> > > Node_B may help to figure out whether Node_A is malicious, very slow, etc.
> > > 2. Rotating the filters every N minutes (N < 10)
> > >
> > > I can see some issues with latency here, but I believe this problem has a 
> > > solution.
> > >
> > > Feedback is appreciated!
> > >
> > > If you want to look at a draft of the proposal — please let me know.
> > > If there were any similar ideas — please let me know.
> > >
> > > Best,
> > > Gleb
> > >
> > >
> > > ___
> > > bitcoin-dev mailing list
> > > bitcoin-dev@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> > >
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Low-bandwidth transaction relay

2018-04-03 Thread Gregory Maxwell via bitcoin-dev 
On Mon, Apr 2, 2018 at 10:18 PM, Gleb Naumenko via bitcoin-dev
 wrote:
> Hi all,
> I have a couple of ideas regarding transaction relay protocol and wanted to
> share it with and probably get some feedback.

https://bitcointalk.org/index.php?topic=1377345.0

https://people.xiph.org/~greg/mempool_sync_relay.txt
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Low-bandwidth transaction relay

2018-04-03 Thread Jim Posen via bitcoin-dev 
Hey. This idea sounds quite interesting. It'd be helpful to see some more
numbers to evaluate it.

- How much bandwidth is consumed by redundant tx INVs currently? What is
this as a % of overall bandwidth usage?
- How would filtering txs through N=2 links affect network propagation?
This probably requires simulation to determine.
- Do you propose setting filters on inbound peers as well?

On Mon, Apr 2, 2018 at 3:18 PM, Gleb Naumenko via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi all,
> I have a couple of ideas regarding transaction relay protocol and wanted
> to share it with and probably get some feedback.
>
> I did some emulation and simulation and found out that around 90% of INV
> messages sent by public-IP nodes are idle (duplicate), obviously because
> each node creates 8 connections.  I also realized that sending INV messages
> is a significant part of the overall bandwidth consumed by a public-IP
> node. At a larger scale, this will result in people not able to run a
> public-IP node.
>
> My idea is in some sense similar to BIP37 but applied to public-IP nodes.
> Here I want to emphasize that all the nodes will still receive *all* of the
> transactions. A new protocol should also keep the same zero-trust,
> robustness, decentralization guarantees and latency.
>
> Idea: while joining the network, a new node agrees on some filter with
> each of 8 nodes it connects to. So that NewNode <-> Node_A will be used to
> relay only a subset of transactions, NewNode <-> Node_B for another subset.
> This will significantly decrease the redundancy.
>
> To keep the guarantees, I would keep some redundancy (for example, each
> transaction INV is sent over 2 links).
>
> To make it robust to attacks, I have 2 extensions in my mind:
> 1. Set reconciliation (for a subset of transactions) with *other* nodes.
> Getting a bloom filter of a subset of the mempool transactions from Node_B
> may help to figure out whether Node_A is malicious, very slow, etc.
> 2. Rotating the filters every N minutes (N < 10)
>
> I can see some issues with latency here, but I believe this problem has a
> solution.
>
> Feedback is appreciated!
>
> If you want to look at a draft of the proposal — please let me know.
> If there were any similar ideas — please let me know.
>
> Best,
> Gleb
>
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev