Re: [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's"

[Peter Tschipper via bitcoin-dev]

Hi All,

Here are some final results of testing with the reference implementation
for compressing blocks and transactions. This implementation also
concatenates blocks and transactions when possible so you'll see data
sizes in the 1-2MB ranges.

Results below show the time it takes to sync the first part of the
blockchain, comparing Zlib to the LZOx library.  (LZOf was also tried
but wasn't found to be as good as LZOx).  The following shows tests run
with and without latency.  With latency on the network, all compression
libraries performed much better than without compression.

I don't think it's entirely obvious which is better, Zlib or LZO. 
Although I prefer the higher compression of Zlib, overall I would have
to give the edge to LZO.  With LZO we have the fastest most scalable
option when at the lowest compression setting which will be a boost in
performance for users that want peformance over compression, and then at
the high end LZO provides decent compression which approaches Zlib,
(although at a higher cost) but good for those that want to save more
bandwidth.

Uncompressed 60ms   Zlib-1 (60ms)   Zlib-6 (60ms)   LZOx-1 (60ms)   LZOx-999
(60ms)
219 299 296 294 291
432 568 565 558 548
652 835 836 819 811
866 1106110710811071
10821372138113411333
13091644165416051600
15351917193618731875
17622191221021412141
19922463248624112411
22572748278026942697
26273034307629702983
32263416339732663302
40103983377336253703
49144503429241274287
58064928471945294821
66745249516448405314
75635603566952896002
84776054626858586638
98437085727868687679
11338   8215843380448795



These results from testing on a highspeed wireless LAN (very small latency)

Results in seconds  




Num blocks sync'd   UncompressedZlib-1  Zlib-6  LZOx-1  LZOx-999
1   255 232 233 231 257
2   464 414 420 407 453
3   677 594 611 585 650
4   887 782 795 760 849
5   1099961 977 933 1048
6   13101145116711101259
7   15121330136212911470
8   17141519155214691679
9   19171707174716501882
10  21221905195018432111
11  23332107215120382329
12  25602333237622562580
13  28352656267925582921
14  32743259316130513466
15  36623793354734403919
16  40404172393737674416
17  44254625437942154958
18  48605149489547815560
19  58556160589858056557
20  70047234705169837770



The following show the compression ratio acheived for various sizes of
data.  Zlib is the clear
winner for compressibility, with LZOx-999 coming close but at a cost.

range   Zlib-1 cmp%
Zlib-6 cmp% LZOx-1 cmp% LZOx-999 cmp%
0-250b  12.44   12.86   10.79   14.34
250-500b19.33   12.97   10.34   11.11
600-700 16.72   n/a 12.91   17.25
700-800 6.377.654.838.07
900-1KB 6.546.955.647.9
1KB-10KB25.08   25.65   21.21   22.65
10KB-100KB  19.77   21.57   14.37   19.02
100KB-200KB 21.49   23.56   15.37   21.55
200KB-300KB 23.66   24.18   16.91   22.76
300KB-400KB 23.423.716.521.38
400KB-500KB 24.624.85   17.56   22.43
500KB-600KB 25.51   26.55   18.51   23.4
600KB-700KB 27.25   28.41   19.91   25.46
700KB-800KB 27.58   29.18   20.26   27.17
800KB-900KB 27  29.11   20  27.4
900KB-1MB   28.19   29.38   21.15   26.43
1MB -2MB27.41   29.46   21.33   27.73


The following shows the time in seconds to compress data of various
sizes.  LZO1x is the
fastest and as file sizes increase, LZO1x time hardly increases at all. 
It's interesing
to note as compression ratios increase LZOx-999 performs much worse than
Zlib.  So LZO is faster
on the low end and slower (5 to 6 times slower) on the high end.

range   Zlib-1  Zlib-6  LZOx-1  LZOx-999 cmp%
0-250b  0.001   0   0   0
250-500b0   0   0   0.001
500-1KB 0   0   0   0.001
1KB-10KB0.001   0.001   0   0.002
10KB-100KB  0.004   0.006   0.001   0.017
100KB-200KB 0.012   0.017   0.002   0.054
200KB-300KB 0.018   0.024   0.003   0.087
300KB-400KB 0.022   0.030.003   0.121
400KB-500KB 0.027   0.037   0.004   0.151
500KB-600KB 0.031   0.044   0.004   0.184
600KB-700KB 0.035   0.051   0.006   0.211
700KB-800KB 0.039   0.057   0.006   0.243
800KB-900KB 0.045   

[bitcoin-dev] Test Results for : Datasstream Compression of Blocks and Tx's

[Peter Tschipper via bitcoin-dev]

Hi All,

Here are some final results of testing with the reference implementation
for compressing blocks and transactions. This implementation also
concatenates blocks and transactions when possible so you'll see data
sizes in the 1-2MB ranges.

Results below show the time it takes to sync the first part of the
blockchain, comparing Zlib to the LZOx library.  (LZOf was also tried
but wasn't found to be as good as LZOx).  The following shows tests run
with and without latency.  With latency on the network, all compression
libraries performed much better than without compression.

I don't think it's entirely obvious which is better, Zlib or LZO. 
Although I prefer the higher compression of Zlib, overall I would have
to give the edge to LZO.  With LZO we have the fastest most scalable
option when at the lowest compression setting which will be a boost in
performance for users that want peformance over compression, and then at
the high end LZO provides decent compression which approaches Zlib,
(although at a higher cost) but good for those that want to save more
bandwidth.

Uncompressed 60ms   Zlib-1 (60ms)   Zlib-6 (60ms)   LZOx-1 (60ms)   LZOx-999
(60ms)
219 299 296 294 291
432 568 565 558 548
652 835 836 819 811
866 1106110710811071
10821372138113411333
13091644165416051600
15351917193618731875
17622191221021412141
19922463248624112411
22572748278026942697
26273034307629702983
32263416339732663302
40103983377336253703
49144503429241274287
58064928471945294821
66745249516448405314
75635603566952896002
84776054626858586638
98437085727868687679
11338   8215843380448795



These results from testing on a highspeed wireless LAN (very small latency)

Results in seconds  




Num blocks sync'd   UncompressedZlib-1  Zlib-6  LZOx-1  LZOx-999
1   255 232 233 231 257
2   464 414 420 407 453
3   677 594 611 585 650
4   887 782 795 760 849
5   1099961 977 933 1048
6   13101145116711101259
7   15121330136212911470
8   17141519155214691679
9   19171707174716501882
10  21221905195018432111
11  23332107215120382329
12  25602333237622562580
13  28352656267925582921
14  32743259316130513466
15  36623793354734403919
16  40404172393737674416
17  44254625437942154958
18  48605149489547815560
19  58556160589858056557
20  70047234705169837770



The following show the compression ratio acheived for various sizes of
data.  Zlib is the clear
winner for compressibility, with LZOx-999 coming close but at a cost.

range   Zlib-1 cmp%
Zlib-6 cmp% LZOx-1 cmp% LZOx-999 cmp%
0-250b  12.44   12.86   10.79   14.34
250-500b19.33   12.97   10.34   11.11
600-700 16.72   n/a 12.91   17.25
700-800 6.377.654.838.07
900-1KB 6.546.955.647.9
1KB-10KB25.08   25.65   21.21   22.65
10KB-100KB  19.77   21.57   14.37   19.02
100KB-200KB 21.49   23.56   15.37   21.55
200KB-300KB 23.66   24.18   16.91   22.76
300KB-400KB 23.423.716.521.38
400KB-500KB 24.624.85   17.56   22.43
500KB-600KB 25.51   26.55   18.51   23.4
600KB-700KB 27.25   28.41   19.91   25.46
700KB-800KB 27.58   29.18   20.26   27.17
800KB-900KB 27  29.11   20  27.4
900KB-1MB   28.19   29.38   21.15   26.43
1MB -2MB27.41   29.46   21.33   27.73


The following shows the time in seconds to compress data of various
sizes.  LZO1x is the
fastest and as file sizes increase, LZO1x time hardly increases at all. 
It's interesing
to note as compression ratios increase LZOx-999 performs much worse than
Zlib.  So LZO is faster
on the low end and slower (5 to 6 times slower) on the high end.

range   Zlib-1  Zlib-6  LZOx-1  LZOx-999 cmp%
0-250b  0.001   0   0   0
250-500b0   0   0   0.001
500-1KB 0   0   0   0.001
1KB-10KB0.001   0.001   0   0.002
10KB-100KB  0.004   0.006   0.001   0.017
100KB-200KB 0.012   0.017   0.002   0.054
200KB-300KB 0.018   0.024   0.003   0.087
300KB-400KB 0.022   0.030.003   0.121
400KB-500KB 0.027   0.037   0.004   0.151
500KB-600KB 0.031   0.044   0.004   0.184
600KB-700KB 0.035   0.051   0.006   0.211
700KB-800KB 0.039   0.057   0.006   0.243
800KB-900KB 0.045   

[bitcoin-dev] Use CPFP as consensus critical for Full-RBF

[Vincent Truong via bitcoin-dev]

(I haven't been following this development recently so apologies in advance
if I've made assumptions about RBF)

If you made CPFP consensus critical for all Full-RBF transactions, RBF
should be safer to use. I see RBF as a necessity for users to fix mistakes
(and not for transaction prioritisation), but we can't know for sure if
miners are playing with this policy fairly or not. It is hard to spot a
legitimate RBF and a malicious one, but if the recipient signs off on the
one they know about using CPFP, there should be no problems. This might
depend on the CPFP implementation, because you'll need a way for the
transaction to mark which output is a change address and which is a payment
to prevent the sender from signing off his own txns. (This might be bad for
privacy, but IMO a lot safer than allowing RBF double spending sprees... If
you value privacy then don't use RBF?) Or maybe let them sign it off but
make all outputs sign off somehow.

Copy/Paste from my reddit post:

https://www.reddit.com/r/Bitcoin/comments/3ul1kb/slug/cxgegkj

Going to chime in my opinion: opt-in RBF eliminates the trust required with
miners. You don't know if they're secretly running RBF right now anyway.
Whether Peter Todd invented this is irrelevant, it was going to happen
either way either with good intentions or with malice, so better to develop
this with good intentions.

Perhaps the solution to this problem is simple. Allow Full-RBF up to the
point where a recipient creates a CPFP transaction. Any transaction with
full RBF that hasn't been signed off with a CPFP cannot go into a block,
and this can become a consensus rule rather than local policy thanks to the
opt-in flags that's inside transactions.

> P.S. (When I wrote this, I'm actually not sure how the flag looks like
and am just guessing it can be used this way. I'm not familiar with the
implementation.)

CPFP is needed so that merchants can bear the burden of fees (double
bandwidth costs aside, and frankly if RBF is allowed bandwidth is going to
increase regardless anyway). That's always the way I've being seeing its
purpose. And this makes RBF much safer to use by combining the two.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's"

[Jonathan Toomim via bitcoin-dev]

It appears you're using the term "compression ratio" to mean "size reduction". 
A compression ratio is the ratio (compressed / uncompressed). A 1 kB file 
compressed with a 10% compression ratio would be 0.1 kB. It seems you're using 
(1 - compressed/uncompressed), meaning that the compressed file would be 0.9 kB.

On Nov 28, 2015, at 6:48 AM, Peter Tschipper via bitcoin-dev 
 wrote:

> The following show the compression ratio acheived for various sizes of data.  
> Zlib is the clear
> winner for compressibility, with LZOx-999 coming close but at a cost.
> 
> range Zlib-1 cmp%
> Zlib-6 cmp%   LZOx-1 cmp% LZOx-999 cmp%
> 0-250b12.44   12.86   10.79   14.34
> 250-500b  19.33   12.97   10.34   11.11
> 
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's"

[Peter Tschipper via bitcoin-dev]

yes, you're right, it's just the percentage compressed (size reduction)

On 28/11/2015 4:30 PM, Jonathan Toomim wrote:
> It appears you're using the term "compression ratio" to mean "size
> reduction". A compression ratio is the ratio (compressed /
> uncompressed). A 1 kB file compressed with a 10% compression ratio
> would be 0.1 kB. It seems you're using (1 - compressed/uncompressed),
> meaning that the compressed file would be 0.9 kB.
>
> On Nov 28, 2015, at 6:48 AM, Peter Tschipper via bitcoin-dev
>  > wrote:
>
>> The following show the compression ratio acheived for various sizes
>> of data.  Zlib is the clear
>> winner for compressibility, with LZOx-999 coming close but at a cost.
>>
>> rangeZlib-1 cmp%
>>  Zlib-6 cmp% LZOx-1 cmp% LZOx-999 cmp%
>> 0-250b   12.44   12.86   10.79   14.34
>> 250-500b 19.33   12.97   10.34   11.11
>>
>>  
>>  
>>  
>>  
>>
>

___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev