Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
Building a compressor from scratch may yeild some better compression ratios, or not, but having trust and faith in whether it will stand up against attack vectors another matter. LZO has been around for 20 years with very few problems and no current issues. Maybe something better can be built,

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
ecause we're > definitely gonna have issues. It's not that difficult to add compression. Even if there was an issue, the compression feature can be completely turned off. > > On 12/02/15 20:16, Peter Tschipper via bitcoin-dev wrote: >> Building a compressor from scratch may ye

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
On 30/11/2015 9:28 PM, Matt Corallo wrote: > I'm really not a fan of this at all. To start with, adding a compression > library that is directly accessible to the network on financial software is a > really, really scary idea. Why scary? LZO has no current security issues, and it will be

[bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-11-30 Thread Peter Tschipper via bitcoin-dev
@gmaxwell Bip Editor, and the Bitcoin Dev Community, After several weeks of experimenting and testing with various compression libraries I think there is enough evidence to show that compressing blocks and transactions is not only beneficial in reducing network bandwidth but is also provides a

Re: [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's"

2015-11-28 Thread Peter Tschipper via bitcoin-dev
Hi All, Here are some final results of testing with the reference implementation for compressing blocks and transactions. This implementation also concatenates blocks and transactions when possible so you'll see data sizes in the 1-2MB ranges. Results below show the time it takes to sync the

[bitcoin-dev] Test Results for : Datasstream Compression of Blocks and Tx's

2015-11-28 Thread Peter Tschipper via bitcoin-dev
Hi All, Here are some final results of testing with the reference implementation for compressing blocks and transactions. This implementation also concatenates blocks and transactions when possible so you'll see data sizes in the 1-2MB ranges. Results below show the time it takes to sync the

Re: [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's"

2015-11-28 Thread Peter Tschipper via bitcoin-dev
kB file compressed with a 10% compression ratio > would be 0.1 kB. It seems you're using (1 - compressed/uncompressed), > meaning that the compressed file would be 0.9 kB. > > On Nov 28, 2015, at 6:48 AM, Peter Tschipper via bitcoin-dev > <bitcoin-dev@lists.linuxfoundation.org

[bitcoin-dev] More findings: Block Compression (Datastream Compression) test results using the PR#6973 compression prototype

2015-11-18 Thread Peter Tschipper via bitcoin-dev
Hi all, I'm still doing a little more investigation before opening up a formal bip PR, but getting close. Here are some more findings. After moving the compression from main.cpp to streams.h (CDataStream) it was a simple matter to add compression to transactions as well. Results as follows:

[bitcoin-dev] request to use service bit 28 for testing

2015-11-14 Thread Peter Tschipper via bitcoin-dev
I'd like to use service bit 28 for testing the block compression prototype unless anyone has any objections or is using it already. Thanks. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org

[bitcoin-dev] Block Compression (Datastream Compression) test results using the PR#6973 compression prototype

2015-11-13 Thread Peter Tschipper via bitcoin-dev
Some further Block Compression tests results that compare performance when network latency is added to the mix. Running two nodes, windows 7, compressionlevel=6, syncing the first 20 blocks from one node to another. Running on a highspeed wireless LAN with no connections to the outside

Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

2015-11-11 Thread Peter Tschipper via bitcoin-dev
Here are the latest results on compression ratios for the first 295,000 blocks, compressionlevel=6. I think there are more than enough datapoints for statistical significance. Results are very much similar to the previous test. I'll work on getting a comparison between how much time

Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

2015-11-11 Thread Peter Tschipper via bitcoin-dev
r that. > > Something that could take advantage of of special knowledge of the > specific data, instead, would be an entirely different matter. > > Just my 2c. > > On Wed, Nov 11, 2015 at 7:35 PM, Peter Tschipper via bitcoin-dev > <bitcoin-dev@lists.linuxfoundation.org

Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

2015-11-10 Thread Peter Tschipper via bitcoin-dev
On 10/11/2015 8:11 AM, Peter Tschipper wrote: > On 10/11/2015 1:44 AM, Tier Nolan via bitcoin-dev wrote: >> The network protocol is not quite consensus critical, but it is >> important. >> >> Two implementations of the decompressor might not be bug for bug >> compatible. This (potentially) means

Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

2015-11-10 Thread Peter Tschipper via bitcoin-dev
On 10/11/2015 8:46 AM, Jeff Garzik via bitcoin-dev wrote: > Comments: > > 1) cblock seems a reasonable way to extend the protocol. Further > wrapping should probably be done at the stream level. agreed. > > 2) zlib has crappy security track record. > Zlib had a bad buffer overflow bug but that