Hello, Igniters! I implement network compression as described earlier [1]. I implemented Zstd, LZ4, and Deflater [2, 3]. LZ4 have good speed (1.2 G/s per connection) but compression ratio is not more than 2. Zstd has good compression ratio, up to 5, but relatively slow throughputs (0.28 G/s per connection). In addition, we boost SSL throughput from 0.8 G/s up to 1.0 G/s, because we encrypt fewer data.
Alexander (sharple...@gmail.com) has developed benchmarks for a network. They measure effective and real throughput, compression ratio, and latency. The test generates data with a structure similar to natural language. At first, generated "words" of lengths from the Poisson distribution. Each word is a byte array filled randomly. These "words" form a "language" that is being added to the buffer by Zipf's law distribution until the buffer is full. In case SSL is turned on results follow: On 500kb messages, Zstd shows best results per thread (Real throughput 0.202 G/s, compression ratio 5), LZ4 is second (Real throughput 0.498 G/s, compression ratio 1.92). A base throughput of SSL is 0.76 G/s. Enabling Zstd compression boosts effective throughput up to 30%. LZ4 - to 26%. On 1kb messages, Zstd shows best results per thread (Real throughput 0.042 G/s, compression ratio 2.5), LZ4 is second (Real throughput 0.065 G/s, compression ratio 1.74). A real throughput of SSL is 0.107 G/s. Enabling LZ4 compression boosts effective throughput up to 6%. Zstd only compresses the data. These are good results even if a network is not fully utilized. Throughput can be boosted up to several times if network speed is limited. I wrote the result into comments of the issue [1]. Now I cover code with tests. Any thoughts? 1. https://issues.apache.org/jira/browse/IGNITE-7024 2. https://reviews.ignite.apache.org/ignite/review/IGNT-CR-467 3. https://github.com/apache/ignite/pull/3101/files 2017-11-30 13:00 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>: > I would start with communication only, and put discovery and clients aside > for now. Let's confirm that communication works well with compression first > in terms of performance. > > On Thu, Nov 30, 2017 at 10:36 AM, Nikita Amelchev <nsamelc...@gmail.com> > wrote: > > > Hello, everybody. > > I propose the following design for network compression. > > > > I suggest to implement it like SSL implementation that already works. I > add > > compression filter to a chain of NIO filters. It changes the logic of > > TcpCommunicationSpi.safeTcpHandshake() and > > DirectNioClientWorker.processWrite() methods, where depending on whether > > SSL and/or compression are turned on we compress and/or encrypt data and > > messages using compress handler. Also, I will use application buffer > > (implemented in "GridNioCompressHandler") for decompressing. > > > > For compression verification, I have to write the header with a length of > > compressed data (or get it from an algorithm headers) and a flag of > > compression disabled (for small messages). When we are reading compressed > > data from a channel, we check length, the same way as an isInboundDone > flag > > is checked in SSL implementation. It is conveniently to implement this > flag > > in "CompressEngine" with wrap and unwrap methods for compress and > > decompress to byte buffer. Compression settings should be placed in a > > configuration (Ignite configuration/discovery SPI and > > GridClientConfiguration). > > > > Any thoughts? > > > > > > 2017-11-28 10:39 GMT+03:00 Nikita Amelchev <nsamelc...@gmail.com>: > > > > > Hi, > > > I've filed a ticket [1]. I'll try to share design details in a couple > of > > > days. > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-7024 > > > > > > 2017-11-23 18:31 GMT+03:00 Denis Magda <dma...@apache.org>: > > > > > >> Nikita, > > >> > > >> Sounds like a good plan. Please share the design details prior getting > > >> down to the implementation. > > >> > > >> — > > >> Denis > > >> > > >> > On Nov 23, 2017, at 4:38 AM, Nikita Amelchev <nsamelc...@gmail.com> > > >> wrote: > > >> > > > >> > Hi Igniters! > > >> > > > >> > I’m working on the similar feature for my own project. > > >> > I would like to suggest use in-line compression and write encoded > > bytes > > >> in > > >> > network channel by bytes array buffer. It allows us avoiding > expensive > > >> > memory allocation. > > >> > The described design may be implemented in TcpCommunicationSpi > level. > > We > > >> > can introduce pluggable compressor on TCP level where we will be > able > > to > > >> > describe our compression strategy, for example, exclude some small > > >> messages > > >> > and many other. > > >> > > > >> > If the community doesn't mind I will file the ticket and will start > > >> > implementing it. > > >> > Any thoughts? > > >> > > > >> > 2017-11-23 12:06 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>: > > >> > > > >> >> Denis, > > >> >> > > >> >> Regarding zipped marshaller - this would be inefficient, because > > >> >> compression rate will be lower. > > >> >> > > >> >> On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda <dma...@apache.org> > > >> wrote: > > >> >> > > >> >>> Nikita, > > >> >>> > > >> >>> Your solution sounds reasonable from the first glance. However, > the > > >> >>> communication layer processes a dozen of small system messages > that > > >> >> should > > >> >>> be excluded from the compression. Guess, that we will spend more > > time > > >> on > > >> >>> compressing/decompressing them thus diminishing the positive > effect > > of > > >> >> the > > >> >>> compression. > > >> >>> > > >> >>> Alexey K., Vladimir O., > > >> >>> > > >> >>> What if we create Zip version of the binary marshaller the same > way > > we > > >> >>> implemented GridClientZipOptimizedMarshaller? > > >> >>> > > >> >>> — > > >> >>> Denis > > >> >>> > > >> >>>> On Nov 22, 2017, at 5:36 AM, Alexey Kuznetsov < > > akuznet...@apache.org > > >> > > > >> >>> wrote: > > >> >>>> > > >> >>>> I think it is very useful feature. > > >> >>>> I also have experience when server nodes connected via fast > > network. > > >> >>>> But client nodes via very slow network. > > >> >>>> > > >> >>>> I implemeted GridClientZipOptimizedMarshaller and that solved my > > >> >> issue. > > >> >>>> But this marshaller works only with old > > >> >>>> and org.apache.ignite.internal.client.GridClient and has a lot > of > > >> >>>> limitations. > > >> >>>> But compression was about 6-20x times. > > >> >>>> > > >> >>>> We need a solution for Ignite 2.x and client nodes. > > >> >>>> > > >> >>>> > > >> >>>> On Wed, Nov 22, 2017 at 7:48 PM, Nikita Amelchev < > > >> nsamelc...@gmail.com > > >> >>> > > >> >>>> wrote: > > >> >>>> > > >> >>>>> Hello, Igniters! > > >> >>>>> > > >> >>>>> I think it is a useful feature. I suggest to implement it to > > >> >>> communication > > >> >>>>> SPI like SSL encryption implemented. I have experience with this > > >> >> feature > > >> >>>>> and I can try to develop it. > > >> >>>>> > > >> >>>>> 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin < > > >> >> kukushkinale...@gmail.com > > >> >>>> : > > >> >>>>> > > >> >>>>>> Forwarding to DEV list: Ignite developers, could you please > share > > >> >> your > > >> >>>>>> thoughts on how hard it is to extend Ignite to compress data on > > the > > >> >>>>>> network. > > >> >>>>>> > > >> >>>>>> On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) < > > >> >>>>>> gordon.r...@ninemilefinancial.com> wrote: > > >> >>>>>> > > >> >>>>>>> Hi Igniters, > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> I see there is a lot of discussion in certain threads about > > >> >>>>> compression. > > >> >>>>>>> This seems to have diverged into conversations about object > > versus > > >> >>>>> field > > >> >>>>>>> compression, and even throwing encryption into the mix. For my > > use > > >> >>>>> case, > > >> >>>>>> I > > >> >>>>>>> am not interested in compressing the cache stored in memory, I > > >> have > > >> >>>>>> plenty > > >> >>>>>>> of memory for my application. What I don’t have is a good > > >> network. I > > >> >>>>>> have a > > >> >>>>>>> high latency, low bandwidth network between my C# ignite > client > > >> and > > >> >> my > > >> >>>>>> Java > > >> >>>>>>> ignite server. I only want to compress data when it is sent > over > > >> the > > >> >>>>>>> network to remote nodes. It should be stored in the local > memory > > >> >>>>>>> uncompressed. How can we achive this? Can the > > TcpCommunicationSpi > > >> >>>>> support > > >> >>>>>>> compression? > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> Thanks, > > >> >>>>>>> > > >> >>>>>>> Gordon. > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> This email and any attachments are proprietary & confidential > > and > > >> >> are > > >> >>>>>>> intended solely for the use of the individuals to whom it is > > >> >>> addressed. > > >> >>>>>> Any > > >> >>>>>>> views or opinions expressed are solely for those of the author > > and > > >> >> do > > >> >>>>> not > > >> >>>>>>> necessarily reflect those of Nine Mile Financial Pty. Limited. > > If > > >> >> you > > >> >>>>>> have > > >> >>>>>>> received this email in error, please let us know immediately > by > > >> >> reply > > >> >>>>>> email > > >> >>>>>>> and delete from your system. Nine Mile Financial Pty. Limited. > > >> ABN: > > >> >>> 346 > > >> >>>>>>> 1349 0252 > > >> >>>>>>> > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> -- > > >> >>>>>> Best regards, > > >> >>>>>> Alexey > > >> >>>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> -- > > >> >>>>> Best wishes, > > >> >>>>> Amelchev Nikita > > >> >>>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> -- > > >> >>>> Alexey Kuznetsov > > >> >>> > > >> >>> > > >> >> > > >> > > > >> > > > >> > > > >> > -- > > >> > Best wishes, > > >> > Amelchev Nikita > > >> > > >> > > > > > > > > > -- > > > Best wishes, > > > Amelchev Nikita > > > > > > > > > > > -- > > Best wishes, > > Amelchev Nikita > > > -- Best wishes, Amelchev Nikita
