Hello! It is somewhat a part of IEP-20, since I have updated it with this particular direction.
Regards, -- Ilya Kasnacheev 2018-08-24 2:56 GMT+03:00 Denis Magda <dma...@apache.org>: > Hi Ilya, > > Sounds terrific! Is this part of the following Ignite enhancement proposal? > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 20%3A+Data+Compression+in+Ignite > > -- > Denis > > On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev <ilya.kasnach...@gmail.com > > > wrote: > > > Hello! > > > > My plan was to add a compression section to cache configuration, where > you > > can enable compression, enable key compression (which has heavier > > performance implications), adjust dictionary gathering settings, and in > the > > future possibly choose betwen algorithms. In fact I'm not sure, since my > > assumption is that you can always just use latest&greatest, but maybe we > > can have e.g. very fast and not very strong vs. slower but stronger one. > > > > I'm not sure yet if we should share dictionary between all caches vs. > > having separate dictionary for every cache. > > > > With regards to data format, of course there will be room for further > > extension. > > > > Regards, > > > > -- > > Ilya Kasnacheev > > > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov <skoz...@gridgain.com>: > > > > > Hi Ilya > > > > > > Is there a plan to introduce it as an option of Ignite configuration? > In > > > that instead the boolean type I suggest to use the enum and reserve the > > > ability to extend compressions algorithms in future > > > > > > On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com> > > > wrote: > > > > > > > Hello! > > > > > > > > I want to share with the developer community my compression > prototype. > > > > > > > > Long story short, it compresses BinaryObject's byte[] as they are > > written > > > > to Durable Memory page, operating on a pre-built dictionary. Typical > > > > compression ratio is 0.4 (meaning 2.5x compression) using custom > > > > LZW+Huffman. Metadata, indexes and primitive values are unaffected > > > > entirely. > > > > > > > > This is akin to DB2's table-level compression[1] but independently > > > > invented. > > > > > > > > On Yardstick tests performance hit is -6% with PDS and up to -25% (in > > > > throughput) with In-Memory loads. It also means you can fit ~twice as > > > much > > > > data into the same IM cluster, or have higher ram/disk ratio with PDS > > > > cluster, saving on hardware or decreasing latency. > > > > > > > > The code is available as PR 4295[2] (set > IGNITE_ENABLE_COMPRESSION=true > > > to > > > > activate). Note that it will not presently survive a PDS node > restart. > > > > The impact is very small, the patch should be applicable to most 2.x > > > > releases. > > > > > > > > Sure there's a long way before this prototype can have hope of being > > > > included, but first I would like to hear input from fellow igniters. > > > > > > > > See also IEP-20[3]. > > > > > > > > 1. > > > > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10. > > > > 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html > > > > 2. https://github.com/apache/ignite/pull/4295 > > > > 3. > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > > 20%3A+Data+Compression+in+Ignite > > > > > > > > Regards, > > > > > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > -- > > > Sergey Kozlov > > > GridGain Systems > > > www.gridgain.com > > > > > >
