Hello! I want to share with the developer community my compression prototype.
Long story short, it compresses BinaryObject's byte[] as they are written to Durable Memory page, operating on a pre-built dictionary. Typical compression ratio is 0.4 (meaning 2.5x compression) using custom LZW+Huffman. Metadata, indexes and primitive values are unaffected entirely. This is akin to DB2's table-level compression[1] but independently invented. On Yardstick tests performance hit is -6% with PDS and up to -25% (in throughput) with In-Memory loads. It also means you can fit ~twice as much data into the same IM cluster, or have higher ram/disk ratio with PDS cluster, saving on hardware or decreasing latency. The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true to activate). Note that it will not presently survive a PDS node restart. The impact is very small, the patch should be applicable to most 2.x releases. Sure there's a long way before this prototype can have hope of being included, but first I would like to hear input from fellow igniters. See also IEP-20[3]. 1. https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html 2. https://github.com/apache/ignite/pull/4295 3. https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite Regards, -- Ilya Kasnacheev
