[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123337#comment-13123337 ]
Jacek Migdal commented on HBASE-4218: ------------------------------------- Performance results on production data. CopyKeyDeltaEncoder: Compression performance: 1136.33 MB/s (+/- 60.91 MB/s) Decompression performance: 373.29 MB/s (+/- 281.22 MB/s) BitsetKeyDeltaEncoder: Compression performance: 147.57 MB/s (+/- 0.58 MB/s) Decompression performance: 166.78 MB/s (+/- 54.81 MB/s) PrefixKeyDeltaEncoder: Compression performance: 293.94 MB/s (+/- 1.97 MB/s) Decompression performance: 233.61 MB/s (+/- 91.97 MB/s) FastDiffDeltaEncoder: Compression performance: 203.47 MB/s (+/- 0.37 MB/s) Decompression performance: 196.77 MB/s (+/- 43.22 MB/s) DiffKeyDeltaEncoder: Compression performance: 187.74 MB/s (+/- 0.24 MB/s) Decompression performance: 163.13 MB/s (+/- 12.17 MB/s) LZO: Compression performance: 260.35 MB/s (+/- 0.76 MB/s) Decompression performance: 173.45 MB/s (+/- 76.13 MB/s) CopyKeyDeltaEncoder Saved bytes: -4 Key compression ratio: -0.00 % All compression ratio: -0.00 % LZO compressed size: 152019 LZO compression ratio: 85.79 % BitsetKeyDeltaEncoder Saved bytes: 747061 Key compression ratio: 75.46 % All compression ratio: 69.82 % LZO compressed size: 124438 LZO compression ratio: 88.37 % PrefixKeyDeltaEncoder Saved bytes: 831602 Key compression ratio: 84.00 % All compression ratio: 77.72 % LZO compressed size: 117285 LZO compression ratio: 89.04 % FastDiffDeltaEncoder Saved bytes: 935275 Key compression ratio: 94.47 % All compression ratio: 87.41 % LZO compressed size: 94360 LZO compression ratio: 91.18 % DiffKeyDeltaEncoder Saved bytes: 909175 Key compression ratio: 91.84 % All compression ratio: 84.97 % LZO compressed size: 96597 LZO compression ratio: 90.97 % Total KV prefix length: 80000 Total key length: 910000 Total key redundancy: 781606 Total value length: 80000 DeltaEncodingSeekPerformance BlockDeltaEncoder onDisk='NONE' inCache='NONE' inMemory=false Read speed: 63.99 (MB/s) Seeks per second: 54901.21 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='BITSET' inMemory=false Read speed: 46.73 (MB/s) Seeks per second: 13570.50 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='PREFIX' inMemory=false Read speed: 55.88 (MB/s) Seeks per second: 20298.89 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='DIFF' inMemory=false Read speed: 54.39 (MB/s) Seeks per second: 15082.79 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='FAST_DIFF' inMemory=false Read speed: 54.12 (MB/s) Seeks per second: 15432.61 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='NONE' inMemory=true Read speed: 64.37 (MB/s) Seeks per second: 56779.82 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='BITSET' inMemory=true Read speed: 35.42 (MB/s) Seeks per second: 46170.87 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='PREFIX' inMemory=true Read speed: 43.54 (MB/s) Seeks per second: 60108.48 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='DIFF' inMemory=true Read speed: 40.62 (MB/s) Seeks per second: 48779.68 (#/s) BlockDeltaEncoder onDisk='NONE' inCache='FAST_DIFF' inMemory=true Read speed: 40.76 (MB/s) Seeks per second: 57291.22 (#/s) > Delta Encoding of KeyValues (aka prefix compression) > ----------------------------------------------------- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io > Affects Versions: 0.94.0 > Reporter: Jacek Migdal > Labels: compression > Attachments: open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira