[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-11331:
---------------------------------
Attachment: lazy-decompress.02.1.pdf
re: lazy-decompress.02.1.pdf
Attaching some metrics after running v02 with less os page cache and feature
enabled/disabled.
This time the total RS heap is increased to 20g, so the effective blockcache
size is ~12g. The machine has 24g of RAM, so this squeezes the OS-available
memory down to ~4g -- when the blockcache is full. I've added blockcache size
and OS page cache to the charts Same table, same compression ratio, same PE
randomRead test, this time run with --size=45, so I expect ~11g of compressed
blocks.
I expected this to be an IO-bound workload with flag=false but to be
cache-at-capacity for flag=true. The massive evictions look to me like the
blockcache heapsize calculation is based on the decompressed size, not
compressed size. Updating patch.
> [blockcache] lazy block decompression
> -------------------------------------
>
> Key: HBASE-11331
> URL: https://issues.apache.org/jira/browse/HBASE-11331
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
> HBASE-11331.02.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf,
> lazy-decompress.02.0.pdf, lazy-decompress.02.1.pdf
>
>
> Maintaining data in its compressed form in the block cache will greatly
> increase our effective blockcache size and should show a meaning improvement
> in cache hit rates in well designed applications. The idea here is to lazily
> decompress/decrypt blocks when they're consumed, rather than as soon as
> they're pulled off of disk.
> This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
