[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106165#comment-14106165
]
Nick Dimiduk commented on HBASE-11331:
--------------------------------------
Thanks for having a look [~stack].
bq. You figure the compressor issue, our not reusing them?
I've punted on the compressor is a non-issue, but I haven't run with a profiler
yet. I think it was related to the non-native gz impl while running locally.
I'll re-enable tracing there as well with the next runs and see what I see.
bq. This is best case (when =false, we are seeking? Or is it always inside
fscache?)
Yes, this configuration is targeting a best case for this patch. The fscache is
minimized with this config, seems to stay down around 3.5g (vs 11.5g
blockcache). Compression ratio is reported as 0.2437, so --size=45 should be
~11g compressed -- larger than the fscache. Because the PE test is random, I
believe we'll be thrashing the fscache with both =true and =false. The iowait
charts indicate both configs are doing io constantly, just more with =false (as
expected).
bq. What is the 'cost' keeping stuff compressed? What if you do a run where all
fits in cache, for both cases?
I'm testing a couple more scenarios, this one was already on the list.
> [blockcache] lazy block decompression
> -------------------------------------
>
> Key: HBASE-11331
> URL: https://issues.apache.org/jira/browse/HBASE-11331
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
> HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch,
> HBASE-11331.05.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf,
> lazy-decompress.02.0.pdf, lazy-decompress.02.1.json,
> lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf,
> v03-20g-045g-true.pdf
>
>
> Maintaining data in its compressed form in the block cache will greatly
> increase our effective blockcache size and should show a meaning improvement
> in cache hit rates in well designed applications. The idea here is to lazily
> decompress/decrypt blocks when they're consumed, rather than as soon as
> they're pulled off of disk.
> This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
