[
https://issues.apache.org/jira/browse/HBASE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973724#comment-16973724
]
Andrew Kyle Purtell commented on HBASE-23277:
---------------------------------------------
bq. do we need evict blockcache when close file, even for a small file ( size
less than 10MB)? it's a lru block cache, the unused block will be
automatically removed from cache when blockcache usage is high?
That's true, we don't need to do this. Maybe it's not worth doing given your
observations. Perhaps you can try a local patch that removes this code and note
the change, if any, in cache efficiency? An analysis of the blockcache log
lines pre and post change would be interesting, especially a change in the hit
rate. Maybe it's not significant..
> CompactedHFilesDischarger may lead to high cpu usage
> -----------------------------------------------------
>
> Key: HBASE-23277
> URL: https://issues.apache.org/jira/browse/HBASE-23277
> Project: HBase
> Issue Type: Brainstorming
> Reporter: junfei liang
> Priority: Minor
> Attachments: compated_discharger.png
>
>
> in one of online cluster(RS:16C72G, hbase version is 2.x) , cpu average
> usage is 30%, but cpu usage may above 80% occasionally. by a enchanced
> 'top -H' script, i find 'RS_COMPACTED_FILES_DISCHARGER' threads can use
> 50%+ cpu ( 10 * 80% ).
>
> !compated_discharger.png|width=1168,height=454!
>
> by jstack , we found most of stacks is at.
> "org.apache.hadoop.hbase.io.hfile.LruBlockCache#evictBlocksByHfileName"
> public int evictBlocksByHfileName(String hfileName) {
> int numEvicted = 0;
> for (BlockCacheKey key : map.keySet()) {
> if (key.getHfileName().equals(hfileName)) {
> if (evictBlock(key))
> ++numEvicted;
> }
> }
> if (victimHandler != null) {
> numEvicted += victimHandler.evictBlocksByHfileName(hfileName);
> }
> return numEvicted;
> }
>
> when close storefile, now we evict it's blocks from cache, but we have to
> traverse the whole blockcache to locate the block cached, and can lead to
> high cpu usage.
> do we need evict blockcache when close file, even for a small file ( size
> less than 10MB)? it's a lru block cache, the unused block will be
> automatically removed from cache when blockcache usage is high?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
