[
https://issues.apache.org/jira/browse/HBASE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004384#comment-17004384
]
zhuobin zheng commented on HBASE-20673:
---------------------------------------
Hi, [~stack]
I also encountered similar problems in my production environment: *Too many
KeyValue implementation types confused JIT* .
And cresult in a large amount of CPU waste, resulting in full cpu usage of
singal server(48 core, cpu 4800%), and a significant decrease in read and write
speeds.
Through Profiler analysis, it is found that itable stub takes a lot of CPU time
(almost 70%)
Through Jstack: A large number of read and write threads are stuck in several
special places where CellComparator compares:
[https://github.com/apache/hbase/blob/branch-1.2/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java#L186]
[https://github.com/apache/hbase/blob/branch-1.2/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java#L157]
The Jstack really confused me. Because this line of code does nothing but
resolve the actual calling code address.
But combining the above profiler result (70% cpu in itable stub).
Now, I think there are two possible reasons for the crazy use of cpu.
# Too many Cell implements confuesd JIT. Result jvm interface call to original
itable scan.
# KeyValue implements too many interface, cause itable too long.
{code:java}
//代码占位符
public class KeyValue implements Cell, HeapSize, Cloneable, SettableSequenceId,
SettableTimestamp{code}
I think this situation will occur when the two types of KeyValue /
NoTagKeyValue are more evenly distributed.
But unfortunately, although it has always appeared in the production
environment, I cannot reproduce it in the test environment, so I cannot provide
a better test solution.
Now I will try to modify a part of the code so that a large number of Cell
implementations are of one type, and put it into the production environment to
see if it can solve the 100% CPU problem.
> Reduce the number of Cell implementations; the profusion is distracting to
> users and JIT
> ----------------------------------------------------------------------------------------
>
> Key: HBASE-20673
> URL: https://issues.apache.org/jira/browse/HBASE-20673
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
> Reporter: Michael Stack
> Assignee: Michael Stack
> Priority: Major
> Attachments: 0001-current.patch, 0001-current.patch, hits.20673.png
>
>
> We have a wild blossom of Cell implementations in hbase. Purge the bulk of
> them. Make it so we do one type well. JIT gets confused if it has an abstract
> or an interface and then the instantiated classes are myriad (megamorphic).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
