Todd Lipcon created HBASE-22134: ----------------------------------- Summary: JIT deoptimization in Cell.write Key: HBASE-22134 URL: https://issues.apache.org/jira/browse/HBASE-22134 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 2.0.2 Reporter: Todd Lipcon
I was looking at a profile of a workload which was running compaction very slowly, and saw that the top CPU consumers were from JVM internals regarding deoptimization. I managed to write a little systemtap script to extract the deoptimization log and got the following in a tight loop: "Uncommon trap: trap_request=0xffffff67 fr.pc=0x00007f85bcdb8644" "Uncommon trap: reason=unstable_if action=none pc=0x00007f85bcdb8644 method=org.apache.hadoop.hbase.io.encoding.NoneEncoder.write(Lorg/apache/hadoop/hbase/Cell;)I @ 67" "DEOPT PACKING pc=0x00007f85bcdb8644 sp=0x00007f84d3d83080" "DEOPT UNPACKING pc=0x00007f85b5005229 sp=0x00007f84d3d82f30 mode 2" The java stack is spending most of its time at: java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hbase.io.encoding.NoneEncoder.write(NoneEncoder.java:57) at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.encode(NoOpDataBlockEncoder.java:55) at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.write(HFileBlock.java:983) at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.append(HFileWriterImpl.java:740) This was with Oracle JDK 1.8.0_112. Likely a JDK bug but perhaps some reorganization of this code path could help avoid triggering the bug. -- This message was sent by Atlassian JIRA (v7.6.3#76005)