[
https://issues.apache.org/jira/browse/HBASE-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735710#comment-16735710
]
Zheng Hu commented on HBASE-21657:
----------------------------------
bq. With patch V3, what is the E2E perf gain that you are seeing
[~anoop.hbase], In the overview picture[1], we can see HBase2.0.4 with
patch.v3 (14264.23 ops/sec) gain 40% throughtput improvement compared to
HBase2.0.4 without patch.v3 (10177.14 ops/sec)
bq. On the PrivateCellUtil method, we can test its perf impact using a simple
standalone JMH testcase?
Yeah, I can try this by evaluating the perf of estimatedSerializedSize for
the SizeCached*KeyValue & ByteBufferKeyValue.
BTW, I see some cpu wasted on ByteBufferKeyValue#getKeyLength, so maybe the
problem is here ? In estimatedSerializedSizeOf(), our
cell.getSerializedSize() should only calculate the cell size with tags, but
the ByteBufferKeyValue#getSerializedSize run the without tags path. Let me take
a look.
{code}
33 16
org.apache.hadoop.hbase.util.UnsafeAccess.getAsInt(UnsafeAccess.java:244)
org.apache.hadoop.hbase.util.UnsafeAccess.toInt(UnsafeAccess.java:218)
org.apache.hadoop.hbase.util.ByteBufferUtils.toInt(ByteBufferUtils.java:808)
org.apache.hadoop.hbase.ByteBufferKeyValue.getKeyLength(ByteBufferKeyValue.java:140)
org.apache.hadoop.hbase.ByteBufferKeyValue.getSerializedSize(ByteBufferKeyValue.java:286)
org.apache.hadoop.hbase.ExtendedCell.getSerializedSize(ExtendedCell.java:94)
org.apache.hadoop.hbase.PrivateCellUtil.estimatedSerializedSizeOf(PrivateCellUtil.java:2739)
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6654)
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6396)
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3120)
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3369)
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
1.
https://issues.apache.org/jira/secure/attachment/12953984/12953984_image-2019-01-07-19-03-55-577.png
> PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100%
> scan case.
> ------------------------------------------------------------------------------------
>
> Key: HBASE-21657
> URL: https://issues.apache.org/jira/browse/HBASE-21657
> Project: HBase
> Issue Type: Bug
> Components: Performance
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21657.v1.patch, HBASE-21657.v2.patch,
> HBASE-21657.v3.patch, HBASE-21657.v3.patch,
> HBase1.4.9-ssd-10000000-rows-flamegraph.svg,
> HBase1.4.9-ssd-10000000-rows-qps-latency.png,
> HBase2.0.4-patch-v2-ssd-10000000-rows-qps-and-latency.png,
> HBase2.0.4-patch-v2-ssd-10000000-rows.svg,
> HBase2.0.4-patch-v3-ssd-10000000-rows-flamegraph.svg,
> HBase2.0.4-patch-v3-ssd-10000000-rows-qps-and-latency.png,
> HBase2.0.4-ssd-10000000-rows-flamegraph.svg,
> HBase2.0.4-ssd-10000000-rows-qps-latency.png, HBase2.0.4-with-patch.v2.png,
> HBase2.0.4-without-patch-v2.png, hbase2.0.4-ssd-scan-traces.2.svg,
> hbase2.0.4-ssd-scan-traces.svg, hbase20-ssd-100-scan-traces.svg,
> image-2019-01-07-19-03-37-930.png, image-2019-01-07-19-03-55-577.png,
> overview-statstics-1.png, run.log
>
>
> We are evaluating the performance of branch-2, and find that the throughput
> of scan in SSD cluster is almost the same as HDD cluster. so I made a
> FlameGraph on RS, and found that the
> PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it
> has been the bottleneck in 100% scan case.
> See the [^hbase20-ssd-100-scan-traces.svg]
> BTW, in our XiaoMi branch, we introduce a
> HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells
> (for metric monitor), so it seems the performance loss was amplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
