[
https://issues.apache.org/jira/browse/HBASE-14697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975126#comment-14975126
]
stack commented on HBASE-14697:
-------------------------------
bq. No obvious hotspot.
Well, metrics update is expensive as is bloom check. setResponse takes 4% of
the CPU during this sampling.... LinkedBlockingQueue in consumerLoop takes 5%
of CPU.... seems like some stuff we could clean up... Can we write better
(writing takes 10% of CPU).
bq. Where is the code which look ups key in a block? It is a sequential scan.
These are random reads. The finding in a block must be inconsequential in the
scheme of things [~vrodionov]
> Random Read Improvements
> ------------------------
>
> Key: HBASE-14697
> URL: https://issues.apache.org/jira/browse/HBASE-14697
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
> Reporter: stack
> Attachments: flamegraph-32667.svg
>
>
> I was looking at something else and then got drawn into graphs of where we
> are spending time doing random reads. Interesting is that most CPU -- ~5% is
> in metrics adding -- and bloomfilter look ups cost.
> This is an umbrella background issue to host random read improvements as
> subtasks.
> Here is what perf top shows doing YCSB workloadc (all random read) against a
> server that is up on a 5-node HDFS cluster hosting 7 regions loaded by YCSB.
> {code}
> 4.91% perf-32667.map [.]
> Lorg/apache/hadoop/metrics2/lib/MutableHistogram;.add
> 2.27% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/ipc/RpcServer$Call;.setResponse
> 2.10% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/io/hfile/CompoundBloomFilter;.checkContains
> 2.06% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/HRegion$RegionScannerImpl;.<init>
> 1.94% perf-32667.map [.] Lorg/apache/hadoop/hbase/ipc/CallRunner;.run
> 1.89% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/io/hfile/CombinedBlockCache;.getBlock
> 1.82% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/RSRpcServices;.get
> 1.66% perf-32667.map [.] Lorg/apache/hadoop/hbase/util/Counter;.add
> 1.66% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.getScannersForStoreFiles
> 1.41% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/util/MurmurHash;.hash
> 1.40% [kernel] [k] _spin_lock
> 1.34% perf-32667.map [.] Lsun/nio/ch/SocketChannelImpl;.read
> 1.32% perf-32667.map [.]
> Ljava/util/concurrent/locks/ReentrantReadWriteLock$ReadLock;.unlock
> 1.28% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.shouldUseScanner
> 1.27% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/StoreFile$Reader;.checkGeneralBloomFilter
> 1.17% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/ipc/RpcServer$Connection;.processRequest
> 1.15% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/ipc/RpcServer$Listener$Reader;.processing
> 1.14% perf-32667.map [.] Lsun/nio/ch/SocketChannelImpl;.write
> 1.13% [vdso] [.] 0x00007fffbedff8d4
> 1.11% [kernel] [k] copy_user_generic_string
> 1.11% perf-32667.map [.] Lsun/nio/ch/IOUtil;.write
> 1.05% [nx_nic] [.] 0x000000005f49014a
> 0.96% perf-32667.map [.]
> Ljava/util/concurrent/ConcurrentHashMap;.replaceNode
> 0.88% perf-32667.map [.]
> Ljava/util/concurrent/locks/ReentrantReadWriteLock$Sync;.tryAcquireShared
> 0.86% perf-32667.map [.]
> Ljava/util/concurrent/ConcurrentHashMap;.putVal
> 0.82% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/regionserver/StoreFile$Reader;.passesKeyRangeFilter
> 0.81% perf-32667.map [.] Lorg/apache/hadoop/hbase/ipc/RpcServer;.call
> 0.77% perf-32667.map [.]
> Lorg/apache/hadoop/hbase/protobuf/ProtobufUtil;.toGet
> 0.76% perf-32667.map [.] Lsun/nio/ch/Util;.getTemporaryDirectBuffer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
