[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421932#comment-16421932
]
stack commented on HBASE-20188:
-------------------------------
Making sure 1.2.7 and 2.0.0 have same dfs client version and that they use same
RPC scheduler (because the fast path Semaphore shines bright in lock and cpu
compares as a main culprit), it is not plain looking at these profiler dumps
why so much slower; we are missing something obvious, something big; e.g. the
block fetches seem to be remote in 2.0.0 but shortcircuit in 1.2.7. There is
some dumb stuff I can fix: e.g.
{code} 276 Total: 2457732537 (0.78%) samples: 2444
277 [ 0] jbyte_disjoint_arraycopy
278 [ 1] org.slf4j.impl.Log4jLoggerAdapter.isTraceEnabled
279 [ 2] org.slf4j.impl.Log4jLoggerAdapter.trace
280 [ 3] org.apache.hadoop.hdfs.client.impl.BlockReaderRemote2.read
281 [ 4] org.apache.hadoop.hdfs.client.impl.BlockReaderUtil.readAll
282 [ 5] org.apache.hadoop.hdfs.client.impl.BlockReaderRemote2.readAll
283 [ 6] org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode
284 [ 7] org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode
285 [ 8] org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange
286 [ 9] org.apache.hadoop.hdfs.DFSInputStream.pread
287 [10] org.apache.hadoop.hdfs.DFSInputStream.read
288 [11] org.apache.hadoop.fs.FSDataInputStream.read
289 [12] org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra
290 [13]
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset
291 [14]
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal
292 [15]
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData
293 [16] org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock
294 [17]
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo
295 [18]
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo
296 [19]
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo
297 [20] org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter
298 [21] org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek
299 [22] org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners
300 [23] org.apache.hadoop.hbase.regionserver.StoreScanner.<init>
301 [24] org.apache.hadoop.hbase.regionserver.HStore.createScanner
302 [25] org.apache.hadoop.hbase.regionserver.HStore.getScanner
303 [26]
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners
304 [27]
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>
305 [28]
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner
306 [29] org.apache.hadoop.hbase.regionserver.HRegion.getScanner
307 [30] org.apache.hadoop.hbase.regionserver.HRegion.getScanner
308 [31] org.apache.hadoop.hbase.regionserver.HRegion.getScanner
309 [32] org.apache.hadoop.hbase.regionserver.RSRpcServices.get
310 [33] org.apache.hadoop.hbase.regionserver.RSRpcServices.get
311 [34]
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
312 [35] org.apache.hadoop.hbase.ipc.RpcServer.call
313 [36] org.apache.hadoop.hbase.ipc.CallRunner.run
314 [37] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
315 [38] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
{code}
> [TESTING] Performance
> ---------------------
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png,
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png,
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png,
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
