[
https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394751#comment-14394751
]
stack commented on HBASE-13291:
-------------------------------
Filed HBASE-13389 for regression.
On the mvcc parse cost, the hard part is skipping the mvcc... to see the
difference it introduces; you can't if mvcc is present because the parse will
run off the rails.
But 6 days have now elapsed so I can see what it is like when mvcc is 0. I see
an uptick in scan rate from about 9k to about 9.4k... about 5% diff in
throughput?
Profile looks like this in my modified 1.0 (it includes the attached patches):
{code}
Samples: 13M of event 'cycles', Event count (approx.): 439449675139
15.46% perf-14056.map [.] 0x00007f38f13fa41f
10.94% perf-14056.map [.]
Lorg/apache/hadoop/hbase/util/Bytes;.toLongUnsafe in
Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.next
3.53% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/KeyValueHeap;.next in
Lorg/apache/hadoop/hbase/regionserver/HRegion$RegionScannerImpl;.populateResult
3.33% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.peek in
Lorg/apache/hadoop/hbase/regionserver/HRegion$RegionScannerImpl;.populateResult
2.74% perf-14056.map [.]
Lorg/apache/hadoop/hbase/util/Bytes;.toLongUnsafe in
Lorg/apache/hadoop/hbase/regionserver/ScanQueryMatcher;.moreMatch
2.50% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.getNextIndexedKey in
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.next
2.05% perf-14056.map [.] Ljava/util/PriorityQueue;.peek in
Lorg/apache/hadoop/hbase/regionserver/KeyValueHeap;.next
1.54% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.getNextIndexedKey in
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.next
1.43% perf-14056.map [.]
Lorg/apache/hadoop/hbase/util/Bytes;.toShortUnsafe in
Lorg/apache/hadoop/hbase/regionserver/ScanQueryMatcher;.match
1.41% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/ScanDeleteTracker;.reset in
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.next
1.39% perf-14056.map [.]
Lorg/apache/hadoop/hbase/KeyValue;.getDelimiter in
Lorg/apache/hadoop/hbase/regionserver/ScanQueryMatcher;.match
1.33% perf-14056.map [.] Ljava/nio/ByteBuffer;.arrayOffset in
Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.next
1.10% perf-14056.map [.]
Lorg/apache/hadoop/hbase/regionserver/StoreScanner;.next
1.05% perf-14056.map [.]
Lorg/apache/hadoop/hbase/io/hfile/AbstractHFileReader$Scanner;.isSeeked in
Lorg/apache/hadoop/hbase/regionserver/StoreFileScanner;.next
1.01% perf-14056.map [.]
Lorg/apache/hadoop/hbase/KeyValue;.getRowLength in
Lorg/apache/hadoop/hbase/regionserver/ScanQueryMatcher;.match
{code}
I can't really tell what the opaque 0x00007f38f13fa41f that is missing its
symbol is. It could be Bytes... compareTo since this is what is in flight
recorder output but is missing here (interesting how different the picture perf
top and flight recorder present... same basic cast of characters but
attribution is wildly different).
Before the mvcc went to zero, I saw decodeVIntSize taking 15% of CPU (both in
flight recorder and in perf top).
> Lift the scan ceiling
> ---------------------
>
> Key: HBASE-13291
> URL: https://issues.apache.org/jira/browse/HBASE-13291
> Project: HBase
> Issue Type: Improvement
> Components: Scanners
> Affects Versions: 1.0.0
> Reporter: stack
> Assignee: stack
> Attachments: 13291.hacks.txt, 13291.inlining.txt, Screen Shot
> 2015-03-26 at 12.12.13 PM.png, Screen Shot 2015-03-26 at 3.39.33 PM.png,
> hack_to_bypass_bb.txt, nonBBposAndInineMvccVint.txt, q (1).png, traces.7.svg,
> traces.filterall.svg, traces.nofilter.svg, traces.small2.svg,
> traces.smaller.svg
>
>
> Scanning medium sized rows with multiple concurrent scanners exhibits
> interesting 'ceiling' properties. A server runs at about 6.7k ops a second
> using 450% of possible 1600% of CPUs when 4 clients each with 10 threads
> doing scan 1000 rows. If I add '--filterAll' argument (do not return
> results), then we run at 1450% of possible 1600% possible but we do 8k ops a
> second.
> Let me attach flame graphs for two cases. Unfortunately, there is some
> frustrating dark art going on. Let me try figure it... Filing issue in
> meantime to keep score in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
