[
https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-13291:
--------------------------
Attachment: traces.small2.svg
Screen Shot 2015-03-26 at 12.12.13 PM.png
Minor update.
Played with doubling Handlers and Readers but made no difference in
throughtput. I was maybe burning more CPU. Using 20 handlers instead of 30
(default) had us doing about same cpu but less throughput (7k vs 8k). So we are
up against ceiling.
Looking at what we were doing, most is compareTo. I tried using java compareTo
instead of our copy of guava unsafe (our default) but pure java was more cpu
and slower by about 15% (8k vs 7k).
Attached flamegraph is where we are at when up against the ceiling. Has
inlining which gets us some small benefit but means lower methods are missing.
Got perf top working using https://github.com/jrudolph/perf-map-agent. It just
shows us same thing that we spend all our CPU in
HFileReaderV3#ScannerV3#readKeyValueLen and SQM#match (compareTo). If I ask
perf top to show inlined functions, it fingers the HBB#_get again. Hmm...
> Lift the scan ceiling
> ---------------------
>
> Key: HBASE-13291
> URL: https://issues.apache.org/jira/browse/HBASE-13291
> Project: HBase
> Issue Type: Improvement
> Components: Scanners
> Affects Versions: 1.0.0
> Reporter: stack
> Assignee: stack
> Attachments: 13291.inlining.txt, Screen Shot 2015-03-26 at 12.12.13
> PM.png, q (1).png, traces.7.svg, traces.filterall.svg, traces.nofilter.svg,
> traces.small2.svg, traces.smaller.svg
>
>
> Scanning medium sized rows with multiple concurrent scanners exhibits
> interesting 'ceiling' properties. A server runs at about 6.7k ops a second
> using 450% of possible 1600% of CPUs when 4 clients each with 10 threads
> doing scan 1000 rows. If I add '--filterAll' argument (do not return
> results), then we run at 1450% of possible 1600% possible but we do 8k ops a
> second.
> Let me attach flame graphs for two cases. Unfortunately, there is some
> frustrating dark art going on. Let me try figure it... Filing issue in
> meantime to keep score in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
