[
https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372490#comment-14372490
]
stack commented on HBASE-13291:
-------------------------------
Thanks for the +1.
Yeah, I'd imagine that if we went through all code paths and did this hacky
analysis, 1% here, and 7% there, it could add up. During this session, it
seemed like some fundamental code paths were 'too big' to inline.
On, the hbase-env.sh config, you have it right. The flight recorder has to be
actually switched on to record (jcmd or the jmc console). My guess that when
dormant, it impinges little. The PrintCompilation seems to work w/ stock jvm.
The emissions are cryptic enough especially when it starts talking about
zombies but I found this note useful:
http://www.techpaste.com/2012/02/java-command-line-options-jvm-performance-improvement/
It seemed to match my observations.
Let me commit the patch in a subissue. We are still pretty hung up and I'd like
to try figure why.
On a JVM profiling session at a meetup, lets collect stories. I think we need
some big wins first. Also, there is still too art involved; Flight Recorder
says 'there could be an issue' here' but signals are faint, 'lightweight
profiler' says we are spending loads of time in an area sort of related, then,
I happened to enable a particular logging and indeed, bad issue is plain! It
would be war stories like. Need to gather up some more. [~larsh] probably has a
few.
> Lift the scan ceiling
> ---------------------
>
> Key: HBASE-13291
> URL: https://issues.apache.org/jira/browse/HBASE-13291
> Project: HBase
> Issue Type: Improvement
> Components: Scanners
> Affects Versions: 1.0.0
> Reporter: stack
> Assignee: stack
> Attachments: 13291.inlining.txt, q (1).png, traces.7.svg,
> traces.filterall.svg, traces.nofilter.svg, traces.smaller.svg
>
>
> Scanning medium sized rows with multiple concurrent scanners exhibits
> interesting 'ceiling' properties. A server runs at about 6.7k ops a second
> using 450% of possible 1600% of CPUs when 4 clients each with 10 threads
> doing scan 1000 rows. If I add '--filterAll' argument (do not return
> results), then we run at 1450% of possible 1600% possible but we do 8k ops a
> second.
> Let me attach flame graphs for two cases. Unfortunately, there is some
> frustrating dark art going on. Let me try figure it... Filing issue in
> meantime to keep score in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
