[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039560#comment-17039560
]
Mark Robert Miller edited comment on HBASE-23779 at 2/18/20 11:45 PM:
----------------------------------------------------------------------
In the meantime, some some ideas on JVM args that be able to help the situation
a little.
-XX:-UseContainerSupport - when running in docker, query docker for hardware
info, not the host. I have not played with it, but makes sense to me.
-XX:ActiveProcessorCount - fake the active processer count to 1 - when running
multiple jvms, you don't want each one to think it should grab 32 gc threads on
your 32 hyperthreaded core machine. I'd much prefer to run 32 JVMs and have 32
gc threads.
-XX:+AlwaysPreTouch -XX:+UseTransparentHugePages - given so much JVM startup
and shutdowns, some things might help more when there is more JVM reuse - so
much time and cost is in creating large heap jvms and spinning up lots of
unused resources and threads that it's hard to win much with good tuning - but
huge pages, always a nice win.
-XX:+UseParallelGC -XX:TieredStopAtLevel=1 - even when G1 is available, this is
gold standard for test speed. Maybe with like -XX:ParallelGCThreads=1 if not
using ActiveProcessorCout.
Probably not a big advantage when each med/large JVM is spinning up thousands
of useless threads that eat ram, heap and os resources, but something, and
probably pretty nice for small test single jvm runs now.
It's also generally best to pin Xms and Xmx vs eat all the resizing cost. I
suggest the opposite above given the RAM reqs - to keep the JVMs that don't
need it from sucking up so much RAM unnecessarily - but with good reuse, you
want to pin them.
Most of the tests don't need these huge heaps and limits - I think that having
them for the bad apples and outliers just allows the 95% of tests to easily be
wasteful and misbehave.
* Note: UseContainerSupport is enabled by default - the above disables - so
guess that should already be in affect.
was (Author: markrmiller):
In the meantime, some some ideas on JVM args that be able to help the situation
a little.
-XX:-UseContainerSupport - when running in docker, query docker for hardware
info, not the host. I have not played with it, but makes sense to me.
-XX:ActiveProcessorCount - fake the active processer count to 1 - when running
multiple jvms, you don't want each one to think it should grab 32 gc threads on
your 32 hyperthreaded core machine. I'd much prefer to run 32 JVMs and have 32
gc threads.
-XX:+AlwaysPreTouch -XX:+UseTransparentHugePages - given so much JVM startup
and shutdowns, some things might help more when there is more JVM reuse - so
much time and cost is in creating large heap jvms and spinning up lots of
unused resources and threads that it's hard to win much with good tuning - but
huge pages, always a nice win.
-XX:+UseParallelGC -XX:TieredStopAtLevel=1 - even when G1 is available, this is
gold standard for test speed. Maybe with like -XX:ParallelGCThreads=1 if not
using ActiveProcessorCout.
Probably not a big advantage when each med/large JVM is spinning up thousands
of useless threads that eat ram, heap and os resources, but something, and
probably pretty nice for small test single jvm runs now.
It's also generally best to pin Xms and Xmx vs eat all the resizing cost. I
suggest the opposite above given the RAM reqs - to keep the JVMs that don't
need it from sucking up so much RAM unnecessarily - but with good reuse, you
want to pin them.
Most of the tests don't need these huge heaps and limits - I think that having
them for the bad apples and outliers just allows the 95% of tests to easily be
wasteful and misbehave.
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Michael Stack
> Assignee: Michael Stack
> Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: addendum2.patch, test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
