[
https://issues.apache.org/jira/browse/HBASE-23795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040572#comment-17040572
]
Mark Robert Miller commented on HBASE-23795:
--------------------------------------------
Now that I have spent enough time to understand what is possible here, to kind
of zoom out, here is what would need to happen (this can take some time, but is
easily done in parts):
* Tune HBase and hdfs settings to make sense for the small world we are
creating in tests. If you wash a test out with threads, it's really not very
realistic at all. This means pinning settings for the JVM so that the right
number of threads is created, settings for HBase and hdfs that make sense -
handler counts and pool sizes netty 'stuff' - and scaling down thousands of
threads to 200-300 or something. Not many of these threads are running at the
same time - the rest is just flooding and eating resources.
* Make things close and shutdown and enforce this. Lots and lots of various
leaks currently. Once mostly removed, you can start reusing JVM's, but I think
there are other benefits to actually closing and stopping your resources
explicitly. Not closing or shutting down some things explicit can have
lingering OS ramifications even when creating new JVM's.
* Have tests clean up their expensive or config statics and reset sys props,
with enforcement. Needed for JVM reuse.
* Clean up any heavy resource usage you can't currently control or doesn't
seem to make a lot of sense (I think an ipc thread pool is set to core and max
size 200?)
That's kind of the core of it, there is a lot of related useful stuff to do I
think.
A lot of these tests are reasonably fast now. They could be even faster with a
little work, but even without that, they are not that slow. Loading up a JVM,
loading up 10-20k classes, warming up JIT, blah blah, that is super costly.
Just getting to rerunning tests in the same JVM will be super helpful. There is
a lot that can be done after that as well, but such a large win - good enough
goal for now. Most of these tests don't even need that much RAM. They are just
not running with sensible resources.
> Enable all tests to be run in parallel on reused JVMs.
> ------------------------------------------------------
>
> Key: HBASE-23795
> URL: https://issues.apache.org/jira/browse/HBASE-23795
> Project: HBase
> Issue Type: Wish
> Reporter: Mark Robert Miller
> Priority: Major
>
> I'd like to be able to run HBase tests in under 30-40 minutes on good
> parallel hardware.
> It will require some small changes / fixes for that wish to come true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
