Michael Stack created HBASE-23956:
-------------------------------------
Summary: Use less resources running tests
Key: HBASE-23956
URL: https://issues.apache.org/jira/browse/HBASE-23956
Project: HBase
Issue Type: Improvement
Components: test
Reporter: Michael Stack
Our tests can create thousands of threads all up in the one JVM. Using less
means less memory, less contention, likelier passes, and later, more possible
parallelism.
I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to
see what it does as it runs (this test puts up 4 clusters with replication
between). It peaks at 2k threads. After some configuration and using less HDFS,
its possible to get it down to ~800 threads and about 1/2 the memory-used. HDFS
is a main offender. DataXceivers (Server and Client), jetty threads, Volume
threads (async disk 'worker' then another for cleanup...), image savers, ipc
clients -- new thread per incoming connection w/o bound (or reuse), block
responder threads, anonymous threads, and so on. Many are not configurable or
boundable or are hard-coded; e.g. each volume gets 4 workers regardless.
Biggest impact was just downing the count of data nodes. TODO: a follow-on that
turns down DN counts in all tests.
I've been using Java Flight Recorder during this study. Here is how you get a
flight recorder for the a single test run: \{code:java} MAVEN_OPTS="
-XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m"
mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData
-Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 \{code} i.e.
start recording on mvn launch, bound the size of the recording, and have the
test run in the mvn context (DON'T fork). Useful is connecting to the running
test at the same time from JDK Mission Control. We do the latter because the
thread reporting screen is overwhelmed by the count of running threads and if
you connect live, you can at least get a 'live threads' graph w/ count as the
test progresses. Useful. When the test finishes, it dumps a .jfr file which can
be opened in JDK MC.
I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC
Version 7, the non-commercial latest. Works pretty well. Let me put up a patch
for tests that cuts down thread counts where we can.
Let me put up a patch that does first pass on curtailing resource usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
