[
https://issues.apache.org/jira/browse/HBASE-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057416#comment-17057416
]
Michael Stack commented on HBASE-23956:
---------------------------------------
[~ndimiduk] Thanks.
Pushed an addendum on master and branch-2. See below. Thought I was setting for
forked case and for in-context.
{code}
commit 4f92f0f8075d9b177cb1edf782a7eefbe5db5e05 (HEAD -> 2, origin/branch-2)
Author: stack <[email protected]>
Date: Wed Mar 11 14:35:09 2020 -0700
HBASE-23956 Use less resources running tests (#1266)
Addendum to cleanup pom warning.
diff --git a/pom.xml b/pom.xml
index 0559f67a13..34020786d1 100755
--- a/pom.xml
+++ b/pom.xml
@@ -584,17 +584,8 @@
<testFailureIgnore>${surefire.testFailureIgnore}</testFailureIgnore>
<forkedProcessTimeoutInSeconds>${surefire.timeout}</forkedProcessTimeoutInSeconds>
<redirectTestOutputToFile>${test.output.tofile}</redirectTestOutputToFile>
- <!-- These are incontext system properties set on mvn. For forked
- jvms, see hbase-surefire.argLine... So, we might be dup'ing
- configs here and over on what we pass to the forked jvm.
- -->
<systemPropertyVariables>
<test.build.classes>${test.build.classes}</test.build.classes>
- <!--And for eventloops that have no explicit configuration,
netty sets
- nioeventloopgroup thread count to CPU count * 2. Thats too much
- for mini clusters/tests.
- -->
- <io.netty.eventLoopThreads>3</io.netty.eventLoopThreads>
</systemPropertyVariables>
<excludes>
<!-- users can add -D option to skip particular test classes
{code}
> Use less resources running tests
> --------------------------------
>
> Key: HBASE-23956
> URL: https://issues.apache.org/jira/browse/HBASE-23956
> Project: HBase
> Issue Type: Improvement
> Components: test
> Reporter: Michael Stack
> Assignee: Michael Stack
> Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-03-09 at 10.11.26 PM.png, Screen Shot
> 2020-03-09 at 10.11.35 PM.png, Screen Shot 2020-03-09 at 10.20.08 PM.png,
> Screen Shot 2020-03-10 at 7.10.15 AM.png, Screen Shot 2020-03-10 at 7.10.28
> AM.png
>
>
> Our tests can create thousands of threads all up in the one JVM. Using less
> means less memory, less contention, likelier passes, and later, more possible
> parallelism.
> I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to
> see what it does as it runs (this test puts up 4 clusters with replication
> between). It peaks at 2k threads. After some configuration and using less
> HDFS, its possible to get it down to ~800 threads and about 1/2 the
> memory-used. HDFS is a main offender. DataXceivers (Server and Client), jetty
> threads, Volume threads (async disk 'worker' then another for cleanup...),
> image savers, ipc clients -- new thread per incoming connection w/o bound (or
> reuse), block responder threads, anonymous threads, and so on. Many are not
> configurable or boundable or are hard-coded; e.g. each volume gets 4 workers
> regardless. Biggest impact was just downing the count of data nodes. TODO: a
> follow-on that turns down DN counts in all tests.
> I've been using Java Flight Recorder during this study. Here is how you get a
> flight recorder for the a single test run: \{code:java} MAVEN_OPTS="
> -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m"
> mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData
> -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 \{code} i.e.
> start recording on mvn launch, bound the size of the recording, and have the
> test run in the mvn context (DON'T fork). Useful is connecting to the running
> test at the same time from JDK Mission Control. We do the latter because the
> thread reporting screen is overwhelmed by the count of running threads and if
> you connect live, you can at least get a 'live threads' graph w/ count as the
> test progresses. Useful. When the test finishes, it dumps a .jfr file which
> can be opened in JDK MC.
> I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC
> Version 7, the non-commercial latest. Works pretty well. Let me put up a
> patch for tests that cuts down thread counts where we can.
> Let me put up a patch that does first pass on curtailing resource usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
