HorizonNet commented on a change in pull request #1505: HBASE-24180 Edit test doc around forkcount and speeding up test runs URL: https://github.com/apache/hbase/pull/1505#discussion_r407952783
########## File path: src/main/asciidoc/_chapters/developer.adoc ########## @@ -1344,17 +1368,33 @@ For convenience, you can run `mvn test -P runDevTests` to execute both small and [[hbase.unittests.test.faster]] ==== Running tests faster -By default, `$ mvn test -P runAllTests` runs all small tests in 1 forked instance and the medium and large tests in 5 parallel forked instances. Up these counts to get the build to run faster (you may run into -rare issues of test mutual interference). For example, -allowing that you want to have 2 tests in parallel per core, and you need about 2GB of memory per test (at the extreme), if you have an 8 core, 24GB box, you can have 16 tests in parallel. -but the memory available limits it to 12 (24/2), To run all tests with 12 tests in parallel, do this: +mvn test -P runAllTests -Dsurefire.secondPartForkCount=12+. -If using a version earlier than 2.0, do: +mvn test -P runAllTests -Dsurefire.secondPartThreadCount=12 +. -You can also increase the fork count for the first party by setting -Dsurefire.firstPartForkCount to a value > 1. -The values passed as fork counts can be specified as a fraction of CPU as follows: for two forks per available CPU, set the value to 2.0C; for a fork for every two CPUs, set it to 0.5C. -To increase the speed, you can as well use a ramdisk. -You will need 2GB of memory to run all tests. -You will also need to delete the files between two test run. -The typical way to configure a ramdisk on Linux is: +By default, `$ mvn test -P runAllTests` runs all tests using a quarter of the CPUs available on machine +hosting the test run (see `surefire.firstPartForkCount` and `surefire.secondPartForkCount` in the top-level +hbase `pom.xml`). Up these counts to get the build to run faster. You can also have hbase modules +run their tests in parrallel when the dependency graph allows by passing `--threads=N` when you invoke +maven, where `N` is the amount of parallelism wanted. + +For example, allowing that you want to use all cores on a machine to run tests, +you could start up the maven test run with: + +---- + $ x="1.0C"; mvn -Dsurefire.firstPartForkCount=$x -Dsurefire.secondPartForkCount=$x test -PrunAllTests +---- + +If a 32 core machine, you should see periods during which 32 forked jvms appear in your process listing each running unit tests. +Your milage may vary. Dependent on hardware, overcommittment of CPU, memory, etc., can bring the test suite crashing down, +usually complaining of system exit and incomplete test report xml files. Start gently, with the default fork setting which +uses a quarter of the available CPUs. + +Adding the `--threads=N`, maven will run N modules in parallel when dependencies allow. Be aware, if you have +set the forkcount to `1.0C`, and the threads count to '2', the number of concurrent test runners can approach +2 * CPU count likely overcommitting the host machine. + +You will need ~2.2GB of memory per forked JVM plus the memory used by maven itself (3-4G). + Review comment: Remove this additional empty line. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
