Hey everyone, There has been a bunch of work recently on speeding up the testing to make it easier for developers to iterate quickly on new features fixes. Part of the problem is that the test suite takes anywhere from 1-2 hrs to run and have some apparently non-deterministic hanging of tests.
TL;DR To speed it all up, the attack plan would be: (1) Move long running tests to be integration tests, (2) Use a build server for patches so people only run unit tests locally, (3) We add unit tests when integration tests are breaking a lot but unit tests pass, (4) Go from forked to single jvm unit tests, (5) Add in surefire parallelization (6) Entertain using HBaseTestingUtilFactory I recently chatted with Stack and Doug about ways around this. Here is what we came up with: (1) Break up long running tests from medium and short (2 mins max) tests and move the former to be IntegrationTests. This was based on Todd's suggestion in HBASE-4438. Either naming them integration tests or long running functional tests, they would become part of the 'mvn verify' rather than the 'mvn test' suite of tests. Starting point would be to use Doug's spreadsheet from HBASE-4448 and when its done, the script from HBASE-4480. Right now, that means a LOT of tests are going to shift, but it means when developers run 'mvn test' the amount of time spent running unit tests will be cut down dramatically (hopefully towards the sub 10 -15 mins range) There is an implicit problem here: if the soon to be integration tests capture functionality that is not covered by unit tests, then people may incorrectly think that they are not breaking things. Therefore, we would do (2) and (3): (2) Add a patch continuous integration server that goes and actually builds and tests patches as they come in. This would run 'mvn verify' and ensure that the patch actually isn't breaking high level/complex functionality. It would be a requirement before patches are committed that they pass this build. (3) If we find that the unit tests aren't covering a certain level of functionality that is constantly breaking on the build server, we add more unit tests of the breaking functionality to ensure the unit tests are more complete and provide more assurances to developers when running them. This would be an ongoing process of comparing the integration tests vs. the unit tests. (4) Once we have a true unit test suite, we should be able to go from 'forked' jvm mode back to a single jvm for running tests. Unit tests should not do crazy fault injection, full failure scenarios, so they should be able to cleanup after themselves. This means we are going to get some speedup from not spinning up a new jvm for each test. (5) Once we are running in non-forked mode, we can try turning on parallelized test execution in surefire, which does parallel runs in a single jvm. (6) Once things all run in a single jvm, using the HBaseTestUtilFactory (HBASE-4448) make sense reuse the mini clusters across tests. What does everyone think of this approach? Thanks! -Jesse Yates
