Alternatively, a good place for this would also be in the book.xml, under (13) Building and Developing HBase
-Jesse On Mon, Nov 28, 2011 at 3:10 PM, Todd Lipcon <[email protected]> wrote: > Haven't had a chance to review the full note below, but one request: > can we please put this in the code tree itself in a file like > TESTING.txt or README.testing.txt or something? > > It'd be nice to be able to refer back to it from within the code tree. > > -Todd > > On Mon, Nov 28, 2011 at 12:29 PM, N Keywal <[email protected]> wrote: > > Hi, > > > > The surefire modification seems to work well, so we can move to the final > > step. > > Please find below the last proposition for the test. The behavior > described > > can be activated with: > > - a commit of hbase-4847 > > - a modification of the jenkins config to run mvn verify vs. mvn tests. > > > > The main points are: > > - should we run small & medium tests only by default on a developer > > machine => here answer is yes > > - should we run all tests on the central build => here answer is yes > > - should we stop when a small test fails => here answer is yes > > - should we stop when a medium test fails => here answer is yes > > > > > > Also, I just had a discussion with Jesse, who think we should keep/have > the > > notion of integration test. If retained, the points above must be > impacted > > to take this into account. > > > > > > fyi, there are today: > > - 416 small tests, executed in ~3 minutes > > - 489 medium tests, executed in ~35 minutes (without parallelization) > > - 280 large tests, executed in ~90 minutes (without parallelization) > > > > -- > > 1) Running tests > > HBase tests are divided in three categories: small, medium and large, > with > > corresponding JUnit categories: SmallTests, MediumTests, LargeTests. > > > > - Small tests are executed in a shared JVM. We put in this category all > the > > tests that can be executed quickly (the maximum execution time for a test > > is 15 seconds, and they do not use a cluster) in a shared jvm. > > - Medium tests represents tests that must be executed before proposing a > > patch. They are designed to run in less than 30 minutes altogether, and > are > > quite stable in their results. They're designed to last less than 50 > > seconds individually. They can use a cluster, and each of them is > executed > > in a separate JVM. > > - Large tests are everything else. They are typically integration tests, > > regression tests for specific bugs, timeout tests, performance tests. > Some > > of them can be flaky. They are executed before a commit on the > > pre-integration machines. They can be run on the developer machine as > well. > > > > Commands are: > > 1) mvn test > > - execute small tests in a single JVM and medium tests in a separate JVM > > for each test > > - medium tests are NOT executed if there is an error in a small test > > - large tests are NOT executed > > - there is one report for small tests, and one report for medium tests > -if > > they are executed- > > > > 2) mvn verify > > - execute small tests in a single JVM then medium tests in a separate > JVM > > for each test, then large tests in a separate JVM as well. > > - medium tests are NOT executed if there is an error in a small test > > - large tests are NOT executed if there is an error in a small or medium > > test > > - there is one report by test category, small, medium and large > > > > 3) mvn test -P localTests -Dtest=myTests > > - remove any category effect (without this specific profile, the > profiles > > are taken into account) > > - use actually the official release of surefire & the old connector to > > junit > > - tests are executed in separated JVM > > - You will see a new message at the end of the report: "[INFO] Tests are > > skipped". It's harmless. > > > > 4) mvn test -P runAllTests > > - execute small tests in a single JVM then medium & large tests in a > > separate JVM for each test > > - medium and large tests are NOT executed if there is an error in a > small > > test > > - large tests are NOT executed if there is an error in a small or medium > > test > > - there are one report for small tests, and one report for medium & > large > > tests -if they are executed- > > > > 5) Various other profiles > > - mvn test -P runSmallTests - execute small tests only, in a single > JVM. > > - mvn test -P runMediumTests - execute medium tests in a single JVM. > > - mvn test -P runLargeTests - execute medium tests in a single JVM. > > > > It's as well possible to use the script 'hbasetests.sh'. This script runs > > the medium and large tests in parallel with two maven instances, and > > provide a single report. It must be executed from the directory which > > contains the pom.xml. Commands are: > > ./dev-support/hbasetests.sh - execute small and medium tests > > ./dev-support/hbasetests.sh runAllTests - execute all tests > > ./dev-support/hbasetests.sh replayFailed - rerun the failed tests a > second > > time, in a separate jvm and without parallelisation. > > > > 2) Writing tests > > Tests rules & hints are: > > - As most as possible, tests should be written as small tests. > > - All tests must be written to support parallel execution on the same > > machine, hence should not use shared resources as fixed ports or fixed > file > > names. > > - Tests should not overlog. More than 100 lines/second makes the logs > > complex to read and use i/o that are hence not available for the other > > tests. > > - Tests can be written with HBaseTestingUtility . This class offers > helper > > functions to create a temp directory and do the cleanup, or to start a > > cluster. > > > > - Categories and execution time > > - All tests must be categorized, if not they could be skipped. > > - All tests should be written to be as fast as possible. > > - Small tests should last less than 15 seconds, and must not have any > > side effect. > > - Medium tests should last less than 45 seconds. > > - large tests should last less than 3 minutes, this ensure a good > > parallelization for the ones using it, and ease the analysis when the > test > > fails. > > > > - Sleeps: > > - Whenever possible, tests should not use sleep, but rather waiting > for > > the real event. This is faster and clearer for the reader. > > - Tests should not do a 'Thread.sleep' without testing an ending > > condition. This allows understanding what the test is waiting for. > > Moreover, the test will work whatever the machine performances. > > - Sleep should be minimal to be as fast as possible. Waiting for a > > variable should be done in a 40ms sleep loop. Waiting for a socket > > operation should be done in a 200 ms sleep loop. > > > > - Tests using a cluster: > > - Tests using a HRegion do not have to start a cluster: A region can > > use the local file system. > > - Start/stopping a cluster cost around 10 seconds. They should not be > > started per test method but per test class. > > - Started cluster must be shutdown using > > HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. > > - As most as possible, tests should use the default settings for the > > cluster. When they don't, they should document it. This will allow to > share > > the cluster later. > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- ------------------- Jesse Yates 240-888-2200 @jesse_yates
