[
https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161499#comment-13161499
]
nkeywal commented on HBASE-4712:
--------------------------------
updated version after Jesse's comments:
--
1) Running tests
HBase tests are divided in three categories: small, medium and large, with
corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
This appears like this:
{noformat}
@Category(SmallTests.class)
public class TestHRegionInfo {
@Test
public void testCreateHRegionInfoName() throws Exception {
// ...
}
}
{noformat}
Small tests are executed in a shared JVM. We put in this category all the tests
that can be executed quickly (the maximum execution time for a test is 15
seconds, and they do not use a cluster) in a shared jvm.
Medium tests represents tests that must be executed before proposing a patch.
They are designed to run in less than 30 minutes altogether, and are quite
stable in their results. They're designed to last less than 50 seconds
individually. They can use a cluster, and each of them is executed in a
separate JVM.
Large tests are everything else. They are typically integration tests,
regression tests for specific bugs, timeout tests, performance tests. Some of
them can be flaky. They are executed before a commit on the pre-integration
machines. They can be run on the developer machine as well.
Commands are:
1.1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for
each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there are two reports, one report for small tests, and one report for medium
tests if they are executed
1.2) mvn test -P runAllTests
- execute small tests in a single JVM then medium and large tests in a separate
JVM for each test.
- medium and large tests are NOT executed if there is an error in a small test
- there are two reports, one report for small tests, and one report for medium
and large tests if they are executed
1.3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are
taken into account)
- use actually the official release of surefire & the old connector to junit
- tests are executed in separated JVM
- you will see a new message at the end of the report: "[INFO] Tests are
skipped". It's harmless.
1.4) Various other profiles
- mvn test -P runSmallTests - execute small tests only, in a single JVM.
- mvn test -P runMediumTests - execute medium tests in a separate JVM.
- mvn test -P runLargeTests - execute large tests in a separate JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the
medium and large tests in parallel with two maven instances, and provide a
single report. It must be executed from the directory which contains the
pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second
time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine,
hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex
to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper
functions to create a temp directory and do the cleanup, or to start a cluster.
Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side
effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good
parallelization for people using it, and ease the analysis when the test fails.
Sleeps
- Whenever possible, tests should not use Thread.sleep, but rather waiting for
the real event they need. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition.
This allows understanding what the test is waiting for. Moreover, the test will
work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable
should be done in a 40ms sleep loop. Waiting for a socket operation should be
done in a 200 ms sleep loop.
Tests using a cluster
- Tests using a HRegion do not have to start a cluster: A region can use the
local file system instead of a distributed one.
- Start/stopping a cluster cost around 10 seconds. They should not be started
per test method but per test class.
- Started cluster must be shutdown using
HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster.
When they don't, they should document it. This will allow to share the cluster
later.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could
> help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with
> corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less
> than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50
> seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting
> a patch.
> Large tests are everything else. They are typically integration tests,
> non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same
> machine, hence should not use shared resources as fixed ports or fixed file
> names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex
> to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper
> function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending
> condition. This allows understanding what the test is waiting for. Moreover,
> the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a
> variable should be done in a 40ms sleep loop. Waiting for a socket operation
> should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use
> the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be
> started per test method but per class.
> - Started cluster must be shutdown using
> HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the
> cluster. When they don't, they should document it. This will allow to share
> the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
