I propose a minor change in semantics for TestFactory. TestDescription
includes the following:
/**
* The <code>TestDescription</code> is an object which describes a test;
* it is a factory for the test object and provides support facilities
* for setting up and executing the test.
* <p>
* The description contains of a set of properties. Some of these
* properties have well defined names which are used to construct
* the execution environment of the test. These well defined names are:
* <ul>
So we could rename TestFactory and setup method to this:
public interface TestExecutionEnvironment {
/**
* Performs construction of execution environment for this test.
* If an exception is thrown, the <code>tearDown</code> method will
* still be called.
*
* @param config the test properties
* @throws Exception if construct fails for any reason
*/
Test construct(QAConfig config) throws Exception;
/**
* Tears down any construct that was needed to run the test. This
method is
* called even if construct throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
void tearDown();
}
Peter Firmstone wrote:
I've completed the first phase of refactoring, separating out the
environment setup and test run into separate interfaces as per
appended. This should allow the setup and teardown and execution of
the tests to remain separate concerns. At present all tests implement
both interfaces, so nothing has really changed, yet.
When the qa test run is complete and has passed all tests, I'll upload
the changes for review.
The next stage will be the separation of setup and test execution for
the tests that have experienced failures recently in trunk. This will
simplify test execution and should make it easier to manage shared state.
From there, perhaps we should merge the changes made in trunk back in
incrementally.
Since a considerable amount of work is required to refactor all tests,
I suggest we migrate the tests to the new format slowly over time on
an as needed basis.
The names of interfaces and methods can be changed if necessary:
public interface TestFactory {
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
Test setup(QAConfig config) throws Exception;
/**
* Tears down any setup that was needed to run the test. This
method is
* called even if setup throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
void tearDown();
}
public interface Test {
/**
* Failure type. Indicates that a test failed because of a recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
int INDEF = 2;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
int PASSED = 6;
/**
* Failure type indicating that the test should be rerun.
*/
int RERUN = 4;
/**
* Failure type indicating that the test should not be run.
*/
int SKIP = 3;
/**
* Failure type. Indicates that a test failed because the product
failed,
* or an unrecognized environmental failure occured.
*/
int TEST = 0;
/**
* Failure type (for analyzers) indicating failure not recognized
*/
int UNKNOWN = 5;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
void run() throws Exception;
}
Regards,
Peter.
Peter Firmstone wrote:
Presently most tests have a lot of shared mutable state.
This isn't helped by the setup(QAConfig) mutator method in the Test
interface appended.
An alternative might be:
public Test setup(QAConfig config) throws Exception;
This would allow the test to return another Test object, fully
constructed with immutable state.
It would also allow the test to create an object that can run several
other Test objects, rather than relying on inheritance.
Thoughts?
/**
* This interface must implemented by all tests supported by the test
* harness. The following sequence of events is performed for each
* test:
* <p><ul>
* <li> the test class is instantiated and it's no-arg constructor
called.
* The constructor will typically perform minimal initialization,
* since the test does not have access to the test environment
* <li> the setup method is called, passing the config object.
* This provides an opportunity for performing any test setup
* that relies on accessing configuration parameters.
* <li> the run method is called to run the test
* <li> the teardown method is called to clean up state or services
* created by the test. This method is called
* even if <code>setup</code> or <code>run</code> throws an
exception.
* </ul>
*/
public interface Test {
/**
* Failure type. Indicates that a test failed because the product
failed,
* or an unrecognized environmental failure occured.
*/
public final static int TEST = 0;
/**
* Failure type. Indicates that a test failed because of a recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
public final static int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
public final static int INDEF = 2;
/** Failure type indicating that the test should not be run. */
public final static int SKIP = 3;
/** Failure type indicating that the test should be rerun. */
public final static int RERUN = 4;
/** Failure type (for analyzers) indicating failure not recognized */
public final static int UNKNOWN = 5;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
public final static int PASSED = 6;
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
public void setup(QAConfig config) throws Exception;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
public void run() throws Exception;
/**
* Tears down any setup that was needed to run the test. This
method is
* called even if setup throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
public void tearDown();
}
