Date: 2004-08-20T10:20:06
Editor: EddieOneil <[EMAIL PROTECTED]>
Wiki: Apache Beehive Wiki
Page: NetUI/Testing
URL: http://wiki.apache.org/beehive/NetUI/Testing
Add contents of NetUI test e-mail to the wiki.
New Page:
The NetUI tests consist of two parts -- standalone JUnit tests and Test
Recorder (TR) tests. Both types of test are based on JUnit. The standalone
JUnit tests are just that; !TestCase subclasses that test the NetUI APIs
outside of a running server. The Test Recorder tests are tests that run
against a running Tomcat instance in Beehive.
Both the standalone JUnit and Test Recorder tests have two broad
categorizations -- DRTs and BVTs.
The DRT (Developer Regression Test) suite *must* pass at 100% before any
checkins can be submitted into the `$BEEHIVE_HOME/netui` tree.
The BVT (Build Verification Tests) suite consists of all of tests to run. Most
of us tend to run the BVTs before we checkin, though they don't always pass at
100%; for example, today we've got 21 TR tests that fail out 404 total.
Generally, additional BVTs shouldn't break on checkins either. <g>
Currently, all test suites produce JUnit test results in the usual format using
the `<junitreport>` Ant task. These results are produced in a subdirectory of
`$BEEHIVE_HOME/netui/build`. The specific locations for the JUnit and TR tests
are below.
'''Contents'''
[[TableOfContents]]
= NetUI Server Tests =
In addition to the API-level JUnit tests, NetUI uses a piece of home-grown
software called the Test Recorder (TR) to run tests against a live server --
Tomcat is the server in the context of Beehive.
The DRT suite currently runs from this directory
{{{
$BEEHIVE_HOME/netui/test/webapps/drt/
}}}
against a webapp in a subdirectory of the above called "coreWeb" using the Test
Recorder configuration / tests stored subdirectories `testRecorder/config` and
`testRecorder/tests` respectively. There is also a `build.xml` file in the
above directory that controls the Tomcat server, coreWeb build, and running
test suites. This directory is assumed to be the working directory for the
rest of the Ant commands in this section.
In order to run a set of TR tests, the following steps occur:
* build the coreWeb webapp
* start Tomcat
* deploy the coreWeb webapp
* run a test suite (drt, bvt, data binding, etc), which produces an HTML test
result report
* stop Tomcat
Currently in Beehive, the instance of Tomcat that is used lives in:
{{{
$BEEHIVE_HOME/installed/jakarta-tomcat-5.0.25/...
}}}
When the DRTs run as part of an "ant clean deploy drt" from either
`$BEEHIVE_HOME/` or `$BEEHIVE_HOME/netui`, this process is done automatically
meaning that the build takes care of building the webapp, starting / stopping
Tomcat, and running the tests. The result of a DRT run can be found here:
{{{
$BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html
}}}
In addition to an automatic DRT run, the same steps can be run manually from
the `$BEEHIVE_HOME/netui/test/webapps/drt` directory. This is very useful when
developing tests or just learning about NetUI as a normal web browser can be
used to walk the tests manually. To start the Tomcat server, you'll need two
shells both with a Beehive development environment (see
`$BEEHIVE_HOME/BUILDING.txt`) in the directory
`$BEEHIVE_HOME/netui/test/webapps/drt`.
In the first shell, start Tomcat:
{{{
ant start
}}}
In the second shell, build the coreWeb web application:
{{{
ant build
}}}
Now, the "coreWeb" web application is ready to be installed on the running
Tomcat server. The steps needed to deploy to Tomcat can vary depending on the
state of Tomcat. There are three relevant targets:
* deploy -- deploys the webapp for the first time. This will fail if the
webapp is already running.
* undeploy -- undeploys a running webapp from Tomcat. This will fail if the
webapp is not running or did not start successfully
* redeploy -- redeploy a successfully running webapp. This will fail if the
webapp did not start successfully
These targets can be run in combination to make sure that a webapp deploys
correctly to Tomcat; for example, if the "coreWeb" webapp is already deployed,
it needs to be redeployed or undeployed and then deployed. Feedback for each
step is available on the console of shell two. Also, it is possible for a
webapp to fail deployment, so just be sure to read the console output messages
to make sure that deployment was in fact successful. To deploy "coreWeb"
initially, run in shell two:
{{{
ant deploy
}}}
To redeploy this webapp, run `ant redeploy` or `ant undeploy deploy`. The
latter is usually a safe command as it will remove the webapp if it doesn't
exist and then deploy it.
Note, it's acceptable for some of these targets to fail -- if "coreWeb" isn't
running on Tomcat and `ant undeploy` is run, it will fail like this:
{{{
Buildfile: build.xml
undeploy:
undeploy:
[tomcatundeploy] FAIL - No context exists for path /coreWeb
BUILD FAILED
C:\dev\apache\beehive\netui\test\webapps\drt\build.xml:147: Following error
occurred while executing this line
C:\dev\apache\beehive\test\ant\runTomcat.xml:31: FAIL - No context exists for
path /coreWeb
}}}
To deploy it, just run "ant deploy" which succeeds with output like this:
{{{
Buildfile: build.xml
deploy:
deploy:
[echo] deploy webapp from
file://c:\dev\apache\beehive/netui/test/webapps/drt/coreWeb with context path
coreWeb
[tomcatdeploy] OK - Deployed application at context path /coreWeb
BUILD SUCCESSFUL
Total time: 13 seconds
}}}
Once the `coreWeb` webapp is deployed successfully, it is accessible using the
URL
{{{
http://localhost:8080/coreWeb
}}}
which should bring up a test suite index page with the Beehive logo and a list
of tests and test categories. When running a particular test, the yellow bar
at the bottom of each page is the Test Recorder.
To run the DRT test suite against a running server, run this command in shell
two:
{{{
ant drt.running
}}}
which puts the test results here:
{{{
$BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html
}}}
To run the BVT test suite from the same directory, run this command in shell
two:
{{{
ant bvt.running
}}}
which puts the test results here:
{{{
$BEEHIVE_HOME/netui/build/testRecorder/junit/bvt/html/junit-noframes.html
}}}
For more information, take a look through the
`$BEEHIVE_HOME/netui/test/webapps/drt/build.xml` file and look at the targets
and how they call into the webapp build and `start` / `stop` Tomcat targets in
build files that live in:
{{{
$BEEHIVE_HOME/test/ant
}}}
To stop Tomcat, run `ant stop` in shell two.
When diagnosing a test failure or problem running the webapp, there are several
places to look to see if errors have occurred:
* the Tomcat console
* the shell two console for errors reported when test recorder tests fail
* the Tomcat log files, which are stored by date in
`$BEEHIVE_HOME/external/jakarta-tomcat-5.0.25/logs`
* in addition, NetUI can log error messages into the Tomcat console, and soon
into a netui.log file
When reporting errors into the beehive-dev mailing list, having any failures /
errors reported in the above helps when diagnosing a problem.
Also, because we have error / failure test cases, it is often acceptable for
NetUI tests to throw exceptions or errors in the course of regular execution.
While seeing a stack trace can be alarming, lots of tests should do this. <g>
So, just be sure to check the test result output as the source of record for
the success of a particular test or suite.
= Overview of the Test Recorder =
The Test Recorder tests comprise the bulk of the NetUI tests suite. The TR is
a piece of software that NetUI develops which consists of client and server
pieces. It simulates the interaction of a web browser with a running server.
The TR runs in two modes -- "record" and "playback".
Developers use the TR in "record" mode to record a set of user interactions
through a normal web browser (like Firefox) against a running, Test Recorder
enabled web application. The result is a "recorded" test which is stored in an
XML file; examples of these can be found in
`$BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/tests`.
When the DRT or BVT suites run, the client half of the TR uses these "record"
XML files to "playback" a set of user interactions with the server. This is
done by reading request parameters out of the "record" file and building
requests to send to the server; then, the response stream is captured which
results in a "playback" file. The "record" and "playback" results are
compared; if there are no differences between the two, the tests are considered
a pass. Otherwise, the test is a failure. Currently, this diff is done
line-by-line and in the future could be done as an XML or HTML diff.
The TR is a very strict and accurate way for changes in the NetUI runtime to be
verified. Because the response stream of a rendered JSP is differenced against
an expected result, even simple changes can break tests, which can be good
because it helps NetUI developers understand the impact of changes to the code
base. Thus, we these tests help to maintain NetUI stability.
The TR is configured on a per-webapp basis using files like those used to
configure the DRT's coreWeb webapp:
{{{
$BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/config
}}}
The TR uses its own Ant file to run a test suite against a server:
{{{
$BEEHIVE_HOME/netui/test/ant/testRecorder.xml
}}}
Certainly, more documentation will follow on the TR and how to record / update
test results.
= JUnit API Tests =
The source for these tests lives in
`$BEEHIVE_HOME/netui/test/src/junitTests/...`. The !TestCases which run as
part of the DRTs are specified in
`$BEEHIVE_HOME/netui/test/ant/junitDrt.properties`. The JUnit DRTs can be run
like this:
{{{
cd $BEEHIVE_HOME/netui/test/ant
ant drt
}}}
This will build all of the test code and run the JUnit tests specified in the
above file. We use a simple `<netui-junit>` Ant task that takes the
.properties file above and filters all of the classes in the junitTests.jar
file to produce the set of !TestCases to run as DRTs. The Ant to drive the
JUnit DRTs lives in `$BEEHIVE_HOME/netui/test/ant/junit.xml`. The JUnit test
results can be found here:
{{{
$BEEHIVE_HOME/netui/build/drt.testResults/html/junit-noframes.html
}}}
