John Fallows wrote:
Is it always considered a build failure if integration tests do not pass 100%?
I did you mean acceptance tests?
In my understanding integration tests should pass 100% each time they
are run. The difference between unit and integration tests is only in
granularity. Unit tests focus on each component (one or few classes) in
separation, and integration tests focus on many components wired
together into a larger artifact - a GUI or web application for example.
Acceptance tests were (I believe) first introduced in eXtreme
Programming methodology. For each milestone (iteration) of application
development a number of features is scheduled, and a tests that verify
if the application provides the feature are created. At the beginning of
an iteration all of it's acceptance tests are expected to fail, and at
the end all are expected to pass - any failing tests indicate features
that must be rescheduled for further iterations. After the iteration is
completed the acceptance tests become regression tests that guard
developed features from being broken further down the road.
An acceptance test could be both an unit test or integration test,
depending on the type application. Usually the customer cares about
features that are accessible from GUI or web front end and therefore
acceptance tests will be integration tests, but when developing an
utility library unit tests could be valid acceptance tests.
As I understand it, acceptance test has the following features:
1) it expresses a contract item between the customer and the developer,
and therefore it should have an associated bugtracker item / XP story
card id for reference.
2) belongs to a specific milestone - should have a version associated
with it.
3) it's failure should not cause a SNAPSHOT build to fail.
4) it's failure should cause a release build to fail iff release version
= tests milestone.
I think acceptance tests support would be an interesting enhancement to
maven testing framework.
Probably the simplest way of dealing with it would be adding a JavaDoc
annotation to the test methods, and using XDoclet to extract the
information:
/**
* Test adding a book to the catalogue.
* @acceptance 1.0 LIBRARY-9
*/
public void testAddBook() {
// ...
}
The information could be used to display an useful report combined with
versions information from the POM: total/completed features per version,
and as dashboard data source.
Rafal
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
