Hi, Kristian, thanks for your questions. My one overriding thought is
we should take this in incremental steps -- do all of these questions
need to be answered before we can rewrite a single old canon-based test
to a JUnit test? Can some of these questions be deferred?
Kristian Waagan wrote:
Hello,
I have wanted to rewrite a old canon based test to a JUnit test for a
while, but each time I get started I always run into issues that we do
not have guidelines for yet. So, I have a few questions I would like
some feedback on.
I know there are a lot of issues to consider, far more then the one I
mention here, but I hope to get a feeling of the general direction we
want to move in.
1) Should we support specific ordering of tests (within a single class)?
This feature is wanted if a number of tests operate on shared data and
each test method can be seen as a step in the test. If the order is
mixed up, the intermediate results will not be as expected and all the
tests may fail.
In general, assuming anything about the execution order of tests is
strongly disencouraged. However, since we are testing a database system,
we might want to allow for this. This feature can influence the both the
size of the test methods and the execution time for the test/suite.
I thought if you name your tests correctly (e.g. test_010, test_020,
etc.) then they are run in that order. Or is that not the case because
all method names on a class are hashed?
Assuming that's not the case, it seems to me that if you have specific
ordering requirements, you could use multiple test classes, create a
suite of those classes, and use the TestSetup mechanism you describe
below to do any setup work prior to running the suite.
2) Should we support setting up a shared test fixture for a set of tests?
This is a common issue with JUnit, and there are mechanisms to handle
it. For instance, we could let tests that require this to wrap itself in
a TestSetup instance (as described in the JUNit FAQ). Letting
DerbyJUnitTest extend TestSetup is another solution, but it might have
unwanted side-effects.
I think there is a lot of value to having something like this (see above
for an example).
3) Many tests just want a database to work on. Other tests want to
create their own database due to specific requirements. How do we make
this easy for the developer?
I think if you have a TestSetup mechanism, then a suite of tests can
depend on the fact that a database has been created. Creating a
database is pretty darn easy already; are there particular things that
are repetetive or error-prone?
4) Should it be possible to run single JUnit test/suite from the command
line by simply invoking the main method?
Personally, I think this should be possible. This require us to remove
the current use of main for running a test in the old test harness,
which I believe is doable with a few changes (in the old harness).
I think this is essential for debugging and incremental testing. +1 on
this idea.
5) Is the ultimate goal to end up with a pure JUnit based test harness?
If this is the goal, we should design for this and accept some extra
work for plugging JUnit tests into the old test harness.
I can see us getting there over time, but this is a Very Big Effort.
Are there particular things we have to decide upon now in order for this
to be possible in the future?
6) Where, and how, do we want to store settings for a test?
In a database! :)
I personally think a properties file is easier to work with than an XML
file, *as long as* we don't load these in as System properties, gads,
that's a nightmare.
7) How is it determined which framework(s) a JUnit test should be run
in? (i.e. embedded, client, JCC)
I think you should have ways to run a particular suite under a
particular framework, as you do today.
If you have other opinions or comments related to JUnit testing, please
put them forth :)
I think ther are two separate questions:
- Running tests under JUnit with a JUnit harness
- Rewriting existing tests to no longer use canons
I think we need to make sure we discuss these separately. Existing
tests can run under JUnit with canons just fine (that's what we're
already doing). I think that's could be big performance improvement
right there (I remember Dan showing the difference in performance).
Rewriting tests to no longer use canons (and thus .sql files) is a Much
Bigger Task and I suggest that be separated out, and we should discuss
its merits/demerits separately.
I hope we are able to get a discussion on all or some of these issues,
and that it results in a Wiki page describing JUnit testing for Derby. I
created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
I am sure you will get a discussion. In terms of it "resulting" in a
Wiki page, I think that is up to whoever has the itch to do that (it
sounds like you already do).
All the best,
David
--
Kristian
begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:[EMAIL PROTECTED]
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard
