Daniel John Debrunner wrote:
The current approach has very little magic, one decorator for client and
probably one decorator for DB2 client. I strongly agree that tests
should not be overly influenced by the suite setup.
I think this is a sound principle to keep in mind as work progresses on
this new testsuite. It might not be possible to avoid suite influence
totally, but hopefully it is kept at a level where it is not regarded as
a problem with the testsuite.
This is an interesting idea, some thoughts jump out at me:
- A single test would potentially run in many configurations:
embedded, client, db2 client, encryption DES, encryption AES, ...
If the number of configurations grows large, it might pay off to
carefully review wether something should be a configuration of its own
or rather be tested through a set of more specialized tests. When the
master files are gone and we have a new assertion-based testsuite, the
tests should only fail on things they are specifically looking for,
possibly making them less suited for testing of a large number of
configurations? I guess it will be a tradeoff, having many
configurations or more tests.
- The large number of decorators required per test could factor into
large memory use for an entire suite, not sure of the overhead of each
TestSetup.
- Embedded tests would be running with the network server booted by
default, not sure of the impact of this, starting the network server for
each individual TestCase would slow an entire suite down too much
Are you thinking of the impact of the network server running in the same
vm as junit?
- It does make running a single TestCase in all configurations easy
- The potential requirement to modify all tests when a new
configuration is added into the run, e.g. if I add a configuration that
only works in 10% of the tests (that's my itch) then I have to modify
the other 90% of tests to disable it from running. Assuming the default
as you say is to run each test in all possible configurations, I assume
this would be enforced in the base test class. Maybe that's the case
today with the current approach where a test is ideally self describing.
I guess that the default set of configurations that a TestCase supports
could be something smaller than the set of all possible configurations.
The default runtime configuration would be to run all tests in all
possible configurations, but a TestCase with default settings would
override this with its smaller (default) set. You would then have to fix
your 10% tests by adding the new configuration to their supported set,
instead of removing it from the other 90%. Still, that is work, but I
would hope we don't end up with a jungle of configurations that anyhow
would be a pain to maintain.
I do believe, though, that it makes sense that a TestCase itself
maintains the information about what configurations/frameworks it
supports or requires, and not suite classes located somewhere else.
- Maybe there are some configurations that are best handled at the leaf
level, e.g. embedded and client, but others are best handled at a suite
level, e.g. db2 client, encryption. So a mix of the two approaches would
be good.
Yes, that may very well be. As I've mentioned above I also think it
should be considered if all configurations are really needed or wether
it can be tested as well or better with new tests instead.
I'm going to continue my current setup approach because it's not opposed
to what you approach, the majority of the work will be getting tests
converted to JUnit and running in client & db2 client mode. Ie. most of
the work overlaps anyway.
Yes, that is true. The work you are doing now is making it a lot easier
to imagine how things may fit together with a testsuite written and run
only with junit, and to have opinions on that.
Vemund
