Mike Matrigali wrote:
If there are no query plans printed then maybe the right thing to do
is to consider running the test multiple times, with different
properties set.
[snip]
maybe we could configure all the non-queryplan printing tests into
a suite and then enable or disable the interesting optimizer defaults.
I agree that these are good ideas in terms of testing thoroughness. My
knee-jerk reaction, though, is that we could end up running a lot of tests many
times in order to get all of the "interesting" cases. My guess is that such an
approach would significantly increase the running time of suites.All, which is
something I think we'd rather avoid (another reason we're switching to JUnit is
the hope that the suite will run more quickly, I believe?).
Maybe an alternate approach is to make sure that any tests which set
optimizer-specific properties run (and pass) with the "default" optimizer
configuration, as well, where "default" means no optimizer properties are set.
The exception here would of course be tests that print query plans, in which
case we'd either 1) exempt them from the "default" run, or 2) do the "default"
run but skip query plan validation in that case, thereby just verifying that the
queries complete without error and return the correct results.
There are two advantages to this approach. First, the number of tests which
explicitly set "derby.optimizer.*" properties is, I think, relatively small
compared to the total number of tests. And if you ignore the "noTimeout"
property it's even smaller. So the number of tests that see repeat runs would
be pretty minimal. Second, my guess is that most users of Derby do _not_ set
any optimizer-specific properties when they run their apps, in which case the
"default" test case (setting no optimizer properties) would be a (more) typical
user environment, which I think is good. Of course I could be wrong about what
typical users do here...?
Also note that such an approach would have correctly found DERBY-2500.
By converting this to JUNIT it should be much easier to deal with
results that are not guaranteed to be a specific order by having a
single shared routine that checks for ordered, unordered, and maybe also
unordered with possible inconsistent number of duplicates. This is
something that was cludged in current .sql tests but can be done right
in junit tests.
I think Knut Anders just checked in an "assertUnorderedResultSet()" method this
morning for exactly that reason. So we're moving in the right direction...
Army
