If there are no query plans printed then maybe the right thing to do
is to consider running the test multiple times, with different
properties set. Considering you found a bug by effectively doing so
it seems worthwhile - but I don't know how worthwhile.
One the benefits of the whole junit conversion that Dan has been
suggesting is that it will be way easier in the future to configure
a set of tests to run under a different set of configurations. So
maybe we could configure all the non-queryplan printing tests into
a suite and then enable or disable the interesting optimizer defaults.
I would lean toward the default for any test that does not print
queryplans to not set any properties. Maybe there should be a
agreed upon way for all the junit tests to say if they print query plans
or not so that it would be easy to implement a suite of all
non-queryplan printing tests.
As both of you have said it is hard to guess at why the properties are
there without comments, but I will take a stab. I am going to guess
that this set of properties is there to try and force reproducible
ordering of results across all jvm's, OS's, hardware. And I would even
guess that the various properties were likely added at different times
to make some new problem jvm/os/hardware work where the previous
property did not work. Especially in
a test for distinct it probably is not a good idea to add an ORDER by
to all the queries - it may cover what is actually trying to be tested.
By converting this to JUNIT it should be much easier to deal with
results that are not guaranteed to be a specific order by having a
single shared routine that checks for ordered, unordered, and maybe also
unordered with possible inconsistent number of duplicates. This is
something that was cludged in current .sql tests but can be done right
in junit tests.
I noted there was some discussion about only testing for number of
rows returned rather than actual values. That seems like a bad path
to travel down. While for a specific instance it does seem unlikely
for the wrong row to be returned, in the worst case if everyone travels
this path we may have a set of tests that never check the actual return
values. The most interesting bugs uncovered by tests are the ones where
the bug was not expected at all.
Army wrote:
Andrew McIntyre (JIRA) wrote:
and since all the tests pass with or without ruleBasedOptimization set
to true, I'll leave that off. Not really worth investigating, since it
seems to have no effect on the new test.
Unless of course setting that property to true revealed an error once
upon a time and that error was fixed, thus removing the error. In that
case setting or not setting the property would lead to the same
behavior, which is in itself confirmation that whatever the old error
was (if there was one) is actually fixed.
Or as a different (and more pertinent) example: setting the
optimizeJoinOrder property to "true" (which is also the default) exposes
a bug that results in an error--namely, DERBY-2500. If I fix that bug
(which I hope to!) then running the query should return the same results
regardless of what optimizeJoinOrder is set to--i.e. optimizeJoinOrder
will at that point "have no effect on the test". But in the interest of
regression testing we still want to make sure the query is run with
optimizeJoinOrder=true (the default) as proof that DERBY-2500 is fixed.
Maybe something similar once existed for "ruleBasedOptimization=true"...?
*shrug*
Army
