On 1/20/06, Kathey Marsden <[EMAIL PROTECTED]> wrote:
Myrna van Lunteren wrote:
>
>I think there's another change required in the test harness to support this
>type of testing. There'll be need for test skipping, for sed-ing, and
>(hopefully minimal) canons for diffing. Is that being implemented?
>
> I imagine something similar to what is currently in place for different JCC
>clients.
>
>However, with the JCC clients, we always use the derbyTesting.jar of the
>server level.
>
The more I think about it, the JCC model for the test harness is not
right. I think Raman has it right to test with the lower rev testing
jar. The product should always work at least at the level provided by
Min(serverVersion,clientVersion) and that is what she is testing. As
you mention, if you test with the newer testing jar and an older client
or server, the number of tests that can be run as new features are
added and the tests change will deteriorate rapidly.
This is what I propose as a quick way to be able to run these tests
and a reasonable model moving forward.
Summary
- Log a Jira entry for the likely regression.
- Create a new test harness property runwithMixedVersions=<true |
false> default true
- Create a BehaviourChecker class which tests can use to check expected
product behaviour
- Update 10.1 tests as needed to use the runwithMixedVersions property
or BehaviourChecker methods
Details below:
1) File a high priority Jira issue for the one apparent regression
found. <>(Stream of column value in result cannot be retrieved twice
error in connectionJdbc20.java and LobTest.java)
2) Add a property to the test harness, runwithMixedVersions=<true |
false> default true
This property would mean that the test would not run at all if the
server and client are different versions This is a quick way to get
the suite running cleanly.
3)In the functionalTests/util directory we make a BehaviourChecker class
(or better name) which tests can use to check the behaviour based on
combinations, drivers or whatever. It will take a connection in the
constructor and will have methods like
closesForwardOnlyRSOnFetchOfLastRow() or
supportsAutogeneratedKeysForMultipleRowInserts() which check the
DatabaseMetaData to make the right decision. For example,
closesForwardOnlyRSOnFetchOfLastRow() will be true for JCC and false
if the driverVersiion is greater than or equal to 10.2 (DERBY-213).
3) In the 10.1 branch, set the property for the failing tests. Hopefully
someone will start weekly runs with 10.1/10.2 combinations.
4) Java tests that need to handle different behaviour in order to pass
can either do something generic to resolve the problem or use
BehaviourChecker to do the right thing. Always the tests are being
changed in the lower level branch, right now that means 10.1.
Examples:
derbynet/NSinSameJVM.java
Test fails because network server no longer outputs connections to the
console by default. The test can be enabled by changing the location of
the console output for the test, since it is not the console output
being tested.
jdbcapi/autoGeneratedJdbc30.java
Test fails because of different results with (10.1 client/10.2 server). The 10.1 test could be be enabled by.
1) Add a method to BehaviourChecker
public boolean suppportsAutogeneratedKeysOnMultipleRowInsert()
{
// if database version from DatabaseMetaData
greatThanOrEqualTo(10,2,0) return true
// code left as excercise for the reader.
}
2) Change the test
BehaviourChecker checker = new BehaviourChecker(conn);
....
if (checker.suppportsAutogeneratedKeysOnMultipleRowInsert())
{
// expect one thing
}
else
{
// expect something else
}
Pros as I see it:
- We keep messy test changes in the old branches.
- We avoid excluding entire tests because of expected differences or a
single issue
- BehaviourChecker has the beneficial side effect of documenting
expected behaviour.
- BehaviourChecker could be expanded to handle a lot of the harness
property functionality as well as replace the various framework
checking logic in the individual tests.
Cons
- It is just not going to work for .sql tests.
- Those interested in maintaining this testing will have to do work in
the maintenance branches, but I just don't see a way around that and it
keeps our trunk tests clean.
- Behaviour is spelled different ways by different people so probably a
different name would be better.
Kathey
Makes sense.
re sql tests - I found, at least in main, while looking for the 'excludeJCC' property we have in the test harness, (which apparently is currently not used in any test properties file) that possibly the harness will look for canons for clients in directories ver + major number + minor number,
e.g. 'ver10.1'. or 'ver10.2'. It looked like it tries this whether the client is DerbyNet or DerbyNetClient. Worthwhile checking into for whomever is going to do this? This interesting bit of code is in harness/FileCompare.java. I've not tested it.
Myrna
