Kathey Marsden created DERBY-6210:
-------------------------------------
Summary: Create a mechanism to exclude some testing of internal
interfaces and likely to change behavior from compatibility testing
Key: DERBY-6210
URL: https://issues.apache.org/jira/browse/DERBY-6210
Project: Derby
Issue Type: Bug
Reporter: Kathey Marsden
For compatibility and upgrade testing, in order to expose bugs and omitted
release notes I run the tests from an older branch against new releases. The
premise is that the functional tests are really just a JDBC application which
should continue to pass with the new version.
The JUnit tests for Derby though are not pure functional tests which are
backward compatible. Often they make checks on things that are likely to
change with new versions. This creates a large amount of noise in the testing:
For example:
1) Checks for the exact contents or number of system tables.
2) Checks for "Not implemented" JDBC API calls which might become implemented.
3) Tests for client specific behavior which we expect might change to match
embedded.
4) Unit tests which use internal interfaces that are likely to change.
5) Metadata tests which test the exact number of columns when
6) Diff based tests which are likely to change with message changes.
It would be good to have a flag to omit these types of tests when doing
compatibility testing.
My thought is to have a system property derby.tests.testCompat=true and a
method in BaseTestCase to test the property testCompat().
Then blocks of testing which should not be run or should have a different
behavior with compatibility testing can be flagged as:
if (! testCompat()) {
// do testing that we think might change, e.g. system table query.
}
This will allow the more detailed testing under normal circumstances but take
some of the pain out of the compatibility testing moving forward. It would
require folks to think as they add new tests whether their testing as to
whether they expect the tested behavior to remain stable moving forward and
put the block around it if they do not.
I welcome feedback on whether this or some other approach is preferable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
