[
https://issues.apache.org/jira/browse/DERBY-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kathey Marsden updated DERBY-6210:
----------------------------------
Issue Type: Improvement (was: Bug)
> Create a mechanism to exclude some testing of internal interfaces and likely
> to change behavior from compatibility testing
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-6210
> URL: https://issues.apache.org/jira/browse/DERBY-6210
> Project: Derby
> Issue Type: Improvement
> Reporter: Kathey Marsden
>
> For compatibility and upgrade testing, in order to expose bugs and omitted
> release notes I run the tests from an older branch against new releases. The
> premise is that the functional tests are really just a JDBC application which
> should continue to pass with the new version.
> The JUnit tests for Derby though are not pure functional tests which are
> backward compatible. Often they make checks on things that are likely to
> change with new versions. This creates a large amount of noise in the
> testing: For example:
> 1) Checks for the exact contents or number of system tables.
> 2) Checks for "Not implemented" JDBC API calls which might become implemented.
> 3) Tests for client specific behavior which we expect might change to match
> embedded.
> 4) Unit tests which use internal interfaces that are likely to change.
> 5) Metadata tests which test the exact number of columns when
> 6) Diff based tests which are likely to change with message changes.
> It would be good to have a flag to omit these types of tests when doing
> compatibility testing.
> My thought is to have a system property derby.tests.testCompat=true and a
> method in BaseTestCase to test the property testCompat().
> Then blocks of testing which should not be run or should have a different
> behavior with compatibility testing can be flagged as:
> if (! testCompat()) {
> // do testing that we think might change, e.g. system table query.
> }
> This will allow the more detailed testing under normal circumstances but take
> some of the pain out of the compatibility testing moving forward. It would
> require folks to think as they add new tests whether their testing as to
> whether they expect the tested behavior to remain stable moving forward and
> put the block around it if they do not.
> I welcome feedback on whether this or some other approach is preferable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
