[
https://issues.apache.org/jira/browse/DERBY-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Matrigali updated DERBY-6210:
----------------------------------
Component/s: Test
> Create a mechanism to exclude some testing of internal interfaces and likely
> to change behavior from compatibility testing
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-6210
> URL: https://issues.apache.org/jira/browse/DERBY-6210
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.10.1.1
> Reporter: Kathey Marsden
>
> For compatibility and upgrade testing, in order to expose bugs and omitted
> release notes I run the tests from an older branch against new releases. The
> premise is that the functional tests are really just a JDBC application which
> should continue to pass with the new version.
> The JUnit tests for Derby though are not pure functional tests which are
> backward compatible. Often they make checks on things that are likely to
> change with new versions. This creates a large amount of noise in the
> testing: For example:
> 1) Checks for the exact contents or number of system tables.
> 2) Checks for "Not implemented" JDBC API calls which might become implemented.
> 3) Tests for client specific behavior which we expect might change to match
> embedded.
> 4) Unit tests which use internal interfaces that are likely to change.
> 5) Metadata tests which test the exact number of columns when
> 6) Diff based tests which are likely to change with message changes.
> It would be good to have a flag to omit these types of tests when doing
> compatibility testing.
> My thought is to have a system property derby.tests.testCompat=true and a
> method in BaseTestCase to test the property testCompat().
> Then blocks of testing which should not be run or should have a different
> behavior with compatibility testing can be flagged as:
> if (! testCompat()) {
> // do testing that we think might change, e.g. system table query.
> }
> This will allow the more detailed testing under normal circumstances but take
> some of the pain out of the compatibility testing moving forward. It would
> require folks to think as they add new tests whether their testing as to
> whether they expect the tested behavior to remain stable moving forward and
> put the block around it if they do not.
> I welcome feedback on whether this or some other approach is preferable.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
